The existence of an early matter-dominated epoch prior to the Big Bang Nucleosynthesis (BBN) may lead to a scenario where the thermal dark matter cools faster than plasma before the radiation-dominated era begins. We show that the s-wave elastic scattering is sufficient to partially decouple the dark matter from the plasma if the entropy injection during the reheating era depends on the bath temperature. In contrast, p-wave scattering leads to full decoupling in such a cosmological backdrop. The decoupling of the dark matter before the end of reheating results in an additional amount of cooling, reducing its free-streaming horizon compared to the usual radiation-dominated cosmology. The enhanced matter perturbations for scales entering the horizon prior to the end of reheating, combined with the reduced free-steaming horizon, increase the number density of sub-earth mass halos. The resulting boost in the dark matter annihilation signatures could offer an intriguing probe to differentiate pre-BBN non-standard cosmological epochs.