Deep neural networks have become increasingly pivotal in Time Series Classification (TSC), offering applications across diverse fields such as healthcare and human activity recognition. However, the efficacy of these methods based deep neural networks can be hindered by the presence of jitter and noise in real-world time series data. To address this challenge, our study conducts extensive empirical experiments to investigate the impact of temporal scales on TSC accuracy. Our findings reveal that different temporal scales significantly influence classification performance. In light of these insights, we introduce an Adaptive Downscaling Strategy (ADS), designed to dynamically adjust input scales for optimal performance. ADS integrates a scale factor selector for determining the ideal temporal scale, coupled with an importance-based pooling technique to preserve crucial task-relevant information during downscaling. Our comprehensive evaluations across public datasets and multiple classification architectures demonstrate ADS’s superiority over deep learning methods lacking adaptive downscaling capabilities. These results not only validate the efficacy of ADS but also underscore its potential to enhance deep learning approaches for TSC in various domains.