Speech analysis has emerged as a promising tool for depression detection. Currently, deep learning methods modeling Mel spectrograms are widely adopted in speech depression detection (SDD) task. However, few studies have investigated the depress-related acoustic patterns in Mel spectrograms. Therefore, this study systematically identifies two distinctive temporal characteristics: 1) Temporal Locality, where diagnostic features are sparsely distributed in local regions; 2) Multi-scale, where pathological patterns appear across hierarchical temporal resolutions. To address these issues, we propose a Temporal Multi-scale Perception Network (TMP-Net), which involves: a Spectral-Temporal Feature Extractor (STFE) for learning spectral-temporal features and generating condensed representations, and a Multi-Scale Partition-Fusion module via Attention (MS-PFA) for capturing multi-scale temporal patterns by performing temporal attention mechanism. Extensive experiments on the NRAC, CMDC and EATD datasets demonstrate that TMP-Net not only achieves superior performance with merely 0.116M parameters—significantly fewer than those of state-of-the-art methods (1.63M, 0.92M, and 0.55M)—but also enhances the interpretability of depression-specific acoustic biomarkers modeling.