Transformer-based models have demonstrated strong performance in time series classification; however, most existing approaches naively adopt standard Transformer architectures without fully accounting for the unique characteristics of time series data. This oversight introduces several limitations, including reliance on single-domain feature analysis, constrained fixed-scale feature extraction, inappropriate positional encoding, and substantial computational overhead. To address these challenges, we propose the Time-Frequency Integrated Multi-Layer Transformer Network (TFTN), a novel architecture that effectively integrates temporal and frequency-domain information. Specifically, TFTN leverages the Fourier Transform to extract frequency-domain features, which are then fused with time-domain representations. A hierarchical slicing mechanism is introduced to enable multi-scale feature extraction from the combined time-frequency space. To better capture temporal dependencies, we introduce absolute time-based positional encoding. Furthermore, a multi-head latent attention mechanism is employed to enhance classification accuracy while improving computational efficiency. TFTN offers three primary contributions: (1) integration of frequency-domain information into the Transformer framework, (2) hierarchical multi-scale feature extraction, and (3) time-aware positional encoding with reduced computational cost. Extensive experiments conducted on 19 publicly available UEA benchmark datasets demonstrate that TFTN consistently outperforms state-of-the-art baseline models, validating its effectiveness for time series classification tasks.