Existing RGBT tracking methods typically implement cross-modal feature fusion by designing modality interaction modules at different network layers. However, such strategies often rely on redundant layer-wise computations and repeated message passing, resulting in high computational cost and low efficiency when incorporating temporal modeling. This hinders their ability to achieve effective joint spatio-temporal and multimodal representation. To address this issue, we propose a Temporal Multi-modal Fusion Network, named TMFNet, which aims to achieve efficient and robust RGBT tracking with linear complexity. Specifically, TMFNet introduces a Cross-modal Mamba Interaction module based on a state space model, which employs a novel single-step interaction mechanism. By sparsely sampling historical frames, the module performs a single-step “aggregation and distribution” fusion across modalities, effectively capturing both complementary modality information and target evolution dynamics. Furthermore, a Progressive Mamba Fusion module is designed to gradually align RGB and thermal features in a unified semantic space, enhancing deep-level multimodal representation. Extensive experiments on four public RGBT tracking benchmarks demonstrate that TMFNet significantly outperforms state-of-the-art methods in both accuracy and robustness while maintaining high inference efficiency.