Encrypted traffic classification plays a vital role in network security and management. While existing techniques that rely on byte or attribute sequences show promising results in traffic classification, they still have several limitations, including: (1) the sequence-based representations fall short in preserving the multi-modal information and spatial-temporal characteristics of datagrams; (2) the complexity of sequence models increases rapidly as the sequence length grows; (3) training classifiers for specific scenarios often involves a time-consuming and labor-intensive process of labeling data. In this paper, we propose representing datagrams as two homogeneous multi-modal spatial-temporal matrices that retain information from various modalities while maintaining the traffic’s spatial-temporal characteristics. Then, we introduce spatial-temporal state space models to better align with the inherent properties of these matrices and reduce computational complexity. Furthermore, we develop a self-supervised training paradigm called inter-flow contrast learning to capture flow semantics by utilizing large volumes of unlabeled data. Experimental results demonstrate that our method significantly outperforms state-of-the-art approaches across five real-world traffic datasets. We will release our code publicly after the double-blind review process concludes.