RGB-T semantic segmentation enhances model robustness in challenging scenarios by integrating visible (RGB) and thermal infrared (TIR) modalities. However, most existing methods rely on simple fusion strategies—such as concatenation, element-wise addition, or weighted summation—to combine information from both modalities, often neglecting deeper cross-modal interactions. To address this limitation, we propose a cross-modal contrastive learning framework that explicitly models both intra-modal and inter-modal relationships. Intra-modal contrastive learning improves the discriminability of multi-level features within each modality, while inter-modal contrastive learning reduces the discrepancy between RGB and TIR embeddings at corresponding feature levels. Extensive experiments on three representative models (CMX, EAEFNet, and LASNet) and two benchmarks (MFNet and PST900) demonstrate the effectiveness and generalization ability of our approach by consistently improving segmentation performance across datasets.