<p>Multi-modal self-supervised learning has gained prominence in Human Action Recognition (HAR) due to the labor-intensive and poorly defined nature of annotating action data. Most existing methods rely on contrastive learning with positive and negative sample pairs, often incorporating false negatives that hinder learning effectiveness. To address this limitation, we propose a Cross-Modal Contrastive Sample Mining model. Our approach constructs a Temporal Similarity Matrix (TSM) for multi-modal feature sequences using the soft Dynamic Temporal Warping (soft-DTW) algorithm to evaluate possible similarity among random selected action instances with various lengths. Leveraging the TSM, our method precisely identifies potential positive samples within each batch, expanding the positive set and reducing false negatives. Additionally, intra-modal negative samples are incorporated to improve embedding alignment within each modality. An auxiliary soft-DTW loss further enhances cross-modal feature alignment. Extensive experiments on challenging datasets demonstrate that our method learns more effective representations compared to the baseline CMC model. When fine-tuned on a small labeled dataset, our method outperforms a fully supervised model in accuracy.</p>