<p>Current multimodal emotion recognition (MER) methods face significant challenges, particularly in learning across different modalities and dealing with semantic inconsistencies. Speech and text contain unique and valuable emotional information; however, current approaches prioritize integrating multimodal features without adequately examining the emotional details within each modality, which can lead to a failure to capture the specific features of each modality, ultimately underutilizing the potential of unimodal representations. This paper proposes a multimodal emotion recognition network (SCMI-Net) based on semantic constraints and modality interaction. Specifically, we present an intra-modal learning method with emotion constraint that converts universal features into emotion-related features specific to each modality. Moreover, we introduce a feature-level interaction strategy that utilizes cross-modal attention and gating mechanisms to derive deep fused representations from speech and text. Meanwhile, to tackle class imbalance issues, we also incorporate label smoothing to regularize one-hot encodings and apply supervised contrastive learning to obtain highly discriminative emotional representations. Comprehensive experiments conducted on the IEMOCAP and MELD datasets demonstrate the effectiveness of our method. Specifically, under the same data partition settings as the IEMOCAP dataset, we achieve an unweighted accuracy (UA) of 78.6% and a weighted accuracy (WA) of 77.6%, surpassing the performance of recent MER methods. Moreover, in the 5-classification and 7-classification tests on the MELD dataset, our method yields average results of 67.2% WA and 63.7% Weighted F1 (WF1), respectively.</p>