Dynamic Dominate-Modal-Aware Model for Multimodal Sentiment Analysis
摘要
Multimodal sentiment analysis (MSA) has garnered increasing attention for its ability to integrate information from diverse modalities, such as text, audio, and video, to improve sentiment prediction. However, modality imbalance and inter-modal incongruity remain significant challenges, as different modalities contribute unequally to sentiment understanding in various contexts. In this paper, we propose a Dynamic Dominate-Modal-aware Model (DDMM) that addresses these issues by adaptively selecting the most informative modality for each input and enhancing cross-modal interactions through a novel attention-based fusion strategy. Our framework consists of three key components: 1) dedicated encoders for each modality (text, audio, video) to extract modality-specific features; 2) a Dynamic Dominant Modality Selection mechanism that computes an importance score for each modality and selects the most relevant one for sentiment analysis; and 3) a Cross-Dominant-Modal Attention Fusion module that integrates the complementary information from auxiliary modalities into the dominant modality’s representation. This adaptive and selective fusion improves the robustness and precision of sentiment predictions. Extensive experiments on benchmark datasets (CMU-MOSI and CMU-MOSEI) demonstrate that our approach effectively mitigates modality imbalance and achieves superior performance in both sentiment classification and regression tasks, outperforming existing methods. Our model provides a robust solution for real-world multimodal sentiment analysis applications, where input modalities vary in quality and relevance.