<p>Against the backdrop of the rapid growth of multimodal data on social media, effectively integrating complementary information from textual and visual sources to enhance sentiment analysis has become a central focus in multimodal research. Existing approaches, however, often neglect the distinct emotional information inherent in images and text, and are vulnerable to interference from redundant or superficial features, which limits their ability to capture deeper core emotional expressions. To address these challenges, this paper proposes a multimodal sentiment analysis framework that leverages modality-specific information alongside data augmentation. Specifically, the approach first enhances the coherence and completeness of textual and visual sentiment representations by incorporating a BiLSTM network on top of foundational feature encodings. Next, a cross-modal interaction module is employed to align fine-grained semantic associations between image patches and textual words. We then introduce a modality-specific feature learning module to extract unique affective expression information from each modality, which is subsequently fused with shared features to strengthen the complementarity and completeness of cross-modal affective semantics. Additionally, a supervised contrastive learning strategy based on data augmentation is applied to guide the model toward focusing on core affective expression features, thereby improving its robustness to complex emotional expressions. Finally, experimental evaluations are conducted on two multimodal sentiment analysis datasets, benchmarking against state-of-the-art methods. On the MVSA-Single dataset, our method surpasses the state-of-the-art by 1.01% in accuracy and 2.8% in F1 score. On the TumEmo dataset, it outperforms baseline methods by 0.42% in accuracy and 0.57% in F1 score. These experimental results demonstrate the significant advantages of the proposed model.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal sentiment analysis based on modality-specific information and data augmentation

  • Hongbin Wang,
  • Zhucheng Zhang,
  • Di Jiang

摘要

Against the backdrop of the rapid growth of multimodal data on social media, effectively integrating complementary information from textual and visual sources to enhance sentiment analysis has become a central focus in multimodal research. Existing approaches, however, often neglect the distinct emotional information inherent in images and text, and are vulnerable to interference from redundant or superficial features, which limits their ability to capture deeper core emotional expressions. To address these challenges, this paper proposes a multimodal sentiment analysis framework that leverages modality-specific information alongside data augmentation. Specifically, the approach first enhances the coherence and completeness of textual and visual sentiment representations by incorporating a BiLSTM network on top of foundational feature encodings. Next, a cross-modal interaction module is employed to align fine-grained semantic associations between image patches and textual words. We then introduce a modality-specific feature learning module to extract unique affective expression information from each modality, which is subsequently fused with shared features to strengthen the complementarity and completeness of cross-modal affective semantics. Additionally, a supervised contrastive learning strategy based on data augmentation is applied to guide the model toward focusing on core affective expression features, thereby improving its robustness to complex emotional expressions. Finally, experimental evaluations are conducted on two multimodal sentiment analysis datasets, benchmarking against state-of-the-art methods. On the MVSA-Single dataset, our method surpasses the state-of-the-art by 1.01% in accuracy and 2.8% in F1 score. On the TumEmo dataset, it outperforms baseline methods by 0.42% in accuracy and 0.57% in F1 score. These experimental results demonstrate the significant advantages of the proposed model.