Multimodal Sentiment Analysis (MSA) leverages information from multiple modalities to predict sentiment labels. However, existing approaches often suffer from the challenge of fitting spurious correlations between multimodal features and sentiment labels, with varying degrees of spurious correlations across different modalities. To address this issue, we propose MbrCL, a novel framework that integrates dynamic fusion and distance-aware contrastive learning. The dynamic fusion process is carried out in three steps: (1) disentangling robust and biased features within each modality, (2) estimating the degree of bias using the biased features, and (3) determining implicit and dominant modalities based on bias weights. Implicit modalities are employed to guide the computation of inter-modal correlations. Subsequently, the multimodal features are utilized to compute mixed-modal correlations through our proposed distance-aware contrastive learning, which incorporates affective label distance information. Finally, we identify sentiment-related information using a self-attention mechanism based on mixed-modal correlations and robust features extracted from visual and audio modalities. Experimental results on CMU-MOSI and CMU-MOSEI demonstrate that our model achieves state-of-the-art performance by effectively eliminating the interference of spurious correlations, leading to more accurate sentiment predictions.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal Sentiment Analysis with Modality-Robust and -Biased Representations and Distance-Aware Contrastive Learning

  • Lang Shen,
  • Qifei Zhang,
  • Wenjuan Li,
  • Minfeng Lu,
  • Xiubo Liang

摘要

Multimodal Sentiment Analysis (MSA) leverages information from multiple modalities to predict sentiment labels. However, existing approaches often suffer from the challenge of fitting spurious correlations between multimodal features and sentiment labels, with varying degrees of spurious correlations across different modalities. To address this issue, we propose MbrCL, a novel framework that integrates dynamic fusion and distance-aware contrastive learning. The dynamic fusion process is carried out in three steps: (1) disentangling robust and biased features within each modality, (2) estimating the degree of bias using the biased features, and (3) determining implicit and dominant modalities based on bias weights. Implicit modalities are employed to guide the computation of inter-modal correlations. Subsequently, the multimodal features are utilized to compute mixed-modal correlations through our proposed distance-aware contrastive learning, which incorporates affective label distance information. Finally, we identify sentiment-related information using a self-attention mechanism based on mixed-modal correlations and robust features extracted from visual and audio modalities. Experimental results on CMU-MOSI and CMU-MOSEI demonstrate that our model achieves state-of-the-art performance by effectively eliminating the interference of spurious correlations, leading to more accurate sentiment predictions.