Multimodal Sentiment Analysis (MSA) aims to integrate text, audio, and visual information to better recognize human emotions. The reciprocal exchange of semantic information across text, audio, and video modalities often lacks a structured mechanism in previous works. This difficulty in modeling complex cross-modal dependencies consequently restricts their capacity to capture detailed semantic correlations. Moreover, previous methods often fail to align disparate modalities, causing semantic inconsistencies and redundancy during fusion. To address these issues, this paper introduces a novel bidirectional cross-modal fusion framework named BiGMF. The method is built upon a hierarchical cross-modal interaction architecture that enables bidirectional information exchange at multiple levels, enhancing the modeling capacity for cross-modal interactions. In addition, a geometric volume regularization strategy is introduced to reinforce semantic consistency. This strategy explicitly promotes the alignment of modality-specific features by constraining the geometric volume of their joint distribution in a shared embedding space. Extensive experiments on two MSA benchmarks demonstrate the effectiveness of the proposed method.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

BiGMF: Multimodal Sentiment Analysis By Bidirectional Cross-Modal Attention with Geometric Volume Regularization

  • Youwei Zhang,
  • Qishen Chen,
  • Yuzhe Huang,
  • Huahu Xu,
  • Baochao Qi,
  • Lizhi Zhou

摘要

Multimodal Sentiment Analysis (MSA) aims to integrate text, audio, and visual information to better recognize human emotions. The reciprocal exchange of semantic information across text, audio, and video modalities often lacks a structured mechanism in previous works. This difficulty in modeling complex cross-modal dependencies consequently restricts their capacity to capture detailed semantic correlations. Moreover, previous methods often fail to align disparate modalities, causing semantic inconsistencies and redundancy during fusion. To address these issues, this paper introduces a novel bidirectional cross-modal fusion framework named BiGMF. The method is built upon a hierarchical cross-modal interaction architecture that enables bidirectional information exchange at multiple levels, enhancing the modeling capacity for cross-modal interactions. In addition, a geometric volume regularization strategy is introduced to reinforce semantic consistency. This strategy explicitly promotes the alignment of modality-specific features by constraining the geometric volume of their joint distribution in a shared embedding space. Extensive experiments on two MSA benchmarks demonstrate the effectiveness of the proposed method.