<p>One of the key challenges in Multimodal Sentiment Analysis (MSA) is data missing, with existing research primarily focusing on optimizing model architectures to handle missing data scenarios. However, these methods often overlook the potential shared information between noisy auxiliary modalities (audio and visual) and fail to effectively model the complex dependencies among hierarchical representations. To address these challenges, we propose HiGraMI, a hierarchical graph fusion network incorporating mutual information (MI) enhancement. The model tackles the issues mentioned above through two complementary components. First, the Cross-modal MI Enhancement (CMIE) module generates stable auxiliary representations by maximizing MI between audio and visual features. These auxiliary representations are then used to enhance the quality of text representations adaptively through a dynamic adjustment strategy, improving their robustness. Furthermore, the Hierarchical Graph Attention Fusion (HGAF) module treats hierarchical and cross-modal representations as nodes in a graph. It then uses a Graph Attention Network (GAT) to capture interactions between cross-level and cross-modal representations under data missing scenarios. This process generates robust multimodal joint representations, which significantly improve the model’s robustness and sentiment prediction performance. Experimental results demonstrate that HiGraMI outperforms current mainstream baseline models across most metrics and data missing scenarios on the MOSI and MOSEI datasets, achieving state-of-the-art performance.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Hierarchical Graph Fusion with Mutual Information Enhancement for Robust Multimodal Sentiment Analysis

  • Nanjie Zheng,
  • Yiyang Tang,
  • Jiuzhou Chen,
  • Haoming Liu,
  • Qian Chen

摘要

One of the key challenges in Multimodal Sentiment Analysis (MSA) is data missing, with existing research primarily focusing on optimizing model architectures to handle missing data scenarios. However, these methods often overlook the potential shared information between noisy auxiliary modalities (audio and visual) and fail to effectively model the complex dependencies among hierarchical representations. To address these challenges, we propose HiGraMI, a hierarchical graph fusion network incorporating mutual information (MI) enhancement. The model tackles the issues mentioned above through two complementary components. First, the Cross-modal MI Enhancement (CMIE) module generates stable auxiliary representations by maximizing MI between audio and visual features. These auxiliary representations are then used to enhance the quality of text representations adaptively through a dynamic adjustment strategy, improving their robustness. Furthermore, the Hierarchical Graph Attention Fusion (HGAF) module treats hierarchical and cross-modal representations as nodes in a graph. It then uses a Graph Attention Network (GAT) to capture interactions between cross-level and cross-modal representations under data missing scenarios. This process generates robust multimodal joint representations, which significantly improve the model’s robustness and sentiment prediction performance. Experimental results demonstrate that HiGraMI outperforms current mainstream baseline models across most metrics and data missing scenarios on the MOSI and MOSEI datasets, achieving state-of-the-art performance.