Multimodal Sentiment Analysis (MSA) aims to infer human emotions by integrating data from multiple modalities, such as linguistic, visual, and acoustic signals. It has found broad applications in domains such as social media opinion mining and human-computer interaction. Existing MSA methods typically focus on extracting cross-modal shared features or directly fusing heterogeneous modalities. However, processing or fusing all modalities indiscriminately may amplify irrelevant noise and introduce redundant information, thus undermining the effectiveness of the learned representations. To address these challenges, we propose a novel framework, Multimodal Sentiment Analysis via Spatio-Temporal Decoupling and Language-Focused Fusion (ST-DLF), specifically designed for the MSA task. In the feature decoupling stage, ST-DLF incorporates a spatio-temporal dual-attention mechanism to enhance the representational quality of each modality. Meanwhile, a quadruple loss function is introduced to suppress redundant and overlapping information across modalities. In the fusion stage, we design a Language-Focused Attractor, which employs a language-guided cross-attention mechanism to selectively incorporate complementary modality-specific information, thereby enriching sentiment-discriminative representations. Furthermore, by integrating low-level shared features, medium-level cross-modal features, and high-level decoupled features, ST-DLF further enhances the discriminative capacity of the fused representations. Extensive experiments on two benchmark datasets demonstrate the effectiveness of our approach, with consistent improvements over representative state-of-the-art baselines. Our code is available at https://github.com/sakitma-r/ST-DLF .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal Sentiment Analysis via Spatio-Temporal Decoupling and Language-Focused Fusion

  • Jinhong Li,
  • Leheng Zhang,
  • Hui Cui,
  • Jingxian Wang,
  • Rui Li

摘要

Multimodal Sentiment Analysis (MSA) aims to infer human emotions by integrating data from multiple modalities, such as linguistic, visual, and acoustic signals. It has found broad applications in domains such as social media opinion mining and human-computer interaction. Existing MSA methods typically focus on extracting cross-modal shared features or directly fusing heterogeneous modalities. However, processing or fusing all modalities indiscriminately may amplify irrelevant noise and introduce redundant information, thus undermining the effectiveness of the learned representations. To address these challenges, we propose a novel framework, Multimodal Sentiment Analysis via Spatio-Temporal Decoupling and Language-Focused Fusion (ST-DLF), specifically designed for the MSA task. In the feature decoupling stage, ST-DLF incorporates a spatio-temporal dual-attention mechanism to enhance the representational quality of each modality. Meanwhile, a quadruple loss function is introduced to suppress redundant and overlapping information across modalities. In the fusion stage, we design a Language-Focused Attractor, which employs a language-guided cross-attention mechanism to selectively incorporate complementary modality-specific information, thereby enriching sentiment-discriminative representations. Furthermore, by integrating low-level shared features, medium-level cross-modal features, and high-level decoupled features, ST-DLF further enhances the discriminative capacity of the fused representations. Extensive experiments on two benchmark datasets demonstrate the effectiveness of our approach, with consistent improvements over representative state-of-the-art baselines. Our code is available at https://github.com/sakitma-r/ST-DLF .