<p>Shared information refers to the common semantic information across multiple modalities, and complementary information refers to modality-specific information that complements other modalities. How to fully utilize this information is a key issue in the multimodal sentiment analysis. In this paper, we first propose a Slice Aggregation (SA) algorithm to address the issue of correlation over time. We use sliding windows to calculate the horizontal and vertical correlations, and then aggregate slices into a series of chunks, each represents a set of successive slices with consistent correlation. Second, we introduce a Dynamic Fusion (DF) strategy comprising two components: shared information fusion and complementary information fusion. The former utilizes a multilayer perceptron (MLP) to extract high-level shared representations, whereas the latter employs a cross-modal multi-head attention mechanism to fuse low-level complementary information. Finally, we propose an SA-DF framework where SA organizes raw slices into correlation-consistent chunks, and DF progressively fuses features across these chunks. The concatenated fused features are used for final sentiment prediction. The experiments on CMU-MOSI and CH-SIMS datasets show that the proposed SA-DF can achieve the best performance on sentiment analysis tasks when compared with the state-of-the-art baselines.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal sentiment analysis based on slice aggregation and dynamic fusion

  • Zhouwen Zhan,
  • Dongtao Cao,
  • Zheyi Chen,
  • Hongju Cheng,
  • Zhiyong Yu

摘要

Shared information refers to the common semantic information across multiple modalities, and complementary information refers to modality-specific information that complements other modalities. How to fully utilize this information is a key issue in the multimodal sentiment analysis. In this paper, we first propose a Slice Aggregation (SA) algorithm to address the issue of correlation over time. We use sliding windows to calculate the horizontal and vertical correlations, and then aggregate slices into a series of chunks, each represents a set of successive slices with consistent correlation. Second, we introduce a Dynamic Fusion (DF) strategy comprising two components: shared information fusion and complementary information fusion. The former utilizes a multilayer perceptron (MLP) to extract high-level shared representations, whereas the latter employs a cross-modal multi-head attention mechanism to fuse low-level complementary information. Finally, we propose an SA-DF framework where SA organizes raw slices into correlation-consistent chunks, and DF progressively fuses features across these chunks. The concatenated fused features are used for final sentiment prediction. The experiments on CMU-MOSI and CH-SIMS datasets show that the proposed SA-DF can achieve the best performance on sentiment analysis tasks when compared with the state-of-the-art baselines.