<p>Multimodal sentiment analysis (MSA) aims to infer affect from text, audio, and visual signals. Although recent fusion models achieve strong accuracy, many still suffer from two problems: spurious lexical correlations in the text stream and unstable fusion under noisy non-textual evidence. We propose the Text-enhanced Cross-modal Reinforced Fusion Network (TECRFN), a unified framework for unaligned MSA that addresses both issues. TECRFN first employs a Text Debiasing Module (TDM) to suppress sentiment-irrelevant lexical shortcuts and produce a cleaner textual anchor. It then uses a Text-enhanced Transformer (TET) and a Text-guided Cross-modal Reinforced Transformer (TGCRT) to inject textual semantics into audio and vision, perform query-adaptive dual-path fusion, and recalibrate noisy channels through similarity-biased gating and channel attention. A memory encoder and a unimodal label generation module further preserve long-range context and modality-specific discriminability. Experiments on CMU-MOSI and CMU-MOSEI show that TECRFN achieves state-of-the-art or highly competitive performance across multiple metrics. Additional ablation, robustness, and sensitivity studies demonstrate that the proposed design improves alignment quality, stability, and resistance to bias shift.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Text-enhanced cross-modal reinforced fusion network for multimodal sentiment analysis

  • Wantong Zhao,
  • Yongqing Wu

摘要

Multimodal sentiment analysis (MSA) aims to infer affect from text, audio, and visual signals. Although recent fusion models achieve strong accuracy, many still suffer from two problems: spurious lexical correlations in the text stream and unstable fusion under noisy non-textual evidence. We propose the Text-enhanced Cross-modal Reinforced Fusion Network (TECRFN), a unified framework for unaligned MSA that addresses both issues. TECRFN first employs a Text Debiasing Module (TDM) to suppress sentiment-irrelevant lexical shortcuts and produce a cleaner textual anchor. It then uses a Text-enhanced Transformer (TET) and a Text-guided Cross-modal Reinforced Transformer (TGCRT) to inject textual semantics into audio and vision, perform query-adaptive dual-path fusion, and recalibrate noisy channels through similarity-biased gating and channel attention. A memory encoder and a unimodal label generation module further preserve long-range context and modality-specific discriminability. Experiments on CMU-MOSI and CMU-MOSEI show that TECRFN achieves state-of-the-art or highly competitive performance across multiple metrics. Additional ablation, robustness, and sensitivity studies demonstrate that the proposed design improves alignment quality, stability, and resistance to bias shift.