Scholarlink

<p>Multimodal sentiment analysis (MSA) aims to infer sentiment by jointly modeling textual, acoustic, and visual information. Although existing multimodal fusion methods achieve strong performance when all modalities are available, their reliability often decreases when one or more modality streams are absent or unevenly informative. To address this problem, this paper proposes RoME: Robust multimodal expert learning for sentiment analysis with missing modalities, a two-stage robust multimodal embedding framework for sentiment analysis under incomplete modality conditions. In the first stage, RoME refines unimodal representations using modality-specific experts with soft routing, preserving modality-dependent structure under partial observations. In the second stage, RoME constructs adaptive multimodal representations through a Cross-Modal Interaction Transformer for inter-modality modeling, a Modality Reliability Re-Weighting mechanism to regulate modality contributions, a Residual Multimodal Corrector to compensate for fusion errors, and a Cross-Modal Denoising Autoencoder that provides reconstruction-driven regularization. Extensive experiments on the CMU-MOSI and CMU-MOSEI benchmarks demonstrate that RoME achieves strong performance across full-modality, partial-modality, and single-modality settings. Ablation studies, modality contribution analysis, and cross-validation results further confirm the effectiveness and stability of the proposed framework under controlled missing-modality conditions. Our code is available at <a href="https://github.com/atifmallo/RoME">https://github.com/atifmallo/RoME</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Robust multimodal expert learning for sentiment analysis with missing modalities

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月31日

查看全文
  • Atif Hussain,
  • Yu Gu,
  • Iftekharul Islam,
  • Aamir Ali,
  • He Zhang,
  • Chilato Chilato

摘要

Multimodal sentiment analysis (MSA) aims to infer sentiment by jointly modeling textual, acoustic, and visual information. Although existing multimodal fusion methods achieve strong performance when all modalities are available, their reliability often decreases when one or more modality streams are absent or unevenly informative. To address this problem, this paper proposes RoME: Robust multimodal expert learning for sentiment analysis with missing modalities, a two-stage robust multimodal embedding framework for sentiment analysis under incomplete modality conditions. In the first stage, RoME refines unimodal representations using modality-specific experts with soft routing, preserving modality-dependent structure under partial observations. In the second stage, RoME constructs adaptive multimodal representations through a Cross-Modal Interaction Transformer for inter-modality modeling, a Modality Reliability Re-Weighting mechanism to regulate modality contributions, a Residual Multimodal Corrector to compensate for fusion errors, and a Cross-Modal Denoising Autoencoder that provides reconstruction-driven regularization. Extensive experiments on the CMU-MOSI and CMU-MOSEI benchmarks demonstrate that RoME achieves strong performance across full-modality, partial-modality, and single-modality settings. Ablation studies, modality contribution analysis, and cross-validation results further confirm the effectiveness and stability of the proposed framework under controlled missing-modality conditions. Our code is available at https://github.com/atifmallo/RoME.