Scholarlink

<p>Multimodal emotion recognition&#xa0;(MER) plays a vital role in human–computer interaction by integrating textual, visual, and auditory cues to detect emotions accurately. However, MER faces significant challenges in effectively fusing heterogeneous modalities due to differences in data formats, semantic nuances, and representational structures. Most existing methods perform shallow fusion at an early stage, failing to capture complex intra- and inter-modal dependencies and often introducing substantial redundant information unrelated to emotion. This paper proposes a multilayer transformers fusion model with gating network, featuring three key innovations: a multilayer transformer architecture that leverages self-attention to capture complex intra- and inter-modal dependencies; four independent multimodal gating networks that dynamically adjust fusion weights to preserve modality-specific information while reducing redundancy; and a progressive fusion strategy that incrementally integrates salient features across layers for enhanced robustness. Evaluated on the CHERMA dataset, the most recent and comprehensive Chinese multimodal emotion recognition benchmark, the model achieves a weighted F1-score of 72.92%, representing a 9.22% improvement over single-modal baselines and outperforming state-of-the-art methods. Ablation studies confirm the effectiveness of each component. These results demonstrate that the proposed framework improves the robustness and accuracy of multimodal emotion recognition.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A multilayer transformers fusion model with gating network for multimodal emotion recognition

  • 期刊论文

  • 发表日期：

    2026年7月5日

查看全文
  • Jujian Lv,
  • Yaoxin Liang,
  • Haoyuan Chen,
  • Xiaoli Gong,
  • Jiawen Li,
  • Kaihan Lin,
  • Leijun Wang,
  • Jun Yuan,
  • Xianxian Zeng,
  • Rongjun Chen

摘要

Multimodal emotion recognition (MER) plays a vital role in human–computer interaction by integrating textual, visual, and auditory cues to detect emotions accurately. However, MER faces significant challenges in effectively fusing heterogeneous modalities due to differences in data formats, semantic nuances, and representational structures. Most existing methods perform shallow fusion at an early stage, failing to capture complex intra- and inter-modal dependencies and often introducing substantial redundant information unrelated to emotion. This paper proposes a multilayer transformers fusion model with gating network, featuring three key innovations: a multilayer transformer architecture that leverages self-attention to capture complex intra- and inter-modal dependencies; four independent multimodal gating networks that dynamically adjust fusion weights to preserve modality-specific information while reducing redundancy; and a progressive fusion strategy that incrementally integrates salient features across layers for enhanced robustness. Evaluated on the CHERMA dataset, the most recent and comprehensive Chinese multimodal emotion recognition benchmark, the model achieves a weighted F1-score of 72.92%, representing a 9.22% improvement over single-modal baselines and outperforming state-of-the-art methods. Ablation studies confirm the effectiveness of each component. These results demonstrate that the proposed framework improves the robustness and accuracy of multimodal emotion recognition.