Scholarlink

<p>Emotion recognition holds significant applications in fields such as natural language processing, computer vision, and speech recognition. However, traditional unimodal methods struggle to comprehensively capture the diversity of emotional expressions, while existing multimodal methods often focus on the textual modality and lack sufficient exploration in feature-level correlation. To address this, this paper proposes a feature-level interaction and adaptive fusion model based on cross-modal attention. Specifically, the model first extracts emotional representations from audio and visual modalities and aligns them in a shared space. Subsequently, a self-attention module is utilized for intra-modal modeling to capture intra-modal temporal dependencies. Simultaneously, we propose a cross-modal attention computation method based on feature-level interaction to explore fine-grained correlations and information complementarity at the temporal and feature levels between modalities. Finally, an adaptive fusion strategy is adopted to automatically learn modal weights, further enhancing modal complementarity. Experimental results demonstrate that the proposed model exhibits superior performance on both RAVDESS and IEMOCAP datasets, effectively improving the accuracy and robustness of multimodal sentiment analysis. The code is available at <a href="https://github.com/cstan-chun/MAMF/tree/master.">https://github.com/cstan-chun/MAMF/tree/master.</a></p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Feature-level interaction and adaptive fusion model based on cross-modal attention for audiovisual emotion recognition

  • 期刊论文

  • 发表日期：

    2026年1月16日

查看全文
  • Shuqiu Tan,
  • Chunsheng Tan

摘要

Emotion recognition holds significant applications in fields such as natural language processing, computer vision, and speech recognition. However, traditional unimodal methods struggle to comprehensively capture the diversity of emotional expressions, while existing multimodal methods often focus on the textual modality and lack sufficient exploration in feature-level correlation. To address this, this paper proposes a feature-level interaction and adaptive fusion model based on cross-modal attention. Specifically, the model first extracts emotional representations from audio and visual modalities and aligns them in a shared space. Subsequently, a self-attention module is utilized for intra-modal modeling to capture intra-modal temporal dependencies. Simultaneously, we propose a cross-modal attention computation method based on feature-level interaction to explore fine-grained correlations and information complementarity at the temporal and feature levels between modalities. Finally, an adaptive fusion strategy is adopted to automatically learn modal weights, further enhancing modal complementarity. Experimental results demonstrate that the proposed model exhibits superior performance on both RAVDESS and IEMOCAP datasets, effectively improving the accuracy and robustness of multimodal sentiment analysis. The code is available at https://github.com/cstan-chun/MAMF/tree/master.