<p>Emotion recognition in conversations is a critical component of emotional dialogue systems, playing a significant role in areas such as human-computer interaction and social media analysis. However, existing research primarily focuses on modeling contextual semantic information and speaker-specific details, while overlooking the subtle temporal information within long-range context dependencies and the heterogeneity between modalities. This results in insufficient global semantic understanding of conversations and prevents the optimal fusion of different modality information. To address these challenges, this paper proposes the Mamba-Based Emotional Hyper-Modal Assisted Multi-Granularity Fusion Method (MHAMF) for emotion recognition in conversations. This method constructs two feature subspaces and utilizes the mamba structure to extract subtle temporal information from dynamic emotions in the dialogue, thereby building an auxiliary hyper-modal. Additionally, through a hierarchical cross-modal fusion structure, it iteratively optimizes the fusion of fine-grained multi-modal features at each layer, adaptively weighting the two subspace features to maximize the complementary fusion effect of multi-modal features. Extensive experiments on two public datasets, IEMOCAP and MELD, demonstrate that MHAMF achieves significant accuracy and robustness in emotion recognition in conversations.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MHAMF: mamba-based emotional hyper-modal assisted multi-granularity fusion for emotion recognition in conversations

  • Jun Wu,
  • Tianfeng Zhang,
  • Shilong Jing,
  • Yu Zheng,
  • Jinyu Liu,
  • Yu Chen,
  • Fang Deng

摘要

Emotion recognition in conversations is a critical component of emotional dialogue systems, playing a significant role in areas such as human-computer interaction and social media analysis. However, existing research primarily focuses on modeling contextual semantic information and speaker-specific details, while overlooking the subtle temporal information within long-range context dependencies and the heterogeneity between modalities. This results in insufficient global semantic understanding of conversations and prevents the optimal fusion of different modality information. To address these challenges, this paper proposes the Mamba-Based Emotional Hyper-Modal Assisted Multi-Granularity Fusion Method (MHAMF) for emotion recognition in conversations. This method constructs two feature subspaces and utilizes the mamba structure to extract subtle temporal information from dynamic emotions in the dialogue, thereby building an auxiliary hyper-modal. Additionally, through a hierarchical cross-modal fusion structure, it iteratively optimizes the fusion of fine-grained multi-modal features at each layer, adaptively weighting the two subspace features to maximize the complementary fusion effect of multi-modal features. Extensive experiments on two public datasets, IEMOCAP and MELD, demonstrate that MHAMF achieves significant accuracy and robustness in emotion recognition in conversations.