Effective modeling of complex dependencies between utterances is crucial for accurate sentiment recognition in multi-modal dialogue systems. Conventional graph neural network methods rely on static fully connected graphs, where nodes are influenced by both intra-modal and inter-modal information, leading to conflicts during data fusion. Additionally, static graph structures based on fixed time windows lack the flexibility to capture emotional dynamics. To address these challenges, we propose Dynamic Graph Fusion and Sentiment Polarity Contrastive Learning for Conversational Multimodal Emotion Recognition. Our approach alternates between modeling temporal cross-modal dependencies and constructing a hybrid graph with semantically driven connections. The graph weights are dynamically determined by cosine similarity between node features. Sentiment Polarity Contrastive Learning enhances the model’s ability to learn discriminative emotional features. Experimental results show that our method achieves an IEMOCAP-6 accuracy of 71.60% and F1-score of 71.71%, and MELD accuracy of 67.13% and F1-score of 65.90%.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Conversational Multi-modal Emotion Recognition Based on Heterogeneous Dialogue Graph and Sentiment Polarity Contrastive Learning

  • Cheng-Shan Jiang,
  • Zhen-Tao Liu

摘要

Effective modeling of complex dependencies between utterances is crucial for accurate sentiment recognition in multi-modal dialogue systems. Conventional graph neural network methods rely on static fully connected graphs, where nodes are influenced by both intra-modal and inter-modal information, leading to conflicts during data fusion. Additionally, static graph structures based on fixed time windows lack the flexibility to capture emotional dynamics. To address these challenges, we propose Dynamic Graph Fusion and Sentiment Polarity Contrastive Learning for Conversational Multimodal Emotion Recognition. Our approach alternates between modeling temporal cross-modal dependencies and constructing a hybrid graph with semantically driven connections. The graph weights are dynamically determined by cosine similarity between node features. Sentiment Polarity Contrastive Learning enhances the model’s ability to learn discriminative emotional features. Experimental results show that our method achieves an IEMOCAP-6 accuracy of 71.60% and F1-score of 71.71%, and MELD accuracy of 67.13% and F1-score of 65.90%.