Conversational Multi-modal Emotion Recognition Based on Heterogeneous Dialogue Graph and Sentiment Polarity Contrastive Learning
摘要
Effective modeling of complex dependencies between utterances is crucial for accurate sentiment recognition in multi-modal dialogue systems. Conventional graph neural network methods rely on static fully connected graphs, where nodes are influenced by both intra-modal and inter-modal information, leading to conflicts during data fusion. Additionally, static graph structures based on fixed time windows lack the flexibility to capture emotional dynamics. To address these challenges, we propose Dynamic Graph Fusion and Sentiment Polarity Contrastive Learning for Conversational Multimodal Emotion Recognition. Our approach alternates between modeling temporal cross-modal dependencies and constructing a hybrid graph with semantically driven connections. The graph weights are dynamically determined by cosine similarity between node features. Sentiment Polarity Contrastive Learning enhances the model’s ability to learn discriminative emotional features. Experimental results show that our method achieves an IEMOCAP-6 accuracy of 71.60% and F1-score of 71.71%, and MELD accuracy of 67.13% and F1-score of 65.90%.