A Two-Stage Approach for Multimodal Emotion Cause Pair Extraction that is Based on Large Language Models
摘要
Emotions play a crucial role in human communication; however, past research in emotion analysis has focused primarily on the text modality, neglecting the importance of audio and visual modalities in emotion recognition. To address this issue, we propose an innovative multimodal emotion‒cause pair net (MECPN) framework that captures emotions and their causes in conversations more comprehensively by integrating information from text, audio, and visual modalities, thereby achieving more accurate emotion analysis. The MECPN framework consists of two key modules: the multimodal emotion recognition (MER) module and the multimodal cause extraction (MCE) module. The MER module uses RoBERTa, OpenSMILE, and VisExtNet to extract text, audio, and visual features, effectively integrating multimodal information through the MultiAttn multimodal fusion network. The MCE module uses the MiniGPT-v2 model to integrate visual and textual information, generate natural language responses, and extract emotions that cause utterances through similarity matching. The experimental results on the ECF2.0 dataset show that the F1 score of the MECPN framework in the emotional cause extraction task is 0.2692, and the weighted average F1 score is 0.2574. Ablation experiments confirmed the importance of the VisExtNet, DialogueRNN, and MultiAttn modules in the model, demonstrating the effectiveness of these components in processing multimodal information and capturing the dialog context.