Scholarlink

<p>Multimodal Emotion Recognition in Conversations (MERC) is a crucial task for understanding human interactions, where multimodal approaches integrating language, facial expressions, and vocal tone have led to significant advancements. However, a persistent challenge in multimodal architectures, including MERC, is modality misalignment and imbalanced learning. These issues often hinder models from effectively utilizing multimodal information, leading to suboptimal performance despite the availability of multiple modalities. To address this, we design a framework for MERC with a proposed plug-and-play module that builds upon Self-Paced Curriculum Learning (<b>SPCL</b>). As in Curriculum Learning (CL), an effective <i>Difficulty Measurer</i> is essential for structuring a meaningful <i>Learning Scheduler</i>. In this work, we propose a dual-level Difficulty Measurer tailored for MERC, addressing both intra- and inter-conversational dynamics. Unlike conventional approaches that assess difficulty only at the utterance level, our dual-level design incorporates a conversation-level difficulty score. The utterance-level score captures fine-grained modality-specific challenges, while the conversation-level score models broader dialogue structures, including emotional dependencies and modality coherence within the conversation. This holistic evaluation enables our Learning Scheduler to dynamically guide training from easier to more challenging instances. By integrating SPCL into existing MERC architectures, our method effectively mitigates modality imbalance and enhances model robustness. Extensive experiments on the IEMOCAP and MELD datasets confirm consistent improvements: on IEMOCAP, SPCL achieves gains ranging from approximately +1.2% to +6.6% in weighted F1-score over baseline models across different architectures and modality settings, while on MELD, it delivers even more pronounced improvements, with gains reaching up to +10.4% over baseline models. These gains underscore the practical value of SPCL for real-world MERC applications, as it substantially improves emotion recognition accuracy while maintaining compatibility as a plug-and-play module across diverse model architectures.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Leveraging self-paced curriculum learning for enhanced modality balance in multimodal conversational emotion recognition

  • 期刊论文

  • 发表日期：

    2026年6月2日

查看全文
  • Phuong-Anh Nguyen,
  • The-Son Le,
  • Duc-Trong Le,
  • Cam-Van Thi Nguyen

摘要

Multimodal Emotion Recognition in Conversations (MERC) is a crucial task for understanding human interactions, where multimodal approaches integrating language, facial expressions, and vocal tone have led to significant advancements. However, a persistent challenge in multimodal architectures, including MERC, is modality misalignment and imbalanced learning. These issues often hinder models from effectively utilizing multimodal information, leading to suboptimal performance despite the availability of multiple modalities. To address this, we design a framework for MERC with a proposed plug-and-play module that builds upon Self-Paced Curriculum Learning (SPCL). As in Curriculum Learning (CL), an effective Difficulty Measurer is essential for structuring a meaningful Learning Scheduler. In this work, we propose a dual-level Difficulty Measurer tailored for MERC, addressing both intra- and inter-conversational dynamics. Unlike conventional approaches that assess difficulty only at the utterance level, our dual-level design incorporates a conversation-level difficulty score. The utterance-level score captures fine-grained modality-specific challenges, while the conversation-level score models broader dialogue structures, including emotional dependencies and modality coherence within the conversation. This holistic evaluation enables our Learning Scheduler to dynamically guide training from easier to more challenging instances. By integrating SPCL into existing MERC architectures, our method effectively mitigates modality imbalance and enhances model robustness. Extensive experiments on the IEMOCAP and MELD datasets confirm consistent improvements: on IEMOCAP, SPCL achieves gains ranging from approximately +1.2% to +6.6% in weighted F1-score over baseline models across different architectures and modality settings, while on MELD, it delivers even more pronounced improvements, with gains reaching up to +10.4% over baseline models. These gains underscore the practical value of SPCL for real-world MERC applications, as it substantially improves emotion recognition accuracy while maintaining compatibility as a plug-and-play module across diverse model architectures.