Scholarlink

<p>Building effective artificial intelligence (AI) systems for emotional support conversations has become increasingly adopted in mental health care. Effective emotional support conversations require a nuanced understanding of users’ affective states. Although recent studies have incorporated multimodal inputs, human emotions expressed across different modalities may be inconsistent. For example, a user says “I am fine” and displays a polite smile, their affective state is sadness. To address this challenge, we propose MiAgent, a novel Multi-Agent Collaborative Reasoning framework for multimodal emotional support conversations. Our framework employs specialized video, audio, and text agents to independently analyze emotional reasoning from expressive behavior, affective prosody, and linguistic context. Rather than naively aggregating these signals, we introduce an affective reasoning agent that analyzes cross-modal disagreement reasoning. This agent performs cross-modal conflict judging and modality reliability weighting to dynamically attenuate unreliable or contradictory signals to fuse the reasoning from different agents. Building on the fused affective reasoning, a supporter agent is trained via the distillation-based reasoning to generate empathetic and effective responses. Extensive experiments demonstrate that transitioning from simple multimodal fusion to collaborative, disagreement-aware reasoning significantly improves the quality of emotional support generation. Our findings underscore the importance of structured multi-agent reasoning for handling multimodal affective uncertainty and advance the development of more resilient and perceptive conversational AI systems for mental health support.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

From disagreement to insight: multi-agent collaborative reasoning for multimodal emotional support conversations

  • 期刊论文

  • 发表日期：

    2026年6月22日

查看全文
  • Yuqi Chu,
  • Yanrong Guo,
  • Richang Hong

摘要

Building effective artificial intelligence (AI) systems for emotional support conversations has become increasingly adopted in mental health care. Effective emotional support conversations require a nuanced understanding of users’ affective states. Although recent studies have incorporated multimodal inputs, human emotions expressed across different modalities may be inconsistent. For example, a user says “I am fine” and displays a polite smile, their affective state is sadness. To address this challenge, we propose MiAgent, a novel Multi-Agent Collaborative Reasoning framework for multimodal emotional support conversations. Our framework employs specialized video, audio, and text agents to independently analyze emotional reasoning from expressive behavior, affective prosody, and linguistic context. Rather than naively aggregating these signals, we introduce an affective reasoning agent that analyzes cross-modal disagreement reasoning. This agent performs cross-modal conflict judging and modality reliability weighting to dynamically attenuate unreliable or contradictory signals to fuse the reasoning from different agents. Building on the fused affective reasoning, a supporter agent is trained via the distillation-based reasoning to generate empathetic and effective responses. Extensive experiments demonstrate that transitioning from simple multimodal fusion to collaborative, disagreement-aware reasoning significantly improves the quality of emotional support generation. Our findings underscore the importance of structured multi-agent reasoning for handling multimodal affective uncertainty and advance the development of more resilient and perceptive conversational AI systems for mental health support.