This study investigates whether incorporating multimodal emotional input (self-reported emotion and facial emotion) can enhance users’ emotional experiences in interactions with LLM-based chatbots. Twenty-eight participants engaged with four chatbot conditions: (A) text-only, (B) self-reported emotion, (C) facial emotion, and (D) combined input. Despite the hypothesis, perceived empathy was slightly higher in conditions without facial emotion input. However, participants with lower trait emotional expressivity reported more positive affect when facial emotion cues were included. Qualitative interviews further revealed varying perceptions of chatbot sensitivity to emotional needs. Participants with prior chatbot experience also felt a stronger social connection when conversations began with emotional topics. Additionally, older adults, women, and experienced users rated their interactions more positively. These findings suggest that emotional input may not universally enhance affective experiences but could benefit specific user profiles. The study advocates for a personalized, trait-sensitive approach to emotion-aware chatbot design rather than a one-size-fits-all multimodal strategy.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Impact of Multimodal Emotional Input on User Experience in Chatbots

  • Yifei Li,
  • Hunter Fong,
  • Maurizio Mancini,
  • Radoslaw Niewiadomski

摘要

This study investigates whether incorporating multimodal emotional input (self-reported emotion and facial emotion) can enhance users’ emotional experiences in interactions with LLM-based chatbots. Twenty-eight participants engaged with four chatbot conditions: (A) text-only, (B) self-reported emotion, (C) facial emotion, and (D) combined input. Despite the hypothesis, perceived empathy was slightly higher in conditions without facial emotion input. However, participants with lower trait emotional expressivity reported more positive affect when facial emotion cues were included. Qualitative interviews further revealed varying perceptions of chatbot sensitivity to emotional needs. Participants with prior chatbot experience also felt a stronger social connection when conversations began with emotional topics. Additionally, older adults, women, and experienced users rated their interactions more positively. These findings suggest that emotional input may not universally enhance affective experiences but could benefit specific user profiles. The study advocates for a personalized, trait-sensitive approach to emotion-aware chatbot design rather than a one-size-fits-all multimodal strategy.