In this work, we propose a multimodal affective communication framework to enhance human-robot interaction (HRI) by dynamically assessing user engagement through biophysiological responses. The system fuses facial expression analysis, speech emotion recognition, and text sentiment analysis to compute an engagement score, which informs adaptive robot behavior. Each modality is processed independently and contributes to a composite score categorized into low, medium, or high engagement levels. A key contribution of this study is a Bayesian-based engagement estimation mechanism, where the weight of each modality is dynamically adjusted based on its reliability, quantified via normalized inverse entropy. This probabilistic approach enables the robot to prioritize the most stable and informative signals, enhancing robustness in real-world environments. Additionally, the robot uses a Bayesian strategy to optimize predefined behavioral stimuli, learning over time which responses most effectively promote user engagement. The proposed framework supports personalized and context-aware real-time adaptation, with promising applications in assistive robotics, socially interactive systems, and long-term HRI.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal Framework for Adaptive HRI via Dynamic Engagement and Affective Feedback

  • Gayathri Girijadevi Radhakrishnan,
  • Olga Tveretina,
  • Farshid Amirabdollahian,
  • Diego Resende Faria

摘要

In this work, we propose a multimodal affective communication framework to enhance human-robot interaction (HRI) by dynamically assessing user engagement through biophysiological responses. The system fuses facial expression analysis, speech emotion recognition, and text sentiment analysis to compute an engagement score, which informs adaptive robot behavior. Each modality is processed independently and contributes to a composite score categorized into low, medium, or high engagement levels. A key contribution of this study is a Bayesian-based engagement estimation mechanism, where the weight of each modality is dynamically adjusted based on its reliability, quantified via normalized inverse entropy. This probabilistic approach enables the robot to prioritize the most stable and informative signals, enhancing robustness in real-world environments. Additionally, the robot uses a Bayesian strategy to optimize predefined behavioral stimuli, learning over time which responses most effectively promote user engagement. The proposed framework supports personalized and context-aware real-time adaptation, with promising applications in assistive robotics, socially interactive systems, and long-term HRI.