Scholarlink

<p>Engagement recognition plays a crucial role in enhancing the adaptability and effectiveness of Human-Robot Interaction (HRI) in instructional settings. However, reliance on visual information alone (e.g., facial expressions) often fails due to user posture changes leading to occlusion, while physiological modeling (e.g., electroencephalography/EEG) faces challenges with limited labeled data. This study aims to develop a robust multimodal framework to assess learner status during robotic assembly tasks. We propose a facial-guided EEG modeling paradigm to supervise physiological signal training. An SVM-based classifier was established and systematically compared against 1D-CNN and Tabular Transformer baselines. Furthermore, a dynamic switching strategy was designed to integrate facial and EEG signals, ensuring continuous monitoring even when visual tracking is lost. Experimental validation in a MOOC scenario, using a multi-source benchmark comprising quiz scores, instructor evaluations, and self-reports, demonstrated that the proposed EEG-SVM model achieved 77.59% consistency rate with instructor ratings and a test accuracy of 84.02%. It was the only method to show a statistically significant correlation with self-reports. The proposed framework effectively mitigates the monitoring challenges in HRI systems caused by visual occlusion. Through the synergistic integration of facial and EEG signals, this framework provides a robust solution for enabling continuous and reliable assessment of learning states in practical engineering education.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A multimodal framework for engagement recognition in HRI teaching scenarios: combining facial features and EEG

  • 期刊论文

  • 发表日期：

    2026年3月10日

查看全文
  • Wei Pang,
  • Jiahui Wang,
  • Zhengxu Li,
  • Nan Xie,
  • Rui Zhang,
  • Beier Lu,
  • Tingqiang Xiong

摘要

Engagement recognition plays a crucial role in enhancing the adaptability and effectiveness of Human-Robot Interaction (HRI) in instructional settings. However, reliance on visual information alone (e.g., facial expressions) often fails due to user posture changes leading to occlusion, while physiological modeling (e.g., electroencephalography/EEG) faces challenges with limited labeled data. This study aims to develop a robust multimodal framework to assess learner status during robotic assembly tasks. We propose a facial-guided EEG modeling paradigm to supervise physiological signal training. An SVM-based classifier was established and systematically compared against 1D-CNN and Tabular Transformer baselines. Furthermore, a dynamic switching strategy was designed to integrate facial and EEG signals, ensuring continuous monitoring even when visual tracking is lost. Experimental validation in a MOOC scenario, using a multi-source benchmark comprising quiz scores, instructor evaluations, and self-reports, demonstrated that the proposed EEG-SVM model achieved 77.59% consistency rate with instructor ratings and a test accuracy of 84.02%. It was the only method to show a statistically significant correlation with self-reports. The proposed framework effectively mitigates the monitoring challenges in HRI systems caused by visual occlusion. Through the synergistic integration of facial and EEG signals, this framework provides a robust solution for enabling continuous and reliable assessment of learning states in practical engineering education.