This research addresses the problem of an enabling AI systems to become capable of dynamically adapting to the complex, multimodal nature of classroom interactions. We introduce a novel neuro-symbolic AI approach designed to achieve real-time, multi-party situational awareness in educational environments. By integrating verbal, gestural, and physical cues from multiple students, our system constructs a coherent understanding of collaborative learning processes. This involves not only tracking verbal dialogue but also interpreting nonverbal behavior and contextual changes to analyze interaction dynamics. The proposed framework advances the development of AI systems that can effectively support and enhance classroom engagement and learning outcomes.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal Situational Awareness: Neuro-Symbolic AI for Real-Time HCI in the Classroom

  • Yifan Zhu,
  • Kenneth Lai,
  • Ibrahim Khebour,
  • Marc Verhagen,
  • Nikhil Krishnaswamy,
  • James Pustejovsky

摘要

This research addresses the problem of an enabling AI systems to become capable of dynamically adapting to the complex, multimodal nature of classroom interactions. We introduce a novel neuro-symbolic AI approach designed to achieve real-time, multi-party situational awareness in educational environments. By integrating verbal, gestural, and physical cues from multiple students, our system constructs a coherent understanding of collaborative learning processes. This involves not only tracking verbal dialogue but also interpreting nonverbal behavior and contextual changes to analyze interaction dynamics. The proposed framework advances the development of AI systems that can effectively support and enhance classroom engagement and learning outcomes.