The ability of virtual reality (VR) games to provide immersive experiences is an extremely valuable property, especially in an educational setting for skill training, where players are quite often trained in scenarios that they are required to perform in real life. Assessing of training, including debriefing, however, remains largely a manual task in skill training. Video recording-based assessment is a common technique which is unfortunately data crunching particularly when a large number of students are involved in a lengthy training process. Also, automatic skill assessment for students' learning based on the recorded video is still a research topic. To address this issue, this research is keen to investigate stealth assessment or in-process assessment for VR-enabled skill training. A novel technique focusing on hand motion detection is developed for skill assessment and debriefing during the training process. In particular, tracking and detecting of specific and customizable hand motion are performed in real time for skill-based task training. Plugin scripts are designed to track and detect realistic hand motions into portions of the training game. The trajectory of hand motion for skill training can be used to reconstruct the virtual hand motion  for debriefing as a replay of the player’s hand movement step by step. This is significant because of the tremendous saving of the trajectory compared to video recording of the hand movement in terms of data size. Comparison can also be easily developed for automated assessment between the correct answers and the actual movement performed by the students during their skill training. An experiment was conducted with the students learning the aircraft hydraulic maintenance. They learn through playing a hand motion based game built on top of the plugin scripts developed. The script implemented in this game is set to detect a nut-loosening motion. The survey results demonstrate that most players agree that the hand motion tracking and detection enhance the interactivity and learning experiences of VR-based skill training. This technology has the potential to elevate the learning experience for various educational applications, such as medical, engineering, and entertainment.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Stealth Assessment for VR-based Skill Training Utilizing Hand Motion Detection

  • Siang Jian Phang,
  • Siyi Li,
  • Yuan Xie,
  • Lihui Huang,
  • Yiyu Cai

摘要

The ability of virtual reality (VR) games to provide immersive experiences is an extremely valuable property, especially in an educational setting for skill training, where players are quite often trained in scenarios that they are required to perform in real life. Assessing of training, including debriefing, however, remains largely a manual task in skill training. Video recording-based assessment is a common technique which is unfortunately data crunching particularly when a large number of students are involved in a lengthy training process. Also, automatic skill assessment for students' learning based on the recorded video is still a research topic. To address this issue, this research is keen to investigate stealth assessment or in-process assessment for VR-enabled skill training. A novel technique focusing on hand motion detection is developed for skill assessment and debriefing during the training process. In particular, tracking and detecting of specific and customizable hand motion are performed in real time for skill-based task training. Plugin scripts are designed to track and detect realistic hand motions into portions of the training game. The trajectory of hand motion for skill training can be used to reconstruct the virtual hand motion  for debriefing as a replay of the player’s hand movement step by step. This is significant because of the tremendous saving of the trajectory compared to video recording of the hand movement in terms of data size. Comparison can also be easily developed for automated assessment between the correct answers and the actual movement performed by the students during their skill training. An experiment was conducted with the students learning the aircraft hydraulic maintenance. They learn through playing a hand motion based game built on top of the plugin scripts developed. The script implemented in this game is set to detect a nut-loosening motion. The survey results demonstrate that most players agree that the hand motion tracking and detection enhance the interactivity and learning experiences of VR-based skill training. This technology has the potential to elevate the learning experience for various educational applications, such as medical, engineering, and entertainment.