Training in industrial environments is often expensive and time-consuming. While self-learning approaches, customized curricula, and automation offer promising solutions, the creation of new training materials remains a major bottleneck—especially in rapidly evolving industries where processes change frequently. Mixed reality (MR) and virtual reality (VR) technologies introduce a new generation of immersive learning platforms, but their widespread adoption is limited by the expertise and effort required to generate content. In this paper, we present a novel AI-powered system designed to simplify and accelerate the development of MR-based training applications. Using a HoloLens 2 headset, an expert needs to demonstrate and explain a complex task only once. Our system captures this demonstration, interprets the instructions using various AI techniques, and automatically produces a step-by-step tutorial. The resulting guide includes time-aligned video snippets and descriptive text, allowing trainees to visually follow each subtask. This approach significantly reduces the cost and complexity of training content creation, opening the door to scalable and efficient instructorless learning. The system was evaluated in terms of usability, user experience, and the accuracy of the generated content, the results of which confirm the correct functioning and applicability of our approach.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

AI-Driven Mixed Reality for Scalable and Adaptive Training Solutions

  • Bálint György Nagy,
  • Bence Bihari,
  • Balázs Sonkoly

摘要

Training in industrial environments is often expensive and time-consuming. While self-learning approaches, customized curricula, and automation offer promising solutions, the creation of new training materials remains a major bottleneck—especially in rapidly evolving industries where processes change frequently. Mixed reality (MR) and virtual reality (VR) technologies introduce a new generation of immersive learning platforms, but their widespread adoption is limited by the expertise and effort required to generate content. In this paper, we present a novel AI-powered system designed to simplify and accelerate the development of MR-based training applications. Using a HoloLens 2 headset, an expert needs to demonstrate and explain a complex task only once. Our system captures this demonstration, interprets the instructions using various AI techniques, and automatically produces a step-by-step tutorial. The resulting guide includes time-aligned video snippets and descriptive text, allowing trainees to visually follow each subtask. This approach significantly reduces the cost and complexity of training content creation, opening the door to scalable and efficient instructorless learning. The system was evaluated in terms of usability, user experience, and the accuracy of the generated content, the results of which confirm the correct functioning and applicability of our approach.