Toward Context Awareness in Mixed Reality: Supporting Spatial Reasoning with Object Detection and Ontologies
摘要
This paper presents the results of an early work on different concepts and technologies that can be used to recognize the context in Mixed Reality delivered by means of a head-mounted display through spatial reasoning. We focused on reasoning about the detected relationships between physical objects in a mixed reality scene through a head-mounted display (e.g., the Microsoft Hololens 2). We propose a solution implemented through a distributed architecture, where an Artificial Intelligence (AI) server is connected to a Mixed Reality head-mounted display. This server hosts a hybrid reasoning approach with two modules: an object detection module and a knowledge-based module built using ontologies. The first module identifies objects in the video stream from the head-mounted display. Another module deployed on that device then localizes the objects in a 3D coordinate system and sends the result back to the ontological knowledge-based module. This module performs an inference on the object locations for spatial reasoning. Finally, we have performed an evaluation of this solution in a prototype that provides users with details about spatial relationships between geometric shapes in the Mixed Reality scene.