Automatic 3D Object Segmentation and Reconstruction from HoloLens 2 Data for Mixed Reality in Smart Environments
摘要
Reconstructing real-world objects in three dimensions from captured images is a key challenge in the development of intelligent systems for Mixed Reality (MR) environments. Towards the goal of using MR to assist semi-autonomous people living in smart homes, our team has worked on a prototype pipeline that enables automatic object detection, segmentation, and 3D reconstruction from image and depth data collected using a HoloLens 2 headset. The proof of concept involves an application that records image sequences of the environment, which are then transmitted to a remote server for automated processing. This process consists of two core stages: object segmentation in 2D images, followed by the generation of 3D meshes using depth information and spatial clustering. In this paper, we present the results of three segmentation methods: a fast object detection model, a mask-based region proposal approach, and a recent foundation model designed to segment arbitrary objects without task-specific training. Moreover, we tested object reconstruction using OccupancyNet for the Marching Cubes algorithm. Although there are still many challenges ahead, this prototype demonstrates the potential of using MR for smart environments and shows good results.