Abstract: Automated Multimodel Segmentation and Tracking for AR-guided Open Liver Surgery using Scene-aware Self-prompting
摘要
In open liver surgery, augmented reality (AR) guidance requires continuous segmentation and tracking of the liver surface to achieve the necessary alignment between the preoperative model and its corresponding intraoperative state. In this work [1], we present a multimodel approach for semantic segmentation and tracking in ARassisted open liver surgery. Our method combines a domain-specific RGBD model, a foundational semantic segmentation model, and a semi-supervised video object segmentation (VOS) model. The models were integrated into an auto-promptable pipeline, using a scene-aware re-prompting algorithm that adapts to changes in the surgical scene. We tested our method using intraoperative RGBD videos from ten open liver surgeries, collected from a head-mounted AR device. We evaluated the segmentation accuracy (IoU) and temporal resolution (FPS) of our method for different re-prompting strategies, comparing them against the performance of the individual models. Our multimodel approach attained a median IoU of 71% at 13.2 FPS without re-prompting. It outperformed individual models, enhancing segmentation accuracy beyond the stand-alone RGBD model and offering better temporal resolution than the foundation model. With scene-aware reprompting, the multimodel pipeline reaches the upper-bound setting of the semi-supervised VOS model (initialized by an optimal reference frame), achieving 74.7% IoU at 11.5 FPS. Our scene-aware re-prompting strategy offers a balance between segmentation accuracy and temporal resolution, thus addressing the demands of real-time AR-guided open liver surgery.