Mixed reality-guided surgery may benefit from live organ segmentation and tracking, such as adapting virtual models to the deformations of target structures. The use of foundation models allows for widely applicable solutions; however, these models need to be prompted for more demanding tasks, such as surgical applications. This work investigates several user interaction concepts for prompting segmentation foundation models directly in augmented reality (AR) using the Microsoft HoloLens for AR-guided surgical applications. To achieve this, we implement eye tracking, finger tracking, ArUco marker tracking, and pointer tracking concepts and evaluate their accuracy in terms of mean absolute errors during pointing experiments at different distances and with different prompters. Furthermore, we assess their impact on segmentation performance in terms of Dice scores and include an initial user study to query the user confidence and preference. While all methods are accurate enough to perform segmentation in phantom experiments, achieving Dice scores greater than 0.96, ArUco marker tracking and eye tracking prove to be the most accurate throughout our experiments. An initial user questionnaire along with feedback from an experienced visceral surgeon indicates a preference for eye tracking and finger tracking methods. Our code is publicly available at GitHub.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Augmented Reality Prompts for Foundation Model-based Semantic Segmentation

  • Michael Schwimmbeck,
  • Christopher Auer,
  • Johannes Schmidt,
  • Stefanie Remmele

摘要

Mixed reality-guided surgery may benefit from live organ segmentation and tracking, such as adapting virtual models to the deformations of target structures. The use of foundation models allows for widely applicable solutions; however, these models need to be prompted for more demanding tasks, such as surgical applications. This work investigates several user interaction concepts for prompting segmentation foundation models directly in augmented reality (AR) using the Microsoft HoloLens for AR-guided surgical applications. To achieve this, we implement eye tracking, finger tracking, ArUco marker tracking, and pointer tracking concepts and evaluate their accuracy in terms of mean absolute errors during pointing experiments at different distances and with different prompters. Furthermore, we assess their impact on segmentation performance in terms of Dice scores and include an initial user study to query the user confidence and preference. While all methods are accurate enough to perform segmentation in phantom experiments, achieving Dice scores greater than 0.96, ArUco marker tracking and eye tracking prove to be the most accurate throughout our experiments. An initial user questionnaire along with feedback from an experienced visceral surgeon indicates a preference for eye tracking and finger tracking methods. Our code is publicly available at GitHub.