The most exciting frontiers of contemporary endoscopic video processing include tissue and instrument tracking and scene reconstruction together with their applications in surgical navigation, mixed reality visualization, and surgical automation. These tasks are enabled by task-specific techniques such as segmentation, object tracking, structure from motion, simultaneous localization and mapping (SfM and SLAM), and neural rendering. While these techniques vary in purpose and methodology, fundamentally, they rely on the ability to reliably and robustly establish point correspondences across video frames. However, point tracking in endoscopic scenes is hard due to the lack of distinct yet repetitive features, varying illuminations, and continuously changing visual appearance of corresponding points. A dense point-tracking model capable of reliably establishing point correspondences across video frames could catalyze endoscopic video processing and its downstream applications, and drive significant advancement in surgical data science. While any point-tracking foundation models have recently been proposed, they are trained on large simulated and natural scene videos and need to be adapted to endoscopic scenes to address domain-specific challenges. In this work, we present a dense point tracking foundation model for endoscopic scenes by fine-tuning a public foundation model on a large custom dataset comprising 13k endoscopic video sequences (314k frames). We present three benchmark datasets with ground-truth point correspondences to quantitatively evaluate the point tracking performance in variable endoscopic scenes. Through quantitative analysis, we find that models fine-tuned on endoscopic scenes outperform out-of-the-box models, especially when considering conservative thresholds for tracking success, suggesting improved suitability for downstream tasks due to error propagation.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

EndoTracker: Robustly Tracking Any Point in Endoscopic Surgical Scene

  • Lalithkumar Seenivasan,
  • Joanna Cheng,
  • Jin Fang,
  • Jin Bai,
  • Roger D. Soberanis-Mukul,
  • Jan Emily Mangulabnan,
  • S. Swaroop Vedula,
  • Masaru Ishii,
  • Gregory Hager,
  • Russell H. Taylor,
  • Mathias Unberath

摘要

The most exciting frontiers of contemporary endoscopic video processing include tissue and instrument tracking and scene reconstruction together with their applications in surgical navigation, mixed reality visualization, and surgical automation. These tasks are enabled by task-specific techniques such as segmentation, object tracking, structure from motion, simultaneous localization and mapping (SfM and SLAM), and neural rendering. While these techniques vary in purpose and methodology, fundamentally, they rely on the ability to reliably and robustly establish point correspondences across video frames. However, point tracking in endoscopic scenes is hard due to the lack of distinct yet repetitive features, varying illuminations, and continuously changing visual appearance of corresponding points. A dense point-tracking model capable of reliably establishing point correspondences across video frames could catalyze endoscopic video processing and its downstream applications, and drive significant advancement in surgical data science. While any point-tracking foundation models have recently been proposed, they are trained on large simulated and natural scene videos and need to be adapted to endoscopic scenes to address domain-specific challenges. In this work, we present a dense point tracking foundation model for endoscopic scenes by fine-tuning a public foundation model on a large custom dataset comprising 13k endoscopic video sequences (314k frames). We present three benchmark datasets with ground-truth point correspondences to quantitatively evaluate the point tracking performance in variable endoscopic scenes. Through quantitative analysis, we find that models fine-tuned on endoscopic scenes outperform out-of-the-box models, especially when considering conservative thresholds for tracking success, suggesting improved suitability for downstream tasks due to error propagation.