Purpose Understanding surgical instrument–tissue interactions requires not only identifying which instrument performs which action on which anatomical target, but also grounding these interactions spatially within the surgical scene. Existing surgical action triplet recognition methods are limited to learning from frame-level classification, failing to reliably link actions to specific instrument instances. Previous attempts at spatial grounding have primarily relied on class activation maps, which lack the precision and robustness required for detailed instrument–tissue interaction analysis. To address this gap, we propose grounding surgical action triplets with instrument instance segmentation, or triplet segmentation for short, a new unified task which produces spatially grounded <InlineEquation ID="IEq1"> <EquationSource Format="TEX">$\langle $</EquationSource> <EquationSource Format="MATHML"><math> <mo stretchy="false">⟨</mo> </math></EquationSource> </InlineEquation> <Emphasis FontCategory="NonProportional">instrument</Emphasis>,  <Emphasis FontCategory="NonProportional">verb</Emphasis>,  <Emphasis FontCategory="NonProportional">target</Emphasis><InlineEquation ID="IEq2"> <EquationSource Format="TEX">$\rangle $</EquationSource> <EquationSource Format="MATHML"><math> <mo stretchy="false">⟩</mo> </math></EquationSource> </InlineEquation>  outputs. Methods We start by presenting CholecTriplet-Seg, a large-scale dataset containing over 30,000 annotated frames, linking instrument instance masks with action verb and anatomical target annotations, and establishing the first benchmark for strongly supervised, instance-level triplet grounding and evaluation. To learn triplet segmentation, we propose TargetFusionNet, a novel architecture that extends Mask2Former with a target-aware fusion mechanism to address the challenge of accurate anatomical target prediction by fusing weak anatomy priors with instrument instance queries. Results Evaluated across recognition, detection, and triplet segmentation metrics, TargetFusionNet consistently improves performance over existing baselines, demonstrating that strong instance supervision combined with weak target priors significantly enhances the accuracy and robustness of surgical action understanding. Conclusion Triplet segmentation establishes a unified framework for spatially grounding surgical action triplets. The proposed CholecTriplet-Seg benchmark and TargetFusionNet architecture pave the way for more interpretable, fine-grained surgical scene understanding.

Grounding surgical action triplets with instrument instance segmentation: a dataset and target-aware fusion approach

期刊论文
发表日期：

2026年3月25日

查看全文

Oluwatosin Alabi,
Meng Wei,
Charlie Budd,
Tom Vercauteren,
Miaojing Shi

摘要

Purpose

Understanding surgical instrument–tissue interactions requires not only identifying which instrument performs which action on which anatomical target, but also grounding these interactions spatially within the surgical scene. Existing surgical action triplet recognition methods are limited to learning from frame-level classification, failing to reliably link actions to specific instrument instances. Previous attempts at spatial grounding have primarily relied on class activation maps, which lack the precision and robustness required for detailed instrument–tissue interaction analysis. To address this gap, we propose grounding surgical action triplets with instrument instance segmentation, or triplet segmentation for short, a new unified task which produces spatially grounded $\langle $ $⟨$ instrument, verb, target $\rangle $ $⟩$ outputs.

Methods

We start by presenting CholecTriplet-Seg, a large-scale dataset containing over 30,000 annotated frames, linking instrument instance masks with action verb and anatomical target annotations, and establishing the first benchmark for strongly supervised, instance-level triplet grounding and evaluation. To learn triplet segmentation, we propose TargetFusionNet, a novel architecture that extends Mask2Former with a target-aware fusion mechanism to address the challenge of accurate anatomical target prediction by fusing weak anatomy priors with instrument instance queries.

Results

Evaluated across recognition, detection, and triplet segmentation metrics, TargetFusionNet consistently improves performance over existing baselines, demonstrating that strong instance supervision combined with weak target priors significantly enhances the accuracy and robustness of surgical action understanding.

Conclusion

Triplet segmentation establishes a unified framework for spatially grounding surgical action triplets. The proposed CholecTriplet-Seg benchmark and TargetFusionNet architecture pave the way for more interpretable, fine-grained surgical scene understanding.