Purpose Augmented reality guidance in microsurgery is challenged by depth ambiguity, occlusion, and limited situational awareness under the operating microscope. Digital twins, dynamic virtual models of physical systems, can provide the contextual geometric data needed to mitigate these limitations. We introduce a perception-first digital twin framework that uses real-time surgical state to drive depth-, occlusion-, and proximity-aware augmented reality visualization for guidance in otologic surgery. Methods We formulate a hand–eye calibration refinement as a joint <InlineEquation ID="IEq1"> <EquationSource Format="TEX">$\textrm{SE}(3)$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mtext>SE</mtext> <mo stretchy="false">(</mo> <mn>3</mn> <mo stretchy="false">)</mo> </mrow> </math></EquationSource> </InlineEquation> optimization with a bias term and robust weighting to ensure accurate alignment between physical and virtual entities. Our framework integrates this optimized calibration with high-precision optical tracking of key surgical elements and real-time simulation, enabling dynamic updates of the digital twin from the evolving surgical state. In turn, the updated virtual replica is used to compute perceptual fields that drive the augmented reality overlays on the live stereoscopic microscope feed of a 3D-printed anatomical model. Results We evaluated the accuracy of the optimized calibration on both calibration and evaluation datasets. The optimized workflow achieved a median translational error of <InlineEquation ID="IEq2"> <EquationSource Format="TEX">$1.06\,\textrm{mm}$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>1.06</mn> <mspace width="0.166667em" /> <mtext>mm</mtext> </mrow> </math></EquationSource> </InlineEquation> and <InlineEquation ID="IEq3"> <EquationSource Format="TEX">$1.19\,\textrm{mm}$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>1.19</mn> <mspace width="0.166667em" /> <mtext>mm</mtext> </mrow> </math></EquationSource> </InlineEquation> and a median rotational error of <InlineEquation ID="IEq4"> <EquationSource Format="TEX">$0.46^{\circ }$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>0</mn> <mo>.</mo> <msup> <mn>46</mn> <mo>∘</mo> </msup> </mrow> </math></EquationSource> </InlineEquation> and <InlineEquation ID="IEq5"> <EquationSource Format="TEX">$0.28^{\circ }$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>0</mn> <mo>.</mo> <msup> <mn>28</mn> <mo>∘</mo> </msup> </mrow> </math></EquationSource> </InlineEquation>, respectively. In addition, the digital twin was used to derive and overlay perceptual cues onto the microscope video of a 3D-printed temporal bone. An AR-guided microsurgical use case demonstrated how the dynamically updated cues convey spatial relationships between critical structures and the evolving surgical cavity. Conclusion We present a perception-first digital twin framework for augmented reality guidance in microsurgery. As surgical alteration of anatomy progresses, the digital twin is continuously updated and used to drive perception-aware augmented reality cues. Our framework enables enhanced spatial understanding while preserving visibility of the operative field.

Perception-first digital twin for augmented reality microsurgery

期刊论文
发表日期：

2026年5月9日

查看全文

Trishia El Chemaly,
Yunxin Fan,
Fanrui Fu,
Christoph Leuze,
Bruce Daniel,
Brian Hargreaves,
Nikolas H. Blevins

摘要

Purpose

Augmented reality guidance in microsurgery is challenged by depth ambiguity, occlusion, and limited situational awareness under the operating microscope. Digital twins, dynamic virtual models of physical systems, can provide the contextual geometric data needed to mitigate these limitations. We introduce a perception-first digital twin framework that uses real-time surgical state to drive depth-, occlusion-, and proximity-aware augmented reality visualization for guidance in otologic surgery.

Methods

We formulate a hand–eye calibration refinement as a joint $\textrm{SE}(3)$ $SE (3)$ optimization with a bias term and robust weighting to ensure accurate alignment between physical and virtual entities. Our framework integrates this optimized calibration with high-precision optical tracking of key surgical elements and real-time simulation, enabling dynamic updates of the digital twin from the evolving surgical state. In turn, the updated virtual replica is used to compute perceptual fields that drive the augmented reality overlays on the live stereoscopic microscope feed of a 3D-printed anatomical model.

Results

We evaluated the accuracy of the optimized calibration on both calibration and evaluation datasets. The optimized workflow achieved a median translational error of $1.06\,\textrm{mm}$ $1.06 mm$ and $1.19\,\textrm{mm}$ $1.19 mm$ and a median rotational error of $0.46^{\circ }$ $0 . 46^{\circ}$ and $0.28^{\circ }$ $0 . 28^{\circ}$ , respectively. In addition, the digital twin was used to derive and overlay perceptual cues onto the microscope video of a 3D-printed temporal bone. An AR-guided microsurgical use case demonstrated how the dynamically updated cues convey spatial relationships between critical structures and the evolving surgical cavity.

Conclusion

We present a perception-first digital twin framework for augmented reality guidance in microsurgery. As surgical alteration of anatomy progresses, the digital twin is continuously updated and used to drive perception-aware augmented reality cues. Our framework enables enhanced spatial understanding while preserving visibility of the operative field.