Background <p>Curative treatment of resectable esophageal cancer comprises neoadjuvant chemoradiotherapy and esophagectomy. Robot-assisted minimally invasive esophagectomy (RAMIE) is the preferred technique; however, learning RAMIE is challenging due to the complex chest anatomy, patient positioning and zoomed-in camera view. Computer-aided anatomy recognition holds promise for improving surgical navigation. This study aims to develop real-time anatomy recognition and surgical phase recognition algorithms for the thoracic part of RAMIE using deep learning and to understand the challenges of current state-of-the-art algorithms.</p> Methods <p>A retrospective single-center cohort study was conducted on prospectively collected RAMIE videos at University Medical Center Utrecht, The Netherlands. Two datasets were created: an anatomy segmentation dataset with 1504 frames from 53 videos, annotated for eight anatomical classes and four surgical instruments; and a surgical phase dataset, with 38 videos labeled with thirteen distinct phases. Several deep learning models were trained and tested using both datasets.</p> Results <p>The SegNeXt model achieved the most accurate segmentations with an overall overlap score of 0.72 for all classes, of which the surgical instruments&#xa0;were best detectable. The best surgical phase recognition model achieved an overall accuracy of 82.8% and revealed loss of accuracy during phase transitions. Class imbalance affected both datasets which led to less frequent appearing classes performing lowest.</p> Conclusions <p>This study identified suitable models for computer-aided, real-time anatomy and surgical tool segmentation and surgical phase recognition for the thoracic part of RAMIE. Acceptable performance was achieved while both datasets are highly complex. Performance is expected to improve by further expanding and diversifying the datasets.</p> Graphical abstract <p></p>