Objectives <p>To develop and evaluate a deep learning model for segmentation of the coronary artery vessels and coronary plaques in coronary computed tomography angiography (CCTA).</p> Materials and methods <p>CCTA image data from the Swedish CardioPulmonary BioImage Study (SCAPIS) was used for model development (<i>n</i> = 463 subjects) and testing (<i>n</i> = 123) and for an interobserver study (<i>n</i> = 65). A dataset from Linköping University Hospital (<i>n</i> = 28) was used for external validation. The model’s ability to detect coronary artery disease (CAD) was tested in a separate SCAPIS dataset (<i>n</i> = 684). A deep ensemble (k = 6) of a customized 3D vision transformer model was used for voxelwise classification. The Dice coefficient, the average surface distance, Pearson’s correlation coefficient, analysis of segmented volumes by intraclass correlation coefficient (ICC), and agreement (sensitivity and specificity) were used to analyze model performance.</p> Results <p>PlaqueViT segmented coronary plaques with a Dice coefficient = 0.55, an average surface distance = 0.98 mm and ICC = 0.93 versus an expert reader. In the interobserver study, PlaqueViT performed as well as the expert reader (Dice coefficient = 0.51 and 0.50, average surface distance = 1.31 and 1.15 mm, ICC = 0.97 and 0.98, respectively). PlaqueViT achieved 88% agreement (sensitivity 97%, specificity 76%) in detecting any coronary plaque in the test dataset (<i>n</i> = 123) and 89% agreement (sensitivity 95%, specificity 83%) in the CAD detection dataset (<i>n</i> = 684).</p> Conclusion <p>We developed a deep learning model for fully automatic plaque detection and segmentation that identifies and delineates coronary plaques and the arterial lumen with similar performance as an experienced reader.</p> Key Points <p><Emphasis Type="BoldItalic">Question</Emphasis> <i>A tool for fully automatic and voxelwise segmentation of coronary plaques in coronary CTA (CCTA) is important for both clinical and research usage of the CCTA examination.</i></p> <p><Emphasis Type="BoldItalic">Findings</Emphasis> <i>Segmentation of coronary artery plaques by PlaqueViT was comparable to an expert reader’s performance.</i></p> <p><Emphasis Type="BoldItalic">Clinical relevance</Emphasis> <i>This novel, fully automatic deep learning model for voxelwise segmentation of coronary plaques in CCTA is highly relevant for large population studies such as the Swedish CardioPulmonary BioImage Study.</i></p> Graphical Abstract <p></p>