For early colorectal cancer identification, polyp segmentation in colonoscopy pictures is essential. To detect colorectal polyps, this study assesses the interpretability and accuracy of two advanced segmentation models: Advanced U-Net and Swin U-NetR. To evaluate the effect of Explainable AI on segmentation efficiency, we compare the models’ performance using both conventional training methods and AI-based approaches, specifically Grad-CAM masks. Model performance is measured using metrics like the F1 score, Dice Coefficient, and Intersection over Union (IoU). The complex nature of model performance concerning mask selection strategies is demonstrated by our findings, which also show that models trained using Explainable AI techniques—specifically, Grad-CAM—perform better than models trained using conventional methods. This study addresses the research gap in integrating Explainable AI with traditional segmentation models to enhance interpretability and accuracy in colorectal polyp detection.