In medical clinics, the automatic segmentation of polyp images is crucial for assisting in the clinical diagnosis of colorectal cancer. However, existing polyp segmentation methods still encounter challenges in distinguishing polyps from surrounding normal tissue regions. Addressing this challenge, we propose an innovative Distraction Mining Network (DMNet) tailored for polyp segmentation. It extracts the multi-scale features in the image using the Pyramid Vision Transformer. Specifically, it also comprises two key modules: the Location Module (LM) and the Focus Module (FM). The LM is employed to identify potential polyp regions globally in the high-level features, while the FM progressively refines the prediction results by concentrating on the target objects in the lower three levels of features. Furthermore, the FM accomplishes the detection and removal of distracting regions through a distraction mining strategy, thereby enhancing the model’s segmentation performance. Experiments have demonstrated that DMNet yields superior segmentation results across five diverse polyp image datasets, underscoring its potential to bolster clinical diagnostic precision.