Accurate polyp segmentation is crucial for early colorectal cancer screening, as timely detection and intervention significantly improve patient survival rates. While the Segment Anything Model (SAM) and its successor SAM2 have demonstrated remarkable capabilities in medical image segmentation, polyp segmentation remains challenging due to substantial variations in lesion size, complex morphological characteristics, and ambiguous boundaries—limitations that persist even in SAM2. To address these challenges, we propose WES2P (Wavelet-Enhanced SAM2 for Polyp segmentation), a novel framework that adapts SAM2 through two key innovations: the Ladder Wavelet-CNN Adapter (LWCA) and the Spatial-Wavelet Detail Enhancement Module (SWDEM). The LWCA employs a parameter-efficient fine-tuning strategy by integrating a lightweight wavelet CNN branch into the adapter architecture, effectively augmenting the encoder with essential local high-frequency information. Complementarily, SWDEM exploits boundary and texture details from low-level semantic features—typically overlooked by existing approaches—to achieve precise polyp boundary delineation. Extensive experiments on five benchmark datasets demonstrate that WES2P achieves state-of-the-art performance, with significant improvements in both Dice and IoU metrics, effectively addressing the limitations of current methods. The code is publicly available at https://anonymous.4open.science/r/WES2P-0420 .