Colorectal cancer (CRC) remains a leading cause of cancer-related mortality worldwide, with early polyp detection playing a crucial role in improving patient outcomes. While deep-learning-based approaches have shown promise in polyp segmentation, existing Convolutional Neural Networks (CNNs) struggle with long-range dependencies, and Transformer-based models incur high computational costs. The Segment Anything Model (SAM) has emerged as a powerful segmentation tool but suffers from domain gaps when applied to medical imaging, leading to miscalibrated confidence scores and reduced accuracy in polyp segmentation. To address these challenges, we propose Frequency-Sensitive Polyp Segmentation (FSPS), an adaptation of SAM that enhances feature extraction through frequency-domain analysis. FSPS introduces a Spectral Filtering Module (SFM), leveraging the Fourier Transform to capture high-frequency details without increasing encoder complexity. Additionally, our Uncertainty-Guided Prediction Regularization (UPR) module refines SAM’s confidence estimation, improving robustness in out-of-distribution (OOD) cases. Unlike prior approaches, FSPS eliminates the need for prompt engineering, enabling fully automated segmentation. Experiments on standard polyp segmentation benchmarks demonstrate that FSPS achieves a Dice score of 0.895 on CVC-ClinicDB, offering competitive performance among SAM-based methods while maintaining a lightweight architecture. Although it does not surpass dual-encoder approaches, FSPS presents a promising direction for improving SAM’s adaptability to medical imaging.