The precise segmentation of colonic polyps is crucial in assisting clinicians with the detection of polyps and the diagnosis of colorectal cancer. Despite the progress made by current approaches, several obstacles remain: (1) substantial diversity in the morphology and dimensions of polyps, (2) frequent omission of small polyps during detection, (3) indistinct boundaries separating polyps from adjacent tissues, and (4) various types of noise in colonoscopic images—such as those introduced by inconsistent lighting—that can degrade segmentation performance. To overcome these issues, The SFS module is designed for more accurate polyp localization. The FFCD module fuses dual-branch features to reduce the loss of small polyps while focusing on conflict regions between the two branches to guide segmentation. The FEIS module enhances the recognition of polyps with different sizes and shapes while mitigating noise interference. The IRR module recovers feature granularity, thereby mitigating the problem of boundary blurring. Additionally, we introduce FSISNet-T and FSISNet-M, incorporating Transformer and Mamba as backbones, respectively, to adapt to different scenarios. Experimental results demonstrate that both FSISNet-T and FSISNet-M outperform six state-of-the-art (SOTA) methods employed for comparison. Notably, FSISNet-T surpasses the best SOTA method by 5.39% in mDice on the ETIS dataset, while FSISNet-M exceeds the best SOTA method by 2.74% in mDice on the ColonDB dataset.