In recent years, semantic segmentation of remote sensing images has shown great potential in land-use monitoring, urban planning, and disaster assessment. Yet, current models often suffer from high complexity due to self-attention, limited interaction between spatial and frequency features, and underutilization of frequency information. To address these issues, we propose VSFNet, a novel segmentation network that integrates a Visual State Space (VSS) encoder with adaptive frequency-domain filtering. VSFNet employs VSS blocks to extract multi-scale features efficiently, overcoming the restricted receptive fields of CNNs and the computational burden of Transformers. An Adaptive Frequency-Filtered Block (AFF Block) is designed to progressively fuse spatial and frequency representations, balancing global context with detail recovery. Furthermore, a Two-Level Multi-Scale Fusion Pyramid (TLMFP) with VSS blocks and Deformable Spatial Attention (DSAN) enhances multi-scale feature interaction and fine-grained reconstruction. Experiments on LoveDA and Potsdam demonstrate that VSFNet achieves mIoU scores of 54.95% and 83.73%, significantly outperforming mainstream methods such as DeepLabV3+ (CNN), UNetformer (Transformer), and UNetMamba (Mamba).