Crack segmentation is essential for preserving the structural integrity and extending the service life of infrastructure. Although CNN and Transformer-based models have shown promising results, they still struggle to capture long-range dependencies and accurately model fine edge details needed for precise pixel-level segmentation. To address these issues, we propose the Spatial Frequency and Edge Guided (SFEG) network, a lightweight yet effective end-to-end segmentation model tailored for crack segmentation under low-resolution and low-resource constraints. The SFEG network incorporates a backbone composed of the Spatial Fourier Parallel Mixer (SFPM) and Prior Feed Gate Mixer (PFGM) to jointly enhance global and local feature representations by leveraging Fourier transforms and gradient priors. To facilitate structural boundary extraction, we propose a Dynamic Edge Aware Module (DEAM) that adaptively redistributes multi-dimensional features based on edge relevance and spatial context. Furthermore, the Edge Guidance Fusion Modules (EGFM) and Cross-scale Interaction Fusion Module (CIFM) are introduced to reinforce multi-level feature integration under boundary guidance. Extensive experiments conducted on multiple benchmark datasets demonstrate that the proposed SFEG network achieves superior performance compared to existing CNN- and Transformer-based approaches, offering a better balance between segmentation accuracy and computational efficiency. Our method shows strong generalization ability across diverse crack types and backgrounds, highlighting its potential for practical deployment in real-world structural inspection tasks. The code for SFEG is publicly available at https://github.com/123456789hhx/SFEG