Transformer has become a research hotspot in the field of image processing, particularly due to the strong capability of the self-attention mechanism in capturing global contextual information. However, self-attention has limited sensitivity to local details and often struggles to capture critical local structures in images, resulting in insufficiently rich feature representations. Moreover, although some existing lightweight self-attention variants effectively reduce computational cost, they often suffer from significant information loss during compression, thereby affecting the model’s expressive power and final performance. To address these issues, this paper proposes a novel lightweight frequency-spatial domain visual Transformer architecture, named WSFFormer, which aims to enhance image understanding by modeling contextual interactions through wavelet frequency-domain transformation and self-attention mechanisms. WSFFormer employs a wavelet frequency domain grouped convolution module to efficiently capture frequency-domain features at a low computational cost, while using convolution operations to emphasize important feature regions. Additionally, the architecture incorporates a spectral center attention mechanism that performs lossless downsampling and fully leverages the modeling capabilities of traditional self-attention, effectively reducing complexity while maintaining the ability to selectively model critical image regions. Experimental results demonstrate that WSFFormer achieves superior trade-offs between computational cost and performance compared to conventional CNN and Transformer models in image processing tasks.