In medical imaging, low-dose CT (LDCT) is favored over normal-dose CT (NDCT) to reduce radiation risks, but LDCT suffers from increased noise, prompting the need for advanced denoising techniques. Traditional iterative reconstruction algorithms face challenges in balancing noise reduction and preserving fine details, whereas recent CNN or transformer based methods learn mappings from noisy to clean images incorporating encoder-decoder architectures with skip-connections. Conventional skip connections with vanilla convolutions lead to loss of semantic details in deeper layers, while transformer-based methods may struggle to focus on patch edges and exhibit limited efficacy capturing local representations precisely with limited data. To establish the balance between feature retention and noise reduction we introduce SEANet, a novel architecture integrating spectral residual units (SRUs) with encoder-decoder frameworks, synergizing complementary spatial and spectral spaces. In SEANet, SRUs guide the corresponding decoder blocks with synergistic local-to-global features via skip-connections. SRUs with encoding dilation modules (DMs) precisely separate noise from relevant features by distinct frequency bands in spectral space and attends patch-edges well. SEANet eliminates drawback of spatial pooling and felicitates precise denoising for low contrast regions in LDCT images. Extensive experiments and interpretable spectral maps demonstrate SEANet’s superiority over state-of-the-art methods like RED-CNN and CTFormer with a margin of 1.92 in PSNR and 2.75 in SSIM. Find the code for SEANet at https://github.com/aj-das-research/SEANet .