<p>Shadows in natural images degrade visual quality and disrupt feature representations, which in turn affects the reliability of downstream computer vision tasks. Although recent deep learning methods have achieved notable progress in shadow removal, many approaches still formulate the problem as a direct pixel-to-pixel transformation. As a result, semantic structure is often underutilized, and global illumination consistency is insufficiently constrained. These limitations commonly lead to texture degradation in complex shadow regions and noticeable color shifts at the image level. In this work, we present SFDiff, a diffusion-based framework designed to jointly exploit semantic cues and frequency-domain information for shadow removal. The proposed Semantic-Guided Shadow Restoration (SGSR) module introduces a learnable prototype bank to cluster semantic features in an unsupervised manner. By retrieving representative texture prototypes from non-shadow regions, SGSR provides semantic-level guidance for restoring shadow-degraded features. In addition, we develop a Frequency-Aware Feature Refinement (FAFR) module that operates in the frequency domain to correct global illumination inconsistencies. FAFR adjusts frequency components through a mask-guided gating strategy, enabling color correction while maintaining background fidelity. Extensive experiments on benchmark datasets show that the proposed method consistently improves shadow removal performance in both visual quality and quantitative evaluation.</p>