Existing document shadow removal methods primarily rely on Convolutional Neural Networks (CNNs) to capture local features but suffer from limited long-range dependency modeling. While Transformer-based approaches enhance global representation learning, their prohibitive computational overhead hinders deployment in large-scale document processing. Furthermore, conventional methods focus solely on pixel-level reconstruction loss optimization, neglecting the potential of contrastive learning to improve feature discrimination between positive and negative samples. To tackle these challenges, we propose the Frequency and Contrastive Learning-Driven Network (FCD-Net), a framework that integrates frequency-domain processing and contrastive learning for efficient document shadow removal. Specifically, we design the Hybrid Frequency Convolution Module (HFCM), which performs convolutional operations in both spatial and frequency domains, enabling simultaneous global context modeling and local refinement with low complexity. Additionally, we introduce the Structure-Preserving Refinement Module (SPRM) to incorporate structural priors in a gated manner, enhancing text boundary preservation and detail recovery. To further improve feature discrimination, we integrate the Frequency-Contrastive Feature Regularization (FCFR), which leverages contrastive learning in the frequency domain to enforce distinct feature representations between positive and negative samples. Extensive experiments conducted on public benchmarks and Optical Character Recognition (OCR) performance demonstrate the effectiveness of FCD-Net, achieving state-of-the-art performance while maintaining computational efficiency.