Specular highlights on smooth surfaces obscure image details and degrade the performance of vision tasks such as segmentation and detection. Traditional RGB-based highlight removal methods rely heavily on supervised data, suffering from high acquisition costs and limited generalization. Polarization imaging provides complementary cues by separating specular reflections from diffuse components, offering a physical strategy to address this challenge. Inspired by this, we propose the Polarization-guided Specular Removal Diffusion Model (PSR-Diff), which integrates diffusion models with polarization priors. PSR-Diff extracts polarization cues via Fresnel theory to construct a pseudo-diffuse estimate and fuses it with RGB features through a cross-modal attention mechanism. To mitigate data scarcity and modality differences, we introduce a dual-module fusion strategy that leverages a pretrained RGB diffusion model while enabling polarization-guided learning. Experiments on polarization and RGB-only datasets show that PSR-Diff achieves superior highlight suppression, enhanced image quality, and robust performance under diverse lighting and materials.