Undesirable specular highlight degrades both the visual quality of images and the performance of subsequent image processing tasks. While recent deep learning-based methods have achieved notable advancements in highlight removal, challenges such as highlight residual and color inconsistency still exist. To address these challenges, our study proposes an end-to-end highlight removal network with a conditional diffusion model, which has gotten promising results in image restoration areas. To decrease highlight residual, we design a highlight detection module providing binary highlight location masks based on a TransUNet architecture, improving highlight detection accuracy from 0.97 to 0.98. To rectify color distortions, we integrate a color extraction module that supplies illumination invariant color map priors, ensuring color fidelity in the dehighlighted output. Furthermore, we introduce feature loss into our model during the training process rather than dependent on pixel loss only, which provides our model with feature-level information and could better preserve the content and structure of the original image. To the best of our knowledge, this is the first diffusion-based method tailored for highlight removal. Extensive experiments on the benchmark SHIQ dataset demonstrate that our method obtains competitive results compared with current state-of-the-art methods and significantly improves the SSIM value from 0.939 to 0.966.