Open-Vocabulary Semantic Segmentation (OVSS) empowers models to recognize novel classes beyond predefined categories. While contrastive Vision-Language Models (VLMs) like CLIP enable open-vocabulary learning, they struggle with pixel-level semantic localization due to image-level pretraining. We propose a residual diffusion-based cost map refinement strategy to address these challenges. By treating CLIP’s coarse-grained classification maps as initial cost maps, our method iteratively refines them via a multi-step diffusion process, bridging the gap between high-level semantics and low-level spatial details. This enhances pixel-wise discriminative ability without retraining VLMs. Experiments on standard benchmarks demonstrate promising improvements in both quantitative accuracy and qualitative boundary precision, verifying the effectiveness of integrating diffusion for OVSS. Our approach offers a novel paradigm for advancing open-vocabulary visual understanding via foundation model refinement.