Pretrained Vision-Language Models (VLMs) have demonstrated remarkable performance across a wide range of vision tasks, serving as powerful backbone models for transfer learning. A key challenge in few-shot adaptation of VLMs is that existing methods often overlook classification-interfering noise patches in image regions, which adversely affect model performance. To address this issue, we propose CLIP-TNR, a Text-guided Noise Replacement Visual Prompt Learning framework designed to suppress the interference of noisy patches in images. At the core is the noise-replaced visual prompts learning module, which integrates a noise patch filtering mechanism that identifies and replaces noisy patches with learnable visual prompts. This is achieved without modifying the model architecture or introducing additional image tokens, effectively reducing noise while maintaining model efficiency. Additionally, we design a text-level contrastive learning loss function to enhance the class-discriminative power of the learnable prompts and improve overall model generalization. Extensive experiments on standard benchmarks demonstrate that our method consistently outperforms state-of-the-art approaches across various evaluation settings.