Vision-language models like CLIP have shown remarkable zero-shot capabilities across various tasks by aligning image and text representations through contrastive learning. To further boost CLIP’s effectiveness, prompt tuning introduces learnable context tokens into textual inputs. However, existing prompt tuning methods typically assume a fixed test distribution, making them less reliable in real-world settings. Recent test-time adaptation (TTA) techniques allow the model to adapt during inference, yet they often struggle to remain stable under continuous and long-term distributional changes. In such settings, models are prone to overfitting unreliable pseudo-labels, resulting in performance degradation due to error accumulation and forgetting. To address these issues, we propose FuzzyPrompt, a continual TTA approach tailored for CLIP that leverages fuzzy logic to guide prompt-level knowledge distillation. A fixed teacher prompt set provides soft supervision, while fuzzy confidence scores dynamically control the influence of each sample during adaptation. This selective mechanism enables the student prompt to benefit from high-confidence knowledge while reducing the risk posed by uncertain predictions. In addition, we apply entropy-based pseudo-label filtering and maintain a sliding prompt buffer to enhance temporal stability. FuzzyPrompt operates without any offline training and enables efficient online adaptation. Experimental results demonstrate that our method maintains robust and stable performance under prolonged distribution shifts.