This study introduces AdCache-CLIP, an innovative zero-shot anomaly detection method designed to tackle the “cold start” issue faced by conventional unsupervised and semi-supervised anomaly detection techniques when there is an insufficiency of training data, particularly in fields like industrial quality inspection and medical diagnosis. Current zero-shot anomaly detection techniques predominantly depend on pre-trained Vision-Language Models (VLMs), although they encounter challenges including insufficient visual information guidance in textual prompts, cross-modal semantic inconsistencies, and noise interference, which constrain their detection efficacy. The study introduces the Vision Token Aggregation (ViTA) module and the Cross-Modal Semantic Fusion Clustering (CMSFC) module to tackle these issues. The ViTA module improves the detailed segmentation of anomalous regions by retaining intricate visual features and integrating feature aggregation with a visual residual connection mechanism. The CMSFC module implements cross-modal alignment and clustering restrictions to diminish semantic disparities between visual and textual modalities, hence enhancing the precision of image-level anomaly detection. Experimental findings on ten industrial and medical datasets indicate that the AdCache-CLIP method exhibits exceptional anomaly detection capabilities across various datasets, attaining an overall average performance that ranks among the leading state-of-the-art (SOTA) methods.