We introduce a novel technique for detecting core and spurious features in images given a caption, leveraging the capabilities of vision-language models like CLIP. Core features align closely with the caption’s semantics, while spurious features are incidental or irrelevant. Since vision language models are highly dependent on learned correlations rather than causal relationships, their performance can degrade in deployment when faced with out-of-distribution scenarios or spurious correlations. If these models are not causally grounded, their predictions may be unreliable or biased. Our method provides insight into the causal groundings of the CLIP model by employing counterfactual reasoning to systematically analyze how changes to visual elements impact model attention and alignment with textual descriptions. We develop two distinct approaches, both incorporating clustering and thresholding techniques to refine object-based core and spurious classification. This approach advances beyond static methods like attention maps, providing a deeper understanding of model behavior, improving interpretability, insights into robustness, and offering actionable insights into the causal relationships underpinning vision-language model decisions across diverse contexts.