Counterfactuals in medical imaging are synthetic representations of how an individual’s medical image might appear under alternative, typically unobservable conditions, which have the potential to address data limitations and enhance interpretability. However, counterfactual images, which can be generated by causal generative models (CGMs), are inherently hypothetical—raising questions of how to properly validate that they are realistic and accurately reflect the intended modifications. A common approach for quantitatively evaluating CGM-generated counterfactuals involves using a discriminative model as a ‘pseudo-oracle’ to assess whether interventions on specific variables are effective. However, this method is not well-suited for in-depth error identification and analysis of CGMs. To address this limitation, we propose to leverage synthetic, ‘ground truth’ counterfactual datasets as a novel approach for debugging and evaluating CGMs. These synthetic datasets enable the computation of global performance metrics and precise localization of CGM failure modes. To further quantify failures, we introduce a novel metric, the Triangulation of Effectiveness and Amplification (TEA), which precisely quantifies the effectiveness of target variable interventions and the additional amplification of unintended effects. We test and validate our evaluation framework on two state-of-the-art CGMs where the results demonstrate the utility of synthetic datasets in identifying failure modes of CGMs, and highlight the potential of the proposed TEA metric as a robust tool for evaluation of their performance. Code and data are available at https://github.com/ucalgary-miplab/TEA .