Machine learning can significantly improve power side-channel attacks that derive cryptographic keys from hardware devices. The attacker learns a model that maps side-channel information (e.g., a device’s power consumption) to the computation’s intermediate states/values, which in turn serve as evidence for the actual key. The model implicitly learns which portions of the power traces leak information and which are mere noise. This knowledge is of great interest to the designer of the cryptographic hardware. Knowing the “leakage points” allows them to tweak the implementation to prevent the leakage. The community has thus investigated the use of “Explainable AI” (XAI) to derive the machine-learning model’s knowledge about existing leakage. Unfortunately, with limited success for protected (masked) implementations as used in practice so far. In this paper, we show that very much like for a side-channel attack itself, model analysis using XAI must focus on intermediate values rather than keys, accumulate evidence for all intermediates, and make an informed choice at the end. Doing so successfully, however, requires class-discriminative explanations—a fact overlooked up to now. We present a novel analysis method, LeaX, that uses this observation to precisely pinpoint leakage, especially for masked cryptographic implementations, which prior work has failed to do.