The increasing number of new datasets and anomaly detection (AD) algorithms has highlighted the need for a scalable benchmarking framework to showcase algorithms’ performance portfolio and potentials of datasets in an interpretable way. Existing benchmarking methods typically report raw performance metrics without offering interpretable insights into the relationships between algorithms and datasets. In this work, we propose XEvalAD, a novel and explainable evaluation framework that applies Item Response Theory (IRT) to AD benchmarking. By jointly modelling the latent traits of algorithms (e.g., capability) and datasets (e.g., difficulty), XEvalAD enables performance ranking and reveals underlying factors that influence algorithm behaviour. The framework produces intuitive visualisations that facilitate deeper understanding of algorithm-dataset interactions, uncovering robustness patterns and dataset-specific challenges. We demonstrate the effectiveness of XEvalAD through extensive experiments on a large-scale benchmark suite covering diverse AD algorithms and datasets. Results show that our approach provides a more interpretable perspective on AD evaluation. The source code is publicly available at https://github.com/SabaFathi/XEvalAD .