The field of black-box metaheuristic optimisation has seen rapid advancements, with numerous algorithms developed to tackle diverse problem classes. Traditional benchmarking approaches tend to evaluate algorithms on limited scenarios ignoring investigations into effects of algorithm parameters, which restricts the depth of insights into how algorithmic components and hyperparameters influence performance across diverse problem landscapes. This chapter introduces a framework for explainable benchmarking that integrates explainable AI techniques to improve the interpretability of benchmarking results. Focusing on parameterised or modularised algorithms, we employ explainable AI tools to assess the individual contributions of algorithmic components across a range of problem landscapes. Our framework called IOHxplainer enables researchers to systematically analyze a wide array of configurations, facilitating a deeper understanding of how each component influences performance. Two case studies highlight IOHxplainer’s ability to handle large configuration spaces, demonstrating the framework’s effectiveness in capturing detailed interactions between algorithmic components and hyperparameters: one examines dynamic parameter importance over time, while the other evaluates the robustness of the approach by comparing insights from partial configurations data, thus supporting reliable benchmarking and parameter analysis across varied optimisation scenarios.