The reliable detection of objects is of paramount importance for autonomous systems such as self-driving cars, particularly in unfavourable real-world conditions. The present paper introduces a novel methodology for the evaluation of the robustness of object detection algorithms based on Red-Green-Blue (RGB) images or Light Detection and Ranging (LiDAR) data. The utilisation of explainable artificial intelligence (x-AI) facilitates the identification of critical regions within objects. By systematically removing these key pixels or points, it is possible to determine how much of an object can be perturbed before it becomes undetectable. This approach offers a more precise comparison of object recognition systems than traditional methods, providing a meaningful assessment of their resilience to missing or manipulated data. The relative robustness scale employed in this study utilises two sensor modalities, thereby unveiling disparities in the robustness of object detection algorithms. By focusing on the most critical parts of an object, a more precise comparison of detection performance is achieved than with traditional approaches that rely mainly on random effects.