Purpose <p>Deep neural networks (DNNs) have demonstrated excellent performance in classifying arrhythmias from electrocardiogram (ECG) signals. However, their black-box nature pose challenges to interpretability, limiting clinical trust and raising concerns about potential bias. This study aims to evaluate the interpretability of DNN models for atrial fibrillation (AFib) classification using ECG image representations.</p> Methods and results <p>We compared two convolutional neural networks (CNNs) architectures, a ResNet-50 and a previously developed model, trained on DII-long lead images from the private InCor-DB dataset. To enhance interpretability, we applied two widely adopted Explainable AI (XAI) techniques: LIME And SHAP. In addition, we also employed pixel-flipping, a perturbation-based method, to quantitatively assess the contribution of highlighted image regions to model predictions. For external validation, the proposed approach was tested on the publicly available CPSC dataset, from which 1D ECG signals were converted into images. Both models achieved high classification performance, with accuracies of 96.9% for ResNet-50 And 98.0% for the custom model. Visual explanations generated by XAI techniques consistently emphasized clinically relevant features associated with AFib, such as the absence of P waves and irregular R–R intervals. The results of pixel-flipping analysis confirmed the relevance of these features, demonstrating the robustness of the visual interpretations.</p> Conclusion <p>The proposed image-based classification model, when combined with XAI techniques, improves the interpretability of DNN predictions by highlighting clinically relevant regions within ECG images that correspond to established features of AFib. The quantitative pixel flipping method demonstrated the robustness of this approach by objectively assessing the informational content of the visual explanations generated by the XAI techniques.</p>