The integration of Artificial Intelligence (AI) technologies has revolutionized various aspects of the healthcare field. As transparency and accountability in this field are of the utmost importance, eXplainable AI (XAI) has attracted increasing attention. Within the context of diagnostic interpretation, XAI can be exploited to reveal the reasoning behind an AI-based system’s decision. In this study, we prioritize explaining model predictions to medical experts for better interpretability and focus on advancing breast cancer diagnosis through interpretable deep learning models and rigorous analysis of results. The study employs various relevant deep learning and transfer learning models, while investigating an expert mammography lexicon for explainability on the challenging MIAS dataset. Moreover, extensive experiments have been performed in order to assess the influence of several criteria such as dataset balancing, chosen deep learning model, and the selected XAI technique. Reported results prove that Local Interpretable Model-agnostic Explanations (LIME) provide satisfactory results in explainability, compared to the Shapley Additive Explanations (SHAP) technique, after balancing the dataset and selecting the most appropriate deep learning model. Additionally, it should be noted that emphasizing the importance of the region of interest is crucial to avoid incorporating the background within mammograms as a predictive feature.