The paper focuses on the elucidating importance of eXplainable Artificial Intelligence (XAI) to improve the reliability and interpretability of Artificial Intelligence (AI) models for prediction of cardiac diseases. Countering the issue of AI models often being a black box, the paper works with SHAP as one of the most reliable ways to explain predictions made by the machine learning model. The study employs five common machine learning algorithms, namely, Logistic Regression, Random Forest (RF), Support Vector Machine (SVM), Decision Tree, and KNN for evaluating the model’s ability to diagnose heart diseases. Cross validation metrics include accuracy, precision, recall, and F1 – score, with Random Forest being the best model with a high accuracy of 90%. The proposed approach to apply SHAP in the Random Forest model also permits the global and the local interpretation of the results, due to the level of detail with which the model indicates how much features contribute. Features which have been selected for predicting heart diseases includes systolic blood pressure, age and maximum pulse rate. This is specifically important in health care domain where doctors need to understand the rational behind the AI diagnosis made which will in turn help in improving the decision-making process as well as patient outcomes. To that end, this study seeks to fill the gap in understanding between high AI performance and clinical acceptance through implementation of XAI. The paper provides directions for future studies to explore deeper learning frameworks and other XAI techniques including LIME and counter factual explanations for superior effectiveness of AI in healthcare facilities.