Artificial intelligence (AI) offers immense potential in the healthcare area, from medical forecasting to clinical decision-making. Despite AI models’ high performance, there are instances where they operate as black boxes, restricting acceptance and application in critical fields like healthcare. Explainable Artificial Intelligence (XAI) approaches have been created to solve this issue, with the goal of improving the transparency of AI models’ decision-making processes and building trust. The paper focuses on recent advances in XAI approaches for healthcare, e.g., diagnostic imaging, personalized treatment plans, etc., and also includes an overview of methodologies like LIME, SHAP, GradCAM, LRP to improve the interpretability and transparency of AI models in the healthcare domain. The paper advocates for responsible AI implementation and enables stakeholders in healthcare decision-making by summarizing key research findings, highlighting obstacles, and proposing future possibilities, thereby contributing to a better understanding of the role of explainability in AI.