Artificial intelligence (AI) is an emerging field of computer science which is currently being used in many sophisticated applications such as e-commerce, military, education, industry, and healthcare. AI has several subfields like machine learning, neural network, Deep Learning, Natural Language Processing (NLP), and computer vision. In the medical domain, AI with deep learning model plays a crucial role to predict the symptoms of various kinds of disease and clinicians to make decision about analysis critical medical report provided by radiologists and pathologists. However, the adoption of many AI model in healthcare face challenges related to transparency, interpretability, and trustworthiness, due to their “Black-Box” in nature. Usually, it is essential for humans to understand the reasoning behind an AI model’s decision-making. To make a better decision-making, Explainable AI (XAI) is a useful technique that aims to explain the information inside the black-box model of AI algorithms that reveals how the decisions are made. The aim of this paper is to provide a survey of the most novel XAI techniques used in healthcare and related medical imaging applica- tions. In addition, this paper provides the study of various applications of XAI in healthcare and focuses on challenges related to black-box AI models, emphasizing the requirement for interpretable arrangements in healthcare. Furthermore, this paper presents different XAI strategies, including Local Interpretable Model-Agnostic Explanations (LIME), Shapley Additive Explanations (SHAP), and rule-based frameworks, which are displayed and assessed for their viability in making AI models interpretable. Finally, this survey paper provides future direction to help developers and researchers for future prospective investigations in healthcare and discusses future research possibilities in the area of XAI.