Artificial intelligence (AI)Artificial Intelligence (AI) based systems have been used to automate decision-making in various fields, from business, agriculture, communication, and of areas, from industry, agriculture, communication, and more important ones that directly affect people's lives, like healthcare. There is a growing trend of trying to develop medically explicable Artificial Intelligence (XAI) systems employing strategies like attention mechanisms and surrogate models, even though the great majority of these suggested AI systems are considered to be black box prototypes with no explainability. The AI system in which user-end individuals can comprehend how a machine learningMachine Learning (ML) algorithm comes to the conclusion, is considered to be explainable. The transformation of AI-based to XAI models and its applications over AIArtificial Intelligence (AI) models is presented. There is a discussion of several XAI-driven healthcare techniques and how well they performed in the current study. The chapter covers using XAI-assisted medical robotics in the healthcare industry, including early illness prediction and intelligent disease diagnosis.