The application of AI in healthcare sets a greater emphasis on pattern recognition for triaging, screening, classification, reporting cases, and other tasks than in other human endeavors. As a result, the outcome of AI predictions in healthcare frequently requires human approval or decision-making. Given that different users have varying backgrounds and levels of knowledge, it may not always be necessary to fully explain the underlying principles of AI application technologies in the healthcare sector to all users. However, when applying moral and legal guidelines to AI systems that comply with explainability and transparency standards, trustworthy data foundations are crucial to the ongoing creation, gathering, and validation of data. The reason for starting the development of xAI models was the recognition that artificial intelligence models are often described as “black boxes,” meaning that neither engineers nor users of the model can completely explain or comprehend how the algorithm arrives at predictions and recommendations in a particular health procedure. xAI enhances current AI technologies by highlighting application process flaws with the use of easy-to-interpret tools like heatmaps, saliency maps, decision rules, decision trees, and scatter plots. In this chapter, we will analyze application examples of xAI tools and techniques which already exist and are used in healthcare such as Transparency and Interpretability For Understandability (TIFU) framework and others. To fully realize the promise of AI applications in healthcare, we would like to propose a new method for developing measurements for explanations inside the decision-making process, based on the work of existing frameworks. In addition to lowering bias and strengthening the patient-physician relationship, this approach should restore trust in the use of AI in healthcare. Additionally, this approach requires considering legal and regulatory compliance with laws and regulations including the Chinese PIPL, the American HIPAA, and the European GDPR.