There is a paradigm shift toward precision medicine that is impacting all specialties. Artificial intelligence (AI) pertains to computers systems possessing the capability to emulate human intelligence. AI can be utilized for diagnostics, treatment planning, digital workflow, robot surgery, risk assessment, and providing predictive insights. With recent advances in AI, especially Deep Learning (DL) there is a need for explainable AI (XAI), especially in the medical field. The current research aim is to uncover whether with the increase in DL research there is an increase in research on XAI in healthcare, reveal current trends, elaborate the terms and concepts on XAI in medicine, dentistry and orthodontics, provide an overview of available knowledge with its gaps and ideas for future research as well as serve as a push factor toward responsible AI. Bibliometric parameters were extracted and articles were retrieved from the Scopus database with no initial time limit. Research visualization and mapping was performed using R-studio. The results reveal the current research status and bibliometric analysis summarizing current research trends on XAI within the medical field. Research on XAI started to increasing after the year 2020 together with the increase in research on DL in medicine but is still limited in dentistry.