Pneumonia is a common respiratory illness that requires prompt and precise diagnosis to provide appropriate treatment. By concentrating on the interpretability of AI-driven medical diagnoses, this research aims to solve this important problem. The primary aim is to disentangle the decision-making mechanisms of intricate artificial intelligence systems that detect pneumonia. Explainable AI models will be strategically implemented to do this, focusing on Local Interpretable Model-Agnostic Explanations (LIME). LIME is selected because of its capacity to approximate complex model’s behavior in a way that is comprehensible locally, thereby offering clear insights into their predictions. The study entails a thorough assessment and comparison of several AI models, focusing on their interpretability as much as their diagnostic accuracy. By doing this, the project hopes to close the gap between the necessity for healthcare practitioners to understand and have faith in the decision-making processes of AI models and the models’ potent predictive powers. The anticipated results include improving pneumonia detection techniques, guaranteeing that AI models perform accurately and deliver comprehensible and reliable insights.