Artificial intelligence (AI) has shown immense potential in helping medical professionals with diagnosis and treatment recommendations. However, the lack of transparency and interpretability in current AI systems hinders their widespread adoption and integration into clinical workflows. This research study presents a novel approach that combines explainable AI (XAI) techniques, Large Language Models (LLMs), and knowledge graphs to enhance the interpretability and trustworthiness of AI-assisted medical systems for both medical professionals and patients. Our proposed methodology employs XAI techniques such as SHAP and LIME in diagnosis models to provide transparency into the AI decision-making process, while treatment recommendation models leveraged inherently explainable causal graphs. LLMs improve the interpretability of XAI-based models by processing their explanations alongside patient electronic health records, using a RAG system to ground the analysis in verified medical facts, which not only reduces hallucinations, but also enables the generation of trustworthy, personalized patient reports with clear treatment suggestions and explanations. The natural language capabilities of LLMs enable the generation of reports that are easily understandable by both medical professionals and patients. This approach not only enables medical professionals to fully understand and trust the AI decision-making process, but also promotes patient understanding and trust in the AI-assisted medical process. We demonstrate the effectiveness of our methodology using the MIMIC-IV dataset for Early Prediction of Sepsis and Treatment Recommendation as a case study. However, this approach can be seamlessly extended to other diseases and medical conditions. By fostering trust among medical professionals and patients through improved model interpretability, our research contributes to the development of highly interpretable, trustworthy, and clinical workflow-integrated AI systems.