In this study, we evaluate two distinct chatbot models integrated into a comprehensive healthcare platform, with a focus on addressing medical and mental health inquiries. The chatbot, driven by LLAMA2, equipped with an inbuilt Retrieval-Augmented Generation (RAG) mechanism, specializes in retrieving and generating precise responses to medical queries and is tailored to offer personalized and empathetic support for mental health concerns. Through meticulous analysis, we assess the effectiveness of these chatbots against a spectrum of functional and non-functional requirements, encompassing usability, security, scalability, accuracy, and empathy. Our investigation delves into LLAMA2’s performance across four distinct scenarios related to mental health inquiries. These scenarios involve variations in fine-tuning and the provision of custom prompts to the chatbot. We scrutinize LLAMA2’s responsiveness in both fine-tuned and non-fine-tuned states, as well as with and without custom prompts, aiming to discern the impact of these optimization strategies on the chatbot’s capacity to deliver empathetic and supportive responses. Our findings provide valuable insights into the nuanced intricacies of LLAMA2’s role in mental health support within AI-driven healthcare solutions, offering implications for further development and refinement in this critical domain.