This paper presents a novel framework for implementing robust safety guardrails in conversational AI systems powered by large language models (LLMs) for healthcare applications. We propose a multi-layered approach that combines LLM-based classifiers, vector store matching, and dynamic prompt engineering to ensure safe and ethical interactions. Our system, designed to support patients with chronic conditions, demonstrates how LLMs can be effectively constrained to provide helpful information while avoiding potential risks associated with medical misinformation or inappropriate advice. We evaluate our framework using a comprehensive test set, demonstrating its efficacy in maintaining safety without significantly compromising the naturalness of conversations. Our findings contribute to the ongoing discourse on responsible AI deployment in sensitive domains like healthcare, particularly in creating systems that can build rapport and trust while adhering to strict ethical guidelines. These outcomes suggest that our multi-layered guardrail system offers a promising approach to harnessing the power of LLMs in healthcare while prioritizing patient safety and ethical considerations.