The use of Retrieval-Augmented Generation (RAG) models has become a powerful method for generating natural language by incorporating information retrieval. However, the current retrieval phase of RAG models is somewhat simplistic and has several vulnerabilities that affect the quality and relevance of generated responses. One critical issue is determining the distance between the embedded query vector and the external knowledge vectors, alongside the challenge of selecting an optimal number of relevant retrieved documents. The objective of this study is to boost the strength and resilience of LLMs to hallucinations by refining the RAG retrieval process and, consequently, improving the quality of the responses generated by RAG systems for applications like AI-driven customer support, information retrieval, and interactive AI systems with more accurate and contextually appropriate responses. Experiments with different numbers of retrieved documents were conducted to enhance answer completeness and implement a threshold mechanism to use only high-confidence contexts during the generation phase. The results of our experiments demonstrate that these enhancements lead to more accurate and relevant responses in RAG models. By increasing the number of retrieved documents, the Relevancy of Answers improves by 16% for GPT-4, while their Correctness shows slight improvements of 4% for GPT-4 and 2% for GPT-3.5-Turbo. Furthermore, our approach is compared with the AutoRAG framework and overall surpasses it in terms of efficiency and response quality.