This paper addresses the challenges faced by Retrieval-Augmented Language Models (RALMs) in reducing factual errors by introducing a framework for structured relevance assessment. The aim is to enhance the robustness of RALMs by improving document relevance evaluation, balancing intrinsic and external knowledge, and managing unanswerable queries effectively. We propose a multi-dimensional scoring system for document relevance, considering semantic matching and source reliability. Our approach includes embedding-based relevance scoring, the use of synthetic training data with mixed-quality documents, and specialized benchmarking on niche topics. Additionally, we implement a knowledge integration mechanism and an “unknown” response protocol to handle queries where knowledge is insufficient. Preliminary evaluations show significant reductions in hallucination rates and improved transparency in reasoning processes. This work advances the development of more reliable question-answering systems capable of functioning effectively in dynamic environments with variable data quality. While challenges remain in accurately distinguishing credible information and balancing system latency with thoroughness, the proposed framework marks a step forward in enhancing the reliability of RALMs.