Large Language Models (LLMs) have emerged as critical tools across various NLP tasks but often rely on outdated or inaccurate parametric knowledge. Retrieval-Augmented Generation (RAG) addresses this limitation by incorporating external, real-time information into LLM responses. Nevertheless, existing RAG systems frequently suffer from uncertain retrieval timing and unreliable retrieval quality. To address these challenges, we propose INSIGHT-RAG, a novel RAG framework that dynamically optimizes the retrieval and generation processes by quantifying and estimating internal state signals of the LLM, and enhances overall system reliability through multi-agent collaborative decision-making. Specifically, we design a dual-metric evaluation framework based on normalized Gram determinants and singular value entropy to extract generation certainty signals from the LLM’s hidden states, quantifying the model’s confidence in its outputs without relying on output probabilities or external annotations. Furthermore, agents are dynamically activated at multiple stages to perform adaptive filtering and reasoning, leading to more trustworthy outcomes. By this means, INSIGHT-RAG offers a scalable solution for high-reliability, low-noise open-domain question answering, and we have validated the superior performance of INSIGHT-RAG across multiple benchmarks. Its modular design paves the way for more robust knowledge systems.