Closing the Lexical-Semantic Divide: A Unified Approach for Retrieval-Augmented Generation
摘要
Retrieval-Augmented Generation (RAG) systems are constantly challenged to balance factual grounding, relevance, and efficiency. To overcome these constraints, this study evaluates transformer-based generative models for citation-anchored response generation and suggests a novel hybrid retrieval framework that combines the lexical precision of BM25 with the semantic contextualization of Dense Passage Retrieval (DPR). According to empirical validation on the TREC RAG 2024 benchmark, the BM25+DPR hybrid successfully bridges lexical-semantic gaps in retrieval, achieving 36% higher nDCG@20 and 27% enhanced Recall@100 compared to BM25 alone. For example, searches like “health impacts of air pollution” return articles about “COPD mortality rates” (retrieved using DPR) while maintaining precise keyword matches like “air pollution” (retrieved through BM25). However, evaluation is limited to syntactic validation using the Ragnarök toolbox due to the lack of TREC’s generation standards, highlighting outstanding issues in determining factual accuracy and citation fidelity. Reproducibility and adaptability are given top priority in our modular, open-source pipeline, which provides practical insights for RAG system optimization in precision-critical fields like legal analytics and medical informatics. To further reliable, practical RAG applications, these findings support hybrid evaluation systems that combine retrieval efficacy and generation quality.