Scholarlink

<p>The rapid adoption of Retrieval-Augmented Generation (RAG) chatbots has introduced challenges in systematic evaluation, test design, and efficiency measurement. In this paper, we propose RAG-Test, a developer-centric and unified framework that automates test case generation, attribution validation, and productivity estimation. Unlike existing methods, RAG-Test leverages large language models (LLMs) to generate structured question–answer (QA) pairs, ensuring scalable and unbiased test matrix creation. A cosine similarity-based attribution mechanism improves evaluation reliability by validating whether QA pairs originate from the correct document chunks. Additionally, we introduce a theoretical productivity gains model, estimating chatbot efficiency improvements over search engines. To validate RAG-Test, we mined 1,500 random Wikipedia articles and conducted experiments on 10,000 document chunks across eight LLMs (Meta-Llama-3.1-70B, Mixtral-8 × 7B, gemma2, DeepSeek R1, Qwen-2.5-32B, etc.). Over 5,000 QA pairs were analyzed using cosine similarity, statistical significance tests (t-tests, Wilcoxon, ANOVA), and failure rate analysis. While llama3-8b-8192 demonstrated the largest effect size, Meta-Llama-3.1-70B-Instruct achieved the lowest mis-attribution rate (1.58%), indicating more consistent attribution performance. RAG-Test provides a scalable, automated, and statistically validated evaluation framework, enabling developers to improve retrieval accuracy and optimize chatbot efficiency.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

RAG-test: a framework to evaluate RAG chatbot performance and measure productivity gains

  • 期刊论文

  • 发表日期：

    2026年8月3日

查看全文
  • Shashank Parmar

摘要

The rapid adoption of Retrieval-Augmented Generation (RAG) chatbots has introduced challenges in systematic evaluation, test design, and efficiency measurement. In this paper, we propose RAG-Test, a developer-centric and unified framework that automates test case generation, attribution validation, and productivity estimation. Unlike existing methods, RAG-Test leverages large language models (LLMs) to generate structured question–answer (QA) pairs, ensuring scalable and unbiased test matrix creation. A cosine similarity-based attribution mechanism improves evaluation reliability by validating whether QA pairs originate from the correct document chunks. Additionally, we introduce a theoretical productivity gains model, estimating chatbot efficiency improvements over search engines. To validate RAG-Test, we mined 1,500 random Wikipedia articles and conducted experiments on 10,000 document chunks across eight LLMs (Meta-Llama-3.1-70B, Mixtral-8 × 7B, gemma2, DeepSeek R1, Qwen-2.5-32B, etc.). Over 5,000 QA pairs were analyzed using cosine similarity, statistical significance tests (t-tests, Wilcoxon, ANOVA), and failure rate analysis. While llama3-8b-8192 demonstrated the largest effect size, Meta-Llama-3.1-70B-Instruct achieved the lowest mis-attribution rate (1.58%), indicating more consistent attribution performance. RAG-Test provides a scalable, automated, and statistically validated evaluation framework, enabling developers to improve retrieval accuracy and optimize chatbot efficiency.