Scholarlink

Evaluating complex Retrieval Augmented Generation (RAG) systems in real-world settings is challenging. There is often a lack of fine-grained labelled data and the absence of comprehensive evaluation tools that can assess individual components of a pipeline. This hinders rapid, rigorous development, particularly for agentic RAG systems. We describe our experienced at GuideStream.AI, a startup developing an AI for clinical guideline recommendation. To address this gap, we developed Evalugator , a suite of agentic components to support agile development and evaluation. Evalugator features: (1) generation of synthetic queries, relevance assessments, answers and evaluation criteria for training and evaluation in new domains; (2) LLM-based judging agents; and (3) UI and API tools to launch experiments and analyse results. This paper uses Evalugator as a case study to demonstrate how a principled, agent-based evaluation framework can support the rapid development of complex RAG systems in a startup environment.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evalugator A stylized green crocodile with a yellow grid pattern on its back, resembling a calculator. The design combines elements of a crocodile and a calculator, symbolizing a playful or creative concept. —Rapid, Agile Development and Evaluation of Retrieval Augmented Generation Systems Without Labels

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Bevan Koopman,
  • Hang Li,
  • Shuai Wang,
  • Guido Zuccon

摘要

Evaluating complex Retrieval Augmented Generation (RAG) systems in real-world settings is challenging. There is often a lack of fine-grained labelled data and the absence of comprehensive evaluation tools that can assess individual components of a pipeline. This hinders rapid, rigorous development, particularly for agentic RAG systems. We describe our experienced at GuideStream.AI, a startup developing an AI for clinical guideline recommendation. To address this gap, we developed Evalugator , a suite of agentic components to support agile development and evaluation. Evalugator features: (1) generation of synthetic queries, relevance assessments, answers and evaluation criteria for training and evaluation in new domains; (2) LLM-based judging agents; and (3) UI and API tools to launch experiments and analyse results. This paper uses Evalugator as a case study to demonstrate how a principled, agent-based evaluation framework can support the rapid development of complex RAG systems in a startup environment.