<p>Large language models have rapidly transformed conversational AI, creating a pressing need for evaluation methods that go beyond traditional text similarity or translation-based metrics. Modern chatbots demand assessment frameworks capable of measuring complex dimensions such as multi-turn coherence, reasoning quality, factual accuracy, alignment with safety constraints, and robustness under adversarial conditions. This survey provides a comprehensive synthesis of current evaluation methodologies, spanning rule-based scoring, automated metrics, learned evaluators, LLM-as-a-judge systems, consensus-based multi-model evaluation, safety and adversarial stress-testing, and human-in-the-loop protocols. We propose an eight-part taxonomy to categorize the evolving landscape of evaluation techniques, analyze their strengths and limitations, and highlight emerging reliability trends. In addition, we compile and characterize major datasets, benchmarks, and open-source tools used in contemporary chatbot evaluation. Persistent challenges—including bias, hallucination detection, long-context handling, and scalable human evaluation—are identified as critical research gaps. To address these limitations, we outline a conceptual unified evaluation pipeline that integrates multiple assessment paradigms. This work aims to serve as a foundational reference for future research toward more reliable, transparent, and trustworthy evaluation of LLM-driven conversational agents.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Evaluating chatbots in the LLM era: a hybrid survey and qualitative synthesis of evaluation frameworks

  • Siddhant Agwan,
  • Avishkar Tambe,
  • Saad Shah,
  • Govind Pole

摘要

Large language models have rapidly transformed conversational AI, creating a pressing need for evaluation methods that go beyond traditional text similarity or translation-based metrics. Modern chatbots demand assessment frameworks capable of measuring complex dimensions such as multi-turn coherence, reasoning quality, factual accuracy, alignment with safety constraints, and robustness under adversarial conditions. This survey provides a comprehensive synthesis of current evaluation methodologies, spanning rule-based scoring, automated metrics, learned evaluators, LLM-as-a-judge systems, consensus-based multi-model evaluation, safety and adversarial stress-testing, and human-in-the-loop protocols. We propose an eight-part taxonomy to categorize the evolving landscape of evaluation techniques, analyze their strengths and limitations, and highlight emerging reliability trends. In addition, we compile and characterize major datasets, benchmarks, and open-source tools used in contemporary chatbot evaluation. Persistent challenges—including bias, hallucination detection, long-context handling, and scalable human evaluation—are identified as critical research gaps. To address these limitations, we outline a conceptual unified evaluation pipeline that integrates multiple assessment paradigms. This work aims to serve as a foundational reference for future research toward more reliable, transparent, and trustworthy evaluation of LLM-driven conversational agents.