Lexical matching methods remain the default standard for evaluating question-answering (QA) systems. However, these methods exhibit significant limitations when valid answers deviate from predefined reference answers. As a result, evaluating QA systems in a rapidly evolving landscape becomes increasingly challenging. This issue is especially pronounced with the advent and widespread application of large language models (LLMs) in QA systems. LLMs tend to generate longer, more complex answers, further exposing the inadequacies of traditional evaluation metrics. Consequently, modern evaluation approaches are necessary to effectively reflect advancements in contemporary QA systems. In this paper, we analyze several open-domain QA models across three Vietnamese datasets, employing flexible and context-appropriate metrics to achieve more accurate performance evaluations. Our findings reveal that while all models tend to be underrated compared to their actual capabilities, fine-tuned GPT models improve average performance by approximately 25%, achieving results comparable to leading models. Furthermore, we observe that over half of the lexical matching errors stem from answers that are nearly equivalent in semantic meaning.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Evaluation of Question-Answering Systems Based on Semantic Similarity

  • Dinh Minh Hoa,
  • Vo Dinh Bay,
  • Tran Khai Thien

摘要

Lexical matching methods remain the default standard for evaluating question-answering (QA) systems. However, these methods exhibit significant limitations when valid answers deviate from predefined reference answers. As a result, evaluating QA systems in a rapidly evolving landscape becomes increasingly challenging. This issue is especially pronounced with the advent and widespread application of large language models (LLMs) in QA systems. LLMs tend to generate longer, more complex answers, further exposing the inadequacies of traditional evaluation metrics. Consequently, modern evaluation approaches are necessary to effectively reflect advancements in contemporary QA systems. In this paper, we analyze several open-domain QA models across three Vietnamese datasets, employing flexible and context-appropriate metrics to achieve more accurate performance evaluations. Our findings reveal that while all models tend to be underrated compared to their actual capabilities, fine-tuned GPT models improve average performance by approximately 25%, achieving results comparable to leading models. Furthermore, we observe that over half of the lexical matching errors stem from answers that are nearly equivalent in semantic meaning.