Scholarlink

Evaluating retrieval-augmented generation (RAG) systems is challenging in the legal domain due to the nuanced nature of reasoning and scarcity of domain-specific datasets. We investigate semantic similarity and legal correctness in Brazilian legal question answering. We develop a synthetic dataset of 3,012 evaluation instances from the Brazilian Civil Procedure Code spanning seven query types, validated through multi-agent critique. Our evaluation framework combines BERTScore, retrieval metrics, and domain-adapted LLM-as-Judge to assess semantic coherence and legal correctness. Analysis reveals that 53.3% of responses exhibit metric disagreement, with substantial variation by query type. Dual queries show the weakest correlation (r = 0.372), while comparative queries show the lowest legal correctness (0.24–0.36). Explicit article citation strongly predicts quality (LLM-Judge: 0.624 with citation vs 0.144 without). Dense retrieval outperforms BM25 for legal grounding, though no method guarantees semantic and legal correctness. Findings demonstrate that semantic metrics alone are insufficient for legal RAG evaluation, with disagreement patterns structured by query type and retrieval strategy. This work supports query-adaptive retrieval and multidimensional evaluation frameworks development in specialized domains.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

When Semantic Similarity Fails: Analyzing Metric Divergence in Legal RAG Evaluation

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Eryclis Silva

摘要

Evaluating retrieval-augmented generation (RAG) systems is challenging in the legal domain due to the nuanced nature of reasoning and scarcity of domain-specific datasets. We investigate semantic similarity and legal correctness in Brazilian legal question answering. We develop a synthetic dataset of 3,012 evaluation instances from the Brazilian Civil Procedure Code spanning seven query types, validated through multi-agent critique. Our evaluation framework combines BERTScore, retrieval metrics, and domain-adapted LLM-as-Judge to assess semantic coherence and legal correctness. Analysis reveals that 53.3% of responses exhibit metric disagreement, with substantial variation by query type. Dual queries show the weakest correlation (r = 0.372), while comparative queries show the lowest legal correctness (0.24–0.36). Explicit article citation strongly predicts quality (LLM-Judge: 0.624 with citation vs 0.144 without). Dense retrieval outperforms BM25 for legal grounding, though no method guarantees semantic and legal correctness. Findings demonstrate that semantic metrics alone are insufficient for legal RAG evaluation, with disagreement patterns structured by query type and retrieval strategy. This work supports query-adaptive retrieval and multidimensional evaluation frameworks development in specialized domains.