Scholarlink

<p>Large Language Models (LLMs) are increasingly used as evaluators for subjective Information Retrieval (IR) tasks, yet their reliability under ambiguity remains poorly understood. We systematically probe LLM-as-a-judge systems by deliberately maximizing inter-model disagreement, which we formalize as horizontal variance. Across five widely used LLM evaluators, we observe that ambiguity-based prompts induce up to a 63% increase in horizontal variance, while intra-model (vertical) variance remains stable. This indicates systematic, not random, evaluator inconsistency. Our results show that models may appear stable in isolation yet diverge substantially when evaluated collectively, exposing hidden bias and instability. These findings highlight the risks of naive deployment of LLM evaluators in fairness-sensitive IR pipelines and motivate multi-model, variance-aware evaluation strategies.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Probing subjective judgment variance in LLM evaluators: A framework for robust IR evaluation

  • 期刊论文

  • 发表日期：

    2026年4月29日

查看全文
  • Ritesh Kumar

摘要

Large Language Models (LLMs) are increasingly used as evaluators for subjective Information Retrieval (IR) tasks, yet their reliability under ambiguity remains poorly understood. We systematically probe LLM-as-a-judge systems by deliberately maximizing inter-model disagreement, which we formalize as horizontal variance. Across five widely used LLM evaluators, we observe that ambiguity-based prompts induce up to a 63% increase in horizontal variance, while intra-model (vertical) variance remains stable. This indicates systematic, not random, evaluator inconsistency. Our results show that models may appear stable in isolation yet diverge substantially when evaluated collectively, exposing hidden bias and instability. These findings highlight the risks of naive deployment of LLM evaluators in fairness-sensitive IR pipelines and motivate multi-model, variance-aware evaluation strategies.