Scholarlink

Peer review, essential for scientific manuscript assessment, faces efficiency challenges due to increasing submissions, demanding reliable automatic metrics for review quality evaluation. The currently most widely used word-overlap and embedding-based metrics using single peer review references show significant discrepancies with human judgments. To address these issues, we propose a novel automatic metric for evaluating the quality of peer reviews, designed to analyze the quality of atomic review opinions (AROs) using large language models, named ReviewScore. Besides, we construct a high-quality benchmark through human annotations to reliably measure ReviewScore and peer review generation models, called ReviewEval. The reference reviews in ReviewEval are crafted by integrating valuable opinions from multiple reviewers based on meta review, making them more holistic and objective. Experimental results show ReviewScore’s superior alignment with human judgments compared to existing metrics. Besides, using ReviewEval, we comprehensively re-evaluate peer review generation models and conduct detailed analysis, revealing several key insights. The ReviewEval benchmark and the toolkit for ReviewScore will be publicly released.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Automatic Evaluating Scientific Reviews Through Meta-reviewer’s Lens: A Reliable Benchmark for Peer Review Generation

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Shu-Hang Liu,
  • Tian Lan,
  • Yun-He Zhang,
  • Ran Liu,
  • Heyan Huang,
  • Chen Xu,
  • Zhijing Wu,
  • Xian-Ling Mao

摘要

Peer review, essential for scientific manuscript assessment, faces efficiency challenges due to increasing submissions, demanding reliable automatic metrics for review quality evaluation. The currently most widely used word-overlap and embedding-based metrics using single peer review references show significant discrepancies with human judgments. To address these issues, we propose a novel automatic metric for evaluating the quality of peer reviews, designed to analyze the quality of atomic review opinions (AROs) using large language models, named ReviewScore. Besides, we construct a high-quality benchmark through human annotations to reliably measure ReviewScore and peer review generation models, called ReviewEval. The reference reviews in ReviewEval are crafted by integrating valuable opinions from multiple reviewers based on meta review, making them more holistic and objective. Experimental results show ReviewScore’s superior alignment with human judgments compared to existing metrics. Besides, using ReviewEval, we comprehensively re-evaluate peer review generation models and conduct detailed analysis, revealing several key insights. The ReviewEval benchmark and the toolkit for ReviewScore will be publicly released.