Scholarlink

<p>Review quality assessment is essential for evaluating effectiveness of peer review process, while traditional manual assessment has limitations such as potential subjectivity and labor-intensive nature. The rapid advancement of large language models (LLMs) has drawn attention as a potential solution; however, their application remains underexplored. This study examines the performance of three prominent LLMs–ChatGPT-4o, Claude 3.5 Sonnet, and Gemini-EXP-1206–in assessing peer review quality on six widely adopted review quality instruments (RQIs) using peer review reports from 180 biomedical papers published in <i>eLife.</i> We applied two prompting strategies (Zero-Shot and Few-Shot) across three scenarios: direct scoring, simulated double-blind peer review, and acting as domain experts. Results showed that LLM scores exhibited strong consistency measured by observed agreement (Po) reflecting the percentage of perfectly matching scores between raters and fuzzy observed agreement (fuzzy Po) allowing for a 1-point difference between scores across repetitions, especially under Few-Shot prompt in acting as domain experts and simulated double-blind peer review scenarios. However, the performance of LLM repetitions measured by Cohen’s Kappa showed poor to moderate agreement. Comparisons with human evaluators showed varied agreement, with the highest (~ 70% Po and &gt; 90% fuzzy Po) observed in the first RQI tool. Few-Shot prompts slightly improved agreement, but no obvious correlations were found between LLMs and human evaluators. Among the examined three LLMs, Claude 3.5 Sonnet demonstrated higher performance, showing higher consistency and alignment with human evaluations despite limitations. Findings highlight that LLMs are not yet capable of replacing human efforts in review quality assessment. Further exploration of prompting strategies and model capabilities is needed for realize the potential for automating peer review.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Can large language models assess the quality of peer review? An empirical study

  • 期刊论文

  • 发表日期：

    2026年4月10日

查看全文
  • Ying Tang,
  • Yanlan Kang,
  • Sunsi Wu,
  • Ruilin Zhang,
  • Zhuanlan Sun

摘要

Review quality assessment is essential for evaluating effectiveness of peer review process, while traditional manual assessment has limitations such as potential subjectivity and labor-intensive nature. The rapid advancement of large language models (LLMs) has drawn attention as a potential solution; however, their application remains underexplored. This study examines the performance of three prominent LLMs–ChatGPT-4o, Claude 3.5 Sonnet, and Gemini-EXP-1206–in assessing peer review quality on six widely adopted review quality instruments (RQIs) using peer review reports from 180 biomedical papers published in eLife. We applied two prompting strategies (Zero-Shot and Few-Shot) across three scenarios: direct scoring, simulated double-blind peer review, and acting as domain experts. Results showed that LLM scores exhibited strong consistency measured by observed agreement (Po) reflecting the percentage of perfectly matching scores between raters and fuzzy observed agreement (fuzzy Po) allowing for a 1-point difference between scores across repetitions, especially under Few-Shot prompt in acting as domain experts and simulated double-blind peer review scenarios. However, the performance of LLM repetitions measured by Cohen’s Kappa showed poor to moderate agreement. Comparisons with human evaluators showed varied agreement, with the highest (~ 70% Po and > 90% fuzzy Po) observed in the first RQI tool. Few-Shot prompts slightly improved agreement, but no obvious correlations were found between LLMs and human evaluators. Among the examined three LLMs, Claude 3.5 Sonnet demonstrated higher performance, showing higher consistency and alignment with human evaluations despite limitations. Findings highlight that LLMs are not yet capable of replacing human efforts in review quality assessment. Further exploration of prompting strategies and model capabilities is needed for realize the potential for automating peer review.