Large language models in peer review: challenges and opportunities
摘要
The increasing volume of academic publications has placed considerable strain on traditional peer review systems, leading to delays, inconsistencies, and systemic inequities. The emergence and rapid development of large language models (LLMs) provide opportunities to address these challenges. In this review, we evaluate the potential roles, benefits, and limitations of LLMs in peer review processes. First, we comprehend from the literature the application of LLMs in various peer review tasks, which include serving as checklist assistants, aiding reviewer selection during the pre-peer review stage, generating automated feedback that highlights key evaluation aspects, providing human-like recommendations and scoring, detecting biases, and functioning as LLM agents throughout the peer review stage. Several approaches, including prompt engineering strategies, model evaluation protocols, and integrated architectures for editorial systems, have been proposed for implementing LLMs in scholarly peer review. Next, we discuss the challenges and limitations associated with LLM applications. These include inadequate validation of scientific content, limited domain-specific knowledge, difficulties in data analysis and result interpretation, and ethical concerns. Finally, we outline future research directions, such as the behavioral evaluation of LLMs, enhancement of their reasoning capabilities, development of benchmark datasets and prompts, fostering of effective LLM–human collaboration, and the exploration of multiagent LLM systems to promote reliable and trustworthy deployment. We conclude that while research on the application of LLMs in peer review tasks continues to advance, LLMs are currently more effective as supportive tools to aid human evaluators rather than as replacements. Existing limitations and ethical considerations highlight the need for a more in-depth evaluation of the long-term impact of integrating LLMs into peer review workflows.