Peer reviews aim to provide authors with detailed, constructive feedback on aspects such as novelty, clarity, and theoretical soundness, covering sections like the problem statement, datasets, methodology, and results. This study explores whether large language models (LLMs) can reliably distinguish between exhaustive and trivial reviews, assessing the depth and relevance of reviewers’ feedback to help editors evaluate each review’s impact on decision-making. The rise of LLMs, such as Llama \(-\) 3.1, GPT-4, Mixtral-8x7b, and Gemma2-9b, introduces potential risks as reviewers increasingly rely on these models, which could compromise the integrity of the peer review process. To address this hypothesis, we propose a dataset named (InsightfulPeer) leveraging LLMs to classify reviews as exhaustive and trivial based on their thoroughness and actionable feedback. Using Chain of Thought (CoT) reasoning, we assess how well these classifications align with human evaluations. Both qualitative and quantitative analyses are performed to gauge the fairness and effectiveness of these LLM variants in executing this task. As our title, Not All Reviews Are Significant, suggests, we argue that exhaustive reviews—those providing detailed analysis and covering key sections and critical aspects of a paper offer deeper insights that aid in evaluating a paper’s contribution and reliability. By distinguishing these comprehensive (exhaustive) reviews from more superficial, or trivial, ones, we emphasize the added value that thorough peer assessment brings to supporting research integrity and scholarly rigor. However, we do not assert that exhaustiveness alone determines review quality; rather, it serves as one potential signal for estimating review quality. Our dataset and associated code are available at https://github.com/PrabhatkrBharti/InsightfulPeer-CoT.git to replicate our findings.
