REVIEW 6 cited by
ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviews
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Academic paper review is a critical yet time-consuming task within the research community. With the increasing volume of academic publications, automating the review process has become a significant challenge. The primary issue lies in generating comprehensive, accurate, and reasoning-consistent review comments that align with human reviewers' judgments. In this paper, we address this challenge by proposing ReviewAgents, a framework that leverages large language models (LLMs) to generate academic paper reviews. We first introduce a novel dataset, Review-CoT, consisting of 142k review comments, designed for training LLM agents. This dataset emulates the structured reasoning process of human reviewers-summarizing the paper, referencing relevant works, identifying strengths and weaknesses, and generating a review conclusion. Building upon this, we train LLM reviewer agents capable of structured reasoning using a relevant-paper-aware training method. Furthermore, we construct ReviewAgents, a multi-role, multi-LLM agent review framework, to enhance the review comment generation process. Additionally, we propose ReviewBench, a benchmark for evaluating the review comments generated by LLMs. Our experimental results on ReviewBench demonstrate that while existing LLMs exhibit a certain degree of potential for automating the review process, there remains a gap when compared to human-generated reviews. Moreover, our ReviewAgents framework further narrows this gap, outperforming advanced LLMs in generating review comments.
Forward citations
Cited by 6 Pith papers
-
ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchs
ARAC-Bench introduces a three-stage researcher-mimicking benchmark that scores autonomous research systems on proposal, experiment, and synthesis quality; evaluated systems reach at most 67.9 out of 100, and scores co...
-
RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Review
A rubric-first LLM pipeline that splits peer review into rubric generation, rubric-conditioned review writing, and final scoring outperforms existing AI reviewers on alignment with human judgments in a 200-paper test.
-
From peer review nuances to best practices
Paper version, score version, and input format vary across peer-review datasets and measurably affect LLM-based review experiments, so they should be reported explicitly.
-
AI for Auto-Research: Roadmap & User Guide
The paper delivers a stage-by-stage roadmap for AI in research, showing reliable assistance in retrieval and tool tasks but fragility in novelty and judgment, advocating human-governed collaboration.
-
How Far Are AI Scientists from Changing the World?
This survey proposes a four-level capability framework for AI Scientist systems and, using an AI reviewer, finds that current systems produce papers rated well below normal scientific standards.
-
AI4Research: A Survey of Artificial Intelligence for Scientific Research
A survey that organizes AI-for-research work into five tasks, comprehension, survey, discovery, writing, and peer review, and compiles associated tools and benchmarks.
Discussion (0). Continue with ORCID to comment.