REVIEW 13 cited by
ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Given the rapid ascent of large language models (LLMs), we study the question: (How) can large language models help in reviewing of scientific papers or proposals? We first conduct some pilot studies where we find that (i) GPT-4 outperforms other LLMs (Bard, Vicuna, Koala, Alpaca, LLaMa, Dolly, OpenAssistant, StableLM), and (ii) prompting with a specific question (e.g., to identify errors) outperforms prompting to simply write a review. With these insights, we study the use of LLMs (specifically, GPT-4) for three tasks: 1. Identifying errors: We construct 13 short computer science papers each with a deliberately inserted error, and ask the LLM to check for the correctness of these papers. We observe that the LLM finds errors in 7 of them, spanning both mathematical and conceptual errors. 2. Verifying checklists: We task the LLM to verify 16 closed-ended checklist questions in the respective sections of 15 NeurIPS 2022 papers. We find that across 119 {checklist question, paper} pairs, the LLM had an 86.6% accuracy. 3. Choosing the "better" paper: We generate 10 pairs of abstracts, deliberately designing each pair in such a way that one abstract was clearly superior than the other. The LLM, however, struggled to discern these relatively straightforward distinctions accurately, committing errors in its evaluations for 6 out of the 10 pairs. Based on these experiments, we think that LLMs have a promising use as reviewing assistants for specific reviewing tasks, but not (yet) for complete evaluations of papers or proposals.
Forward citations
Cited by 13 Pith papers
-
NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?
NatureBench evaluates ten frontier AI coding agents on 90 tasks from Nature papers under web-search-disabled conditions and finds the strongest agent surpasses published SOTA on only 17.8% of tasks, succeeding mainly ...
-
Stop Automating Peer Review Without Rigorous Evaluation
LLM paper reviewers show excessive agreement and are trivially gamed by zero-shot "paper laundering" rewrites, so they should not automate acceptance-relevant judgment without a science of evaluation.
-
AutoSupervision: Closing the Feedback Loop in Scientific Workflows with Grounded Revision Verification
On a new 8,790-instance benchmark built from Nature Communications review records, LLMs characterize reviewer concerns well (GPT-5.5: 0.754) but verify evidence-backed revision resolution poorly (best 0.501).
-
Can AI Evaluate AI Scientists? A Benchmarking Study of Autonomous Research Generation Systems Using Automated Multi-Model Review
In a benchmark of four AI Scientist systems on 15 FARS proposals, LLM reviewers rated FARS's own papers about twice as high as Sakana v1/v2, CycleResearcher, and Data-to-Paper outputs, but the evaluation lacks human v...
-
BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers?
An LLM agent generating fabricated papers without experiments gets acceptance-level scores from LLM reviewers up to 82% of the time, and simple integrity-checking mitigations barely beat random.
-
Can LLMs Identify Critical Limitations within Scientific Research? A Systematic Evaluation on AI Research Papers
LIMITGEN evaluates LLMs on identifying paper limitations and shows strong models still miss roughly half of obvious flaws, with retrieval augmentation giving modest but consistent gains.
-
Automatic Evaluation Metrics for Artificially Generated Scientific Research
A simple title-and-abstract model predicts citation counts better than review scores and outperforms LLM reviewers in matching human review scores, but remains below human consistency.
-
Are Today's LLMs Ready to Explain Well-Being Concepts?
AI judges can score explanations of well-being concepts, and small models fine-tuned with preference data score better than larger models, although judges and explainers are all AIs.
-
Automated Novelty Evaluation of Academic Paper: A Collaborative Approach Integrating Human and Large Language Model Knowledge
Method novelty prediction from peer-review novelty sentences and ChatGPT method summaries improves accuracy on ICLR 2022 data, but the benchmark leaks reviewer opinions into the input.
-
SC4ANM: Identifying Optimal Section Combinations for Automated Novelty Prediction in Academic Papers
Using ICLR 2022/2023 reviewer novelty scores as labels, the study reports that the introduction, results, and discussion sections form the most effective input combination for automated novelty prediction.
-
How Far Are AI Scientists from Changing the World?
This survey proposes a four-level capability framework for AI Scientist systems and, using an AI reviewer, finds that current systems produce papers rated well below normal scientific standards.
-
Position: The ML Community Must Build an AI-Augmented Peer-Review Ecosystem
The paper argues that AI-assisted peer review is an urgent priority and that its success depends on collecting richer, structured peer review process data.
-
OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models
The paper advocates protecting and leveraging OpenReview's peer review corpus as a community asset for LLM-based review assistance, benchmarks, and alignment.
Discussion (0). Continue with ORCID to comment.