REVIEW 5 major objections 5 minor 18 references
Can Large Language Models Be Trusted Paper Reviewers? A Feasibility Study
T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read This paper tests an LLM-based review pipeline against real conference decisions and finds that, despite large savings in time and cost, its accept list overlaps the human accept list by only 38.6 percent.
desk verdict A credible feasibility study with a real open-source system and useful failure-mode analyses, but the headline 38.6% agreement lacks the defined metric and baseline needed to support its conclusion. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing artifact is the review pipeline itself. Each paper is converted to Markdown, indexed in a vector store, and queried through retrieval-augmented generation by a large language model cast as a reviewer or chair; a prompt decomposes the task into seven fixed steps from layout check through comparative evaluation, scoring, and explanation. A multi-agent orchestration layer runs reviewer agents in parallel on batches and then has chair agents deliberate over the shortlist. The argument rides on two things the pipeline produces: the final accept list, whose overlap with the venue's real accept list is the headline number, and the two controlled probes that isolate why the lists diverge—a content-ablation study on one paper and a sentence-injection study measuring score shift. The 38.6 percent overlap is the number that carries the conclusion.
What would settle it
Re-run the identical pipeline on the same 290 papers with a prompt whose criteria are taken from the actual WASA 2024 review form, or from a rubric elicited from that year's program committee, and check whether the accept-list overlap rises substantially. If overlap jumps above, say, 70 percent, the 38.6 percent result was an artifact of the generic rubric rather than evidence that LLMs cannot judge papers; conversely, a similarly low overlap under rubric-matched conditions would confirm the paper's conclusion.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is a head-to-head comparison between an LLM review pipeline and human decisions on the same real corpus. Using GPT-4o with temperature set to zero, a first stage of reviewer agents followed by chair agents, and an evaluation rubric summarized from major publisher guidelines, the system produced an accept list that matched the WASA 2024 human accept list only 38.6 percent of the time on average over five runs; the first-round screening match was 59.6 percent. The paper argues this divergence is not random noise but the signature of two reproducible biases: when given only a paper's title, the model borrowed text from the wrong paper in the same batch or fabricated an answer, and adding flattering but data-free sentences to a paper's abstract and conclusion raised its average score from 85.8 to 88.4. Taking the human accept list as the authoritative reference, the paper concludes that LLM-based review cannot yet be trusted to decide what gets published.
Load-bearing premise
The comparison assumes that the review criteria loaded into the system prompt—summarized from general publisher guidelines—are the same criteria the WASA 2024 human reviewers actually applied; if the venue weighted novelty, experimental soundness, or topical fit differently, the low 38.6 percent overlap could reflect a rubric mismatch rather than a limit of LLM judgment.
Editorial extensions
If this is right
- A full LLM review pass over 290 submissions costs about 104 dollars and takes under three hours, making the pipeline viable as an inexpensive pre-screening or triage stage before human review.
- If LLM accept lists are used as final decisions today, they will diverge from human judgment by roughly 60 percent of the accepted set, so accept/reject authority should remain with human editors and program committees.
- The documented failures—cross-paper content confusion when input is thin, and score inflation from exaggerated but data-free language—give concrete engineering targets, such as document-isolated retrieval and prompts that require whole-paper synthesis rather than keyword snippets.
- Because the failure modes are tied to retrieval and prompt design, the open-source pipeline can be re-run with different models, rubrics, or retrieval structures to measure whether the 38.6 percent gap closes.
- Venues with very large submission volumes would gain the most from the time and cost savings, provided the system is framed as triage rather than evaluation.
Reading between the lines
- A missing comparison would sharpen the verdict: if a fresh set of human reviewers re-reviewed the same 290 papers and also overlapped the original accept list by roughly 40 percent, the 38.6 percent figure would reflect the inherent noise of peer review as much as any LLM-specific weakness.
- The content-ablation result—responses stayed nearly identical from abstract-only to full-paper input—suggests a cheap fix worth testing: feed the entire paper as one context instead of retrieved snippets, since the bottleneck may be the retrieval step's focus on abstract keywords rather than the model's reading ability.
- The score inflation from exaggerated language implies a gaming risk: authors who know an LLM is screening papers could inflate scores with grandiose, data-free claims, so any real deployment would need prompts designed to resist such gaming.
- The authors' proposed domain fine-tuning can be tested immediately on the same venue by training on earlier years' accept/reject decisions and measuring overlap on the 2024 corpus, since both the pipeline and the historical data already exist.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents an automated paper-review system that combines Retrieval Augmented Generation (RAG), the AutoGen multi-agent framework, and chain-of-thought prompting to emulate reviewer and chair roles. The system was run five times on all 290 WASA 2024 submissions using GPT-4o with temperature set to 0. The reported results are an average review time of 2.48 hours and an average cost of $104.28 per run, with an average similarity of 38.6% between the LLM final acceptance set and the human-accepted set. The authors interpret this low similarity as evidence that LLM-based reviewers are not yet reliable substitutes for human reviewers, and they present two supplementary experiments on content understanding and susceptibility to exaggerated language. The paper concludes that LLMs should serve as assistive tools, with final decisions left to domain experts.
Significance. If the evaluation were properly anchored, this would be a useful feasibility study: it uses real conference data, releases the implementation as open source, provides concrete cost and time measurements, and identifies concrete failure modes such as content confusion and retrieval preference. The repeated runs and the use of an external ground truth (WASA 2024 decisions) are also strengths. However, the central quantitative claim is currently uninterpretable because the similarity metric is never defined and no baseline—such as chance agreement, human-human re-review agreement, or the venue acceptance rate—is provided. The qualitative findings are suggestive but rest on very small samples without statistical testing. With the metric and baseline supplied, the study could support its conclusion; as written, the evidence does not yet establish that 38.6% is low.
major comments (5)
- [Section VI-A, Table I] The similarity measure used for FirstRoundSimilarity and FinalSimilarity is never defined. The numbers could be a Jaccard index, an overlap coefficient, or an accuracy, and the interpretation of the 38.6% figure changes accordingly. Please define the metric explicitly and report the WASA 2024 acceptance rate, the expected overlap under random selection, and a human-human re-review agreement baseline; without these anchors the headline statistic cannot be evaluated.
- [Section VI-A, Table I] Across the five runs, FinalSimilarity ranges from 26.32% to 50.88%. Reporting only the average hides this instability, and with n=5 and no confidence interval or significance test, the claim that 38.6% is 'low' is not statistically grounded. The manuscript should report the variance and, if possible, a confidence interval or a test against a chance baseline.
- [Section VI-B, Experiment 3, Table II] The susceptibility experiment uses one paper and five evaluations per condition; the 2.6-point difference between the original (85.8) and modified (88.4) versions is not tested for statistical significance. This is too thin to support Finding (III) as a general claim about LLM susceptibility to exaggerated language, and the paper should either provide more papers or explicitly frame this as an illustrative case study.
- [Section VI-B, Step 2] The review criteria used in the prompts are summarized from generic IEEE, Elsevier, Springer, and ACM guidelines, but the baseline consists of actual WASA 2024 decisions. If WASA used different criteria or different weights across criteria, the low similarity could reflect a rubric mismatch rather than an inherent deficiency in LLM judgment. The manuscript should compare the prompt criteria against the venue's actual review criteria or at least discuss this confound as a limitation.
- [Section VI-A, Algorithm 2] The system's format-checking step automatically rejects papers that fail format compliance, but the manuscript does not report how many papers were removed in each of the five runs. This number affects the denominator of the similarity calculation and the comparability of the five runs, so it should be disclosed alongside Table I.
minor comments (5)
- [Section I, reference [1]] The conference name 'International Conference on Learning Representatives' should be 'International Conference on Learning Representations'.
- [Algorithm 2] In line 5, the notation 'P P+ = P ′ i' is undefined; please use set-union notation and define P_P before first use.
- [Figure 3] The color-coding of semantically similar responses is difficult to read in grayscale, and the five experimental conditions are not clearly labeled in the figure; please annotate the conditions directly or provide the responses in a table.
- [Throughout the manuscript] The conference name appears as 'W ASA' with a spurious space in several places; the spacing should be corrected to 'WASA'.
- [Section VIII] The conclusion states that the overlap 'averages only 38.6%,' but does not restate the range across the five runs (26.32%–50.88%); including the range would give readers a more accurate picture of the result's stability.
Circularity Check
No circularity: empirical comparison against external human decisions is self-contained.
full rationale
The paper's central claim is an empirical comparison: the set of papers accepted by the LLM-based review system is compared with the set accepted by human reviewers at WASA 2024. This ground-truth set is external to the system and is not produced by any fitted parameter or by the system's own outputs, so the headline similarity statistic is not equivalent to an input by construction. The review criteria in the prompts are summarized from general IEEE/Elsevier/Springer/ACM guidelines before the experiments and do not encode the WASA acceptance labels, so there is no fitted-input-called-prediction pattern. The acknowledged subjectivity of human review and the undefined similarity metric are methodological weaknesses, not circularity. The only self-citation, reference [18], is used as a sample paper in a supplementary content-understanding experiment; it does not justify the paper's central premise or forbid alternative conclusions. No equation is defined in terms of the target result, and no load-bearing claim reduces to a self-citation chain. The paper's conclusion that LLM review is not yet a substitute rests directly on an external benchmark, so the analysis is self-contained and non-circular.
Assumptions & free parameters
assumptions (4)
- domain assumption Human acceptance decisions at WASA 2024 are an authoritative reference standard for review quality.
- domain assumption GPT-4o with temperature set to 0 is a representative and sufficiently stable LLM for the feasibility study.
- domain assumption The review criteria summarized from publisher guidelines approximate the criteria used by WASA 2024.
- domain assumption The similarity metric used in Table I measures meaningful agreement between paper sets.
Cite this review
Pith. "Pith review of Can Large Language Models Be Trusted Paper Reviewers? A Feasibility Study." pith.science (2026). https://pith.science/paper/3QLEHAIY
@misc{pith2026250617311,
author = {Pith},
title = {Pith review of: Can Large Language Models Be Trusted Paper Reviewers? A Feasibility Study},
year = {2026},
howpublished = {\url{https://pith.science/paper/3QLEHAIY}},
note = {Machine review of arXiv:2506.17311}
}
abstract
Academic paper review typically requires substantial time, expertise, and human resources. Large Language Models (LLMs) present a promising method for automating the review process due to their extensive training data, broad knowledge base, and relatively low usage cost. This work explores the feasibility of using LLMs for academic paper review by proposing an automated review system. The system integrates Retrieval Augmented Generation (RAG), the AutoGen multi-agent system, and Chain-of-Thought prompting to support tasks such as format checking, standardized evaluation, comment generation, and scoring. Experiments conducted on 290 submissions from the WASA 2024 conference using GPT-4o show that LLM-based review significantly reduces review time (average 2.48 hours) and cost (average \$104.28 USD). However, the similarity between LLM-selected papers and actual accepted papers remains low (average 38.6\%), indicating issues such as hallucination, lack of independent judgment, and retrieval preferences. Therefore, it is recommended to use LLMs as assistive tools to support human reviewers, rather than to replace them.
Figures
Reference graph
Works this paper leans on
-
[1]
ICLR, “Iclr,” https://iclr.cc/, 2025
work page 2025
-
[2]
On protecting the data privacy of large language models (llms) and llm agents: A literature review,
B. Yan, K. Li, M. Xu, Y . Dong, Y . Zhang, Z. Ren, and X. Cheng, “On protecting the data privacy of large language models (llms) and llm agents: A literature review,” High-Confidence Computing, vol. 5, no. 2, p. 100300, 2025. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2667295225000042
work page 2025
-
[3]
J. Xiong, M. Wei, Z. Lu, and Y . Liu, “Assessing the effectiveness of crawlers and large language models in detecting adversarial hidden link threats in meta computing,” High-Confidence Computing, p. 100292, 2024. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/S2667295224000953
work page 2024
-
[4]
AAAI, “Aaai news,” https://aaai.org/ aaai-launches-ai-powered-peer-review-assessment-system/, 2025
work page 2025
-
[5]
Chain-of-thought prompting elicits reasoning in large language models,
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022
2022
-
[6]
W. Liang, Y . Zhang, H. Cao, B. Wang, D. Y . Ding, X. Yang, K. V odra- halli, S. He, D. S. Smith, Y . Yin et al., “Can large language models provide useful feedback on research papers? a large-scale empirical analysis,” NEJM AI, vol. 1, no. 8, p. AIoa2400196, 2024
work page 2024
-
[7]
The ai review lottery: Widespread ai-assisted peer reviews boost paper scores and acceptance rates,
G. R. Latona, M. H. Ribeiro, T. R. Davidson, V . Veselovsky, and R. West, “The ai review lottery: Widespread ai-assisted peer reviews boost paper scores and acceptance rates,” arXiv preprint arXiv:2405.02150, 2024
arXiv 2024
-
[8]
Is llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks,
R. Zhou, L. Chen, and K. Yu, “Is llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024, pp. 9340–9351
work page 2024
Show all 18 references
-
[9]
Relevai-reviewer: A benchmark on ai reviewers for survey paper relevance,
P. H. Couto, Q. P. Ho, N. Kumari, B. K. Rachmat, T. G. H. Khuong, I. Ullah, and L. Sun-Hosoya, “Relevai-reviewer: A benchmark on ai reviewers for survey paper relevance,” arXiv preprint arXiv:2406.10294, 2024
2024
-
[10]
Llms assist nlp researchers: Critique paper (meta-) reviewing,
J. Du, Y . Wang, W. Zhao, Z. Deng, S. Liu, R. Lou, H. P. Zou, P. N. Venkit, N. Zhang, M. Srinathet al., “Llms assist nlp researchers: Critique paper (meta-) reviewing,” arXiv preprint arXiv:2406.16253, 2024
2024 arXiv
-
[11]
Retrieval-augmented generation for large language models: A survey,
Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, H. Wang, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997, vol. 2, 2023
2023 arXiv
-
[12]
A review on multimodal zero-shot learning,
W. Cao, Y . Wu, Y . Sun, H. Zhang, J. Ren, D. Gu, and X. Wang, “A review on multimodal zero-shot learning,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 13, no. 2, p. e1488, 2023
2023
-
[13]
A compre- hensive survey of few-shot learning: Evolution, applications, challenges, and opportunities,
Y . Song, T. Wang, P. Cai, S. K. Mondal, and J. P. Sahoo, “A compre- hensive survey of few-shot learning: Evolution, applications, challenges, and opportunities,” ACM Computing Surveys, vol. 55, no. 13s, pp. 1–40, 2023
2023
-
[14]
Autogen,
Microsoft, “Autogen,” https://microsoft.github.io/autogen/stable//index. html, 2025
2025
-
[15]
MinerU, “Mineru,” https://github.com/opendatalab/MinerU/blob/master/ README zh-CN.md, 2025
2025
-
[16]
A taxonomy of ddos attack and ddos defense mechanisms,
J. Mirkovic and P. Reiher, “A taxonomy of ddos attack and ddos defense mechanisms,” ACM SIGCOMM Computer Communication Review, vol. 34, no. 2, pp. 39–53, 2004
2004
-
[17]
Wasa2024,
W ASA, “Wasa2024,” http://wasa-conference.org/W ASA2024/, 2024
2024
-
[18]
Anonymity on byzantine-resilient decentralized computing,
K. Ma, M. Xu, Y . Guo, L. Cui, S. Ni, S. Zhang, W. Wang, H. Yang, and X. Cheng, “Anonymity on byzantine-resilient decentralized computing,” in International Conference on Wireless Artificial Intelligent Computing Systems and Applications. Springer, 2024, pp. 400–412
2024
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.