Pith. sign in

REVIEW 5 major objections 5 minor 18 references

Can Large Language Models Be Trusted Paper Reviewers? A Feasibility Study

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read This paper tests an LLM-based review pipeline against real conference decisions and finds that, despite large savings in time and cost, its accept list overlaps the human accept list by only 38.6 percent.

desk verdict A credible feasibility study with a real open-source system and useful failure-mode analyses, but the headline 38.6% agreement lacks the defined metric and baseline needed to support its conclusion. read the letter →

arxiv 2506.17311 v1 pith:3QLEHAIY submitted 2025-06-18 cs.CY

classification cs.CY
keywords LLMpaperreviewpeerautomationretrievalaugmentedgenerationmulti-agentsystemchain-of-thoughtpromptingacceptanceoverlaphallucinationreviewerbias
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether an LLM-based system can take over academic peer review. The authors build a pipeline that combines retrieval-augmented generation, multiple interacting agents, and a seven-step chain-of-thought prompt, and run it on all 290 real submissions to a 2024 wireless-computing conference. The system is far cheaper and faster than human review—about 2.48 hours and 104 dollars for a full pass—but the papers it accepts overlap the conference's actual accept list by only 38.6 percent on average. Two follow-up experiments trace the gap to specific failure modes: the model confuses different papers and invents content when information is thin, and its scores rise when a paper carries exaggerated, data-free claims. The paper's conclusion is that LLM review should assist human reviewers, not replace them.

What carries the argument

The load-bearing artifact is the review pipeline itself. Each paper is converted to Markdown, indexed in a vector store, and queried through retrieval-augmented generation by a large language model cast as a reviewer or chair; a prompt decomposes the task into seven fixed steps from layout check through comparative evaluation, scoring, and explanation. A multi-agent orchestration layer runs reviewer agents in parallel on batches and then has chair agents deliberate over the shortlist. The argument rides on two things the pipeline produces: the final accept list, whose overlap with the venue's real accept list is the headline number, and the two controlled probes that isolate why the lists diverge—a content-ablation study on one paper and a sentence-injection study measuring score shift. The 38.6 percent overlap is the number that carries the conclusion.

What would settle it

Re-run the identical pipeline on the same 290 papers with a prompt whose criteria are taken from the actual WASA 2024 review form, or from a rubric elicited from that year's program committee, and check whether the accept-list overlap rises substantially. If overlap jumps above, say, 70 percent, the 38.6 percent result was an artifact of the generic rubric rather than evidence that LLMs cannot judge papers; conversely, a similarly low overlap under rubric-matched conditions would confirm the paper's conclusion.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a head-to-head comparison between an LLM review pipeline and human decisions on the same real corpus. Using GPT-4o with temperature set to zero, a first stage of reviewer agents followed by chair agents, and an evaluation rubric summarized from major publisher guidelines, the system produced an accept list that matched the WASA 2024 human accept list only 38.6 percent of the time on average over five runs; the first-round screening match was 59.6 percent. The paper argues this divergence is not random noise but the signature of two reproducible biases: when given only a paper's title, the model borrowed text from the wrong paper in the same batch or fabricated an answer, and adding flattering but data-free sentences to a paper's abstract and conclusion raised its average score from 85.8 to 88.4. Taking the human accept list as the authoritative reference, the paper concludes that LLM-based review cannot yet be trusted to decide what gets published.

Load-bearing premise

The comparison assumes that the review criteria loaded into the system prompt—summarized from general publisher guidelines—are the same criteria the WASA 2024 human reviewers actually applied; if the venue weighted novelty, experimental soundness, or topical fit differently, the low 38.6 percent overlap could reflect a rubric mismatch rather than a limit of LLM judgment.

Editorial extensions

If this is right

  • A full LLM review pass over 290 submissions costs about 104 dollars and takes under three hours, making the pipeline viable as an inexpensive pre-screening or triage stage before human review.
  • If LLM accept lists are used as final decisions today, they will diverge from human judgment by roughly 60 percent of the accepted set, so accept/reject authority should remain with human editors and program committees.
  • The documented failures—cross-paper content confusion when input is thin, and score inflation from exaggerated but data-free language—give concrete engineering targets, such as document-isolated retrieval and prompts that require whole-paper synthesis rather than keyword snippets.
  • Because the failure modes are tied to retrieval and prompt design, the open-source pipeline can be re-run with different models, rubrics, or retrieval structures to measure whether the 38.6 percent gap closes.
  • Venues with very large submission volumes would gain the most from the time and cost savings, provided the system is framed as triage rather than evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A missing comparison would sharpen the verdict: if a fresh set of human reviewers re-reviewed the same 290 papers and also overlapped the original accept list by roughly 40 percent, the 38.6 percent figure would reflect the inherent noise of peer review as much as any LLM-specific weakness.
  • The content-ablation result—responses stayed nearly identical from abstract-only to full-paper input—suggests a cheap fix worth testing: feed the entire paper as one context instead of retrieved snippets, since the bottleneck may be the retrieval step's focus on abstract keywords rather than the model's reading ability.
  • The score inflation from exaggerated language implies a gaming risk: authors who know an LLM is screening papers could inflate scores with grandiose, data-free claims, so any real deployment would need prompts designed to resist such gaming.
  • The authors' proposed domain fine-tuning can be tested immediately on the same venue by training on earlier years' accept/reject decisions and measuring overlap on the 2024 corpus, since both the pipeline and the historical data already exist.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The manuscript presents an automated paper-review system that combines Retrieval Augmented Generation (RAG), the AutoGen multi-agent framework, and chain-of-thought prompting to emulate reviewer and chair roles. The system was run five times on all 290 WASA 2024 submissions using GPT-4o with temperature set to 0. The reported results are an average review time of 2.48 hours and an average cost of $104.28 per run, with an average similarity of 38.6% between the LLM final acceptance set and the human-accepted set. The authors interpret this low similarity as evidence that LLM-based reviewers are not yet reliable substitutes for human reviewers, and they present two supplementary experiments on content understanding and susceptibility to exaggerated language. The paper concludes that LLMs should serve as assistive tools, with final decisions left to domain experts.

Significance. If the evaluation were properly anchored, this would be a useful feasibility study: it uses real conference data, releases the implementation as open source, provides concrete cost and time measurements, and identifies concrete failure modes such as content confusion and retrieval preference. The repeated runs and the use of an external ground truth (WASA 2024 decisions) are also strengths. However, the central quantitative claim is currently uninterpretable because the similarity metric is never defined and no baseline—such as chance agreement, human-human re-review agreement, or the venue acceptance rate—is provided. The qualitative findings are suggestive but rest on very small samples without statistical testing. With the metric and baseline supplied, the study could support its conclusion; as written, the evidence does not yet establish that 38.6% is low.

major comments (5)
  1. [Section VI-A, Table I] The similarity measure used for FirstRoundSimilarity and FinalSimilarity is never defined. The numbers could be a Jaccard index, an overlap coefficient, or an accuracy, and the interpretation of the 38.6% figure changes accordingly. Please define the metric explicitly and report the WASA 2024 acceptance rate, the expected overlap under random selection, and a human-human re-review agreement baseline; without these anchors the headline statistic cannot be evaluated.
  2. [Section VI-A, Table I] Across the five runs, FinalSimilarity ranges from 26.32% to 50.88%. Reporting only the average hides this instability, and with n=5 and no confidence interval or significance test, the claim that 38.6% is 'low' is not statistically grounded. The manuscript should report the variance and, if possible, a confidence interval or a test against a chance baseline.
  3. [Section VI-B, Experiment 3, Table II] The susceptibility experiment uses one paper and five evaluations per condition; the 2.6-point difference between the original (85.8) and modified (88.4) versions is not tested for statistical significance. This is too thin to support Finding (III) as a general claim about LLM susceptibility to exaggerated language, and the paper should either provide more papers or explicitly frame this as an illustrative case study.
  4. [Section VI-B, Step 2] The review criteria used in the prompts are summarized from generic IEEE, Elsevier, Springer, and ACM guidelines, but the baseline consists of actual WASA 2024 decisions. If WASA used different criteria or different weights across criteria, the low similarity could reflect a rubric mismatch rather than an inherent deficiency in LLM judgment. The manuscript should compare the prompt criteria against the venue's actual review criteria or at least discuss this confound as a limitation.
  5. [Section VI-A, Algorithm 2] The system's format-checking step automatically rejects papers that fail format compliance, but the manuscript does not report how many papers were removed in each of the five runs. This number affects the denominator of the similarity calculation and the comparability of the five runs, so it should be disclosed alongside Table I.
minor comments (5)
  1. [Section I, reference [1]] The conference name 'International Conference on Learning Representatives' should be 'International Conference on Learning Representations'.
  2. [Algorithm 2] In line 5, the notation 'P P+ = P ′ i' is undefined; please use set-union notation and define P_P before first use.
  3. [Figure 3] The color-coding of semantically similar responses is difficult to read in grayscale, and the five experimental conditions are not clearly labeled in the figure; please annotate the conditions directly or provide the responses in a table.
  4. [Throughout the manuscript] The conference name appears as 'W ASA' with a spurious space in several places; the spacing should be corrected to 'WASA'.
  5. [Section VIII] The conclusion states that the overlap 'averages only 38.6%,' but does not restate the range across the five runs (26.32%–50.88%); including the range would give readers a more accurate picture of the result's stability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical comparison against external human decisions is self-contained.

full rationale

The paper's central claim is an empirical comparison: the set of papers accepted by the LLM-based review system is compared with the set accepted by human reviewers at WASA 2024. This ground-truth set is external to the system and is not produced by any fitted parameter or by the system's own outputs, so the headline similarity statistic is not equivalent to an input by construction. The review criteria in the prompts are summarized from general IEEE/Elsevier/Springer/ACM guidelines before the experiments and do not encode the WASA acceptance labels, so there is no fitted-input-called-prediction pattern. The acknowledged subjectivity of human review and the undefined similarity metric are methodological weaknesses, not circularity. The only self-citation, reference [18], is used as a sample paper in a supplementary content-understanding experiment; it does not justify the paper's central premise or forbid alternative conclusions. No equation is defined in terms of the target result, and no load-bearing claim reduces to a self-citation chain. The paper's conclusion that LLM review is not yet a substitute rests directly on an external benchmark, so the analysis is self-contained and non-circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical systems study; it introduces no free parameters or invented entities. Its central claim rests on the domain assumptions above, especially the equivalence between the system's evaluation criteria and the venue's actual criteria.

assumptions (4)
  • domain assumption Human acceptance decisions at WASA 2024 are an authoritative reference standard for review quality.
    Stated in Section VI: 'the decisions made by human reviewers in practice are typically regarded as more authoritative and thus serve as a valuable reference.'
  • domain assumption GPT-4o with temperature set to 0 is a representative and sufficiently stable LLM for the feasibility study.
    Section VI describes the use of GPT-4o and temperature 0; the paper generalizes findings to LLMs.
  • domain assumption The review criteria summarized from publisher guidelines approximate the criteria used by WASA 2024.
    Section VI, Step 2: the authors summarize criteria from IEEE, Elsevier, Springer, and ACM, but do not verify they match WASA's actual review form.
  • domain assumption The similarity metric used in Table I measures meaningful agreement between paper sets.
    The metric is not formally defined in the paper, and no baseline (random selection, inter-human agreement) is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Large Language Models Be Trusted Paper Reviewers? A Feasibility Study." pith.science (2026). https://pith.science/paper/3QLEHAIY

@misc{pith2026250617311,
  author       = {Pith},
  title        = {Pith review of: Can Large Language Models Be Trusted Paper Reviewers? A Feasibility Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3QLEHAIY}},
  note         = {Machine review of arXiv:2506.17311}
}
abstract

Academic paper review typically requires substantial time, expertise, and human resources. Large Language Models (LLMs) present a promising method for automating the review process due to their extensive training data, broad knowledge base, and relatively low usage cost. This work explores the feasibility of using LLMs for academic paper review by proposing an automated review system. The system integrates Retrieval Augmented Generation (RAG), the AutoGen multi-agent system, and Chain-of-Thought prompting to support tasks such as format checking, standardized evaluation, comment generation, and scoring. Experiments conducted on 290 submissions from the WASA 2024 conference using GPT-4o show that LLM-based review significantly reduces review time (average 2.48 hours) and cost (average \$104.28 USD). However, the similarity between LLM-selected papers and actual accepted papers remains low (average 38.6\%), indicating issues such as hallucination, lack of independent judgment, and retrieval preferences. Therefore, it is recommended to use LLMs as assistive tools to support human reviewers, rather than to replace them.

Figures

Figures reproduced from arXiv: 2506.17311 by the authors.

Figure 1
Figure 1. The workflow of the automated paper review system [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Prompt template essential for review tasks, which require consistent, content￾driven analysis and scoring rather than open-ended or imag￾inative generation. The runtime performance of the review system is primarily determined by the response latency of the LLM server, rather than the local execution environment. Consequently, the system requires only a Python runtime environment. In this work, we do not aim to asses… view at source ↗
Figure 3
Figure 3. Variations in responses based on input information [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 12 canonical work pages

  1. [1]

    ICLR, “Iclr,” https://iclr.cc/, 2025

  2. [2]

    On protecting the data privacy of large language models (llms) and llm agents: A literature review,

    B. Yan, K. Li, M. Xu, Y . Dong, Y . Zhang, Z. Ren, and X. Cheng, “On protecting the data privacy of large language models (llms) and llm agents: A literature review,” High-Confidence Computing, vol. 5, no. 2, p. 100300, 2025. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2667295225000042

  3. [3]

    Assessing the effectiveness of crawlers and large language models in detecting adversarial hidden link threats in meta computing,

    J. Xiong, M. Wei, Z. Lu, and Y . Liu, “Assessing the effectiveness of crawlers and large language models in detecting adversarial hidden link threats in meta computing,” High-Confidence Computing, p. 100292, 2024. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/S2667295224000953

  4. [4]

    Aaai news,

    AAAI, “Aaai news,” https://aaai.org/ aaai-launches-ai-powered-peer-review-assessment-system/, 2025

  5. [5]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

  6. [6]

    Can large language models provide useful feedback on research papers? a large-scale empirical analysis,

    W. Liang, Y . Zhang, H. Cao, B. Wang, D. Y . Ding, X. Yang, K. V odra- halli, S. He, D. S. Smith, Y . Yin et al., “Can large language models provide useful feedback on research papers? a large-scale empirical analysis,” NEJM AI, vol. 1, no. 8, p. AIoa2400196, 2024

  7. [7]

    The ai review lottery: Widespread ai-assisted peer reviews boost paper scores and acceptance rates,

    G. R. Latona, M. H. Ribeiro, T. R. Davidson, V . Veselovsky, and R. West, “The ai review lottery: Widespread ai-assisted peer reviews boost paper scores and acceptance rates,” arXiv preprint arXiv:2405.02150, 2024

  8. [8]

    Is llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks,

    R. Zhou, L. Chen, and K. Yu, “Is llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024, pp. 9340–9351

Show all 18 references
  1. [9]

    Relevai-reviewer: A benchmark on ai reviewers for survey paper relevance,

    P. H. Couto, Q. P. Ho, N. Kumari, B. K. Rachmat, T. G. H. Khuong, I. Ullah, and L. Sun-Hosoya, “Relevai-reviewer: A benchmark on ai reviewers for survey paper relevance,” arXiv preprint arXiv:2406.10294, 2024

  2. [10]

    Llms assist nlp researchers: Critique paper (meta-) reviewing,

    J. Du, Y . Wang, W. Zhao, Z. Deng, S. Liu, R. Lou, H. P. Zou, P. N. Venkit, N. Zhang, M. Srinathet al., “Llms assist nlp researchers: Critique paper (meta-) reviewing,” arXiv preprint arXiv:2406.16253, 2024

  3. [11]

    Retrieval-augmented generation for large language models: A survey,

    Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, H. Wang, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997, vol. 2, 2023

  4. [12]

    A review on multimodal zero-shot learning,

    W. Cao, Y . Wu, Y . Sun, H. Zhang, J. Ren, D. Gu, and X. Wang, “A review on multimodal zero-shot learning,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 13, no. 2, p. e1488, 2023

  5. [13]

    A compre- hensive survey of few-shot learning: Evolution, applications, challenges, and opportunities,

    Y . Song, T. Wang, P. Cai, S. K. Mondal, and J. P. Sahoo, “A compre- hensive survey of few-shot learning: Evolution, applications, challenges, and opportunities,” ACM Computing Surveys, vol. 55, no. 13s, pp. 1–40, 2023

  6. [14]

    Autogen,

    Microsoft, “Autogen,” https://microsoft.github.io/autogen/stable//index. html, 2025

  7. [15]

    MinerU, “Mineru,” https://github.com/opendatalab/MinerU/blob/master/ README zh-CN.md, 2025

  8. [16]

    A taxonomy of ddos attack and ddos defense mechanisms,

    J. Mirkovic and P. Reiher, “A taxonomy of ddos attack and ddos defense mechanisms,” ACM SIGCOMM Computer Communication Review, vol. 34, no. 2, pp. 39–53, 2004

  9. [17]

    Wasa2024,

    W ASA, “Wasa2024,” http://wasa-conference.org/W ASA2024/, 2024

  10. [18]

    Anonymity on byzantine-resilient decentralized computing,

    K. Ma, M. Xu, Y . Guo, L. Cui, S. Ni, S. Zhang, W. Wang, H. Yang, and X. Cheng, “Anonymity on byzantine-resilient decentralized computing,” in International Conference on Wireless Artificial Intelligent Computing Systems and Applications. Springer, 2024, pp. 400–412

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.