Pith. sign in

REVIEW 3 major objections 7 minor 3 cited by

OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews

T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read OpenReviewer, an open-source system built on a fine-tuned 8B model, produces reviews whose recommendations match human reviewers far more closely than GPT-4o does.

desk verdict Useful open system, but the headline metric does not beat a constant predictor, so the central claim is unsupported as written. read the letter →

arxiv 2412.11948 v3 pith:ZJGAFUNW submitted 2024-12-16 cs.AI

Maximilian Idahl , Zahra Ahmadi This is my paper · ORCID
classification cs.AI
keywords automatedpeerreviewLLMfine-tuningrecommendationcalibrationcriticalfeedbackgenerationOpendatasetLlama-3.1LLM-as-a-judgePDF-to-markdownconversion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OpenReviewer is an open-source system that generates peer reviews of machine-learning papers. The paper's central claim is that an 8-billion-parameter model fine-tuned on 79,000 expert reviews from top conferences produces markedly more critical and realistic reviews than general-purpose models such as GPT-4o and Claude-3.5. On 400 held-out papers, OpenReviewer's recommendation matched at least one human reviewer 55.5% of the time, with an average error of 0.96 on a 1–10 scale, whereas GPT-4o matched only 23.8% with an average error of 2.34. The authors position the system as a pre-submission feedback tool, not a replacement for human peer review. If the result holds, it shows that specialized fine-tuning on peer-review data can overcome the overly positive bias of general-purpose LLMs.

What carries the argument

The central object is Llama-OpenReviewer-8B, a full fine-tune of Llama-3.1-8B-Instruct on roughly 79,000 high-confidence reviews of ICLR and NeurIPS papers from 2022 onward. The supporting pipeline extracts paper text from PDFs using the Marker converter, keeps the main text and references, and conditions generation with a system prompt based on the ICLR 2024 reviewer guide plus a venue-specific review template. The recommendation score, normalized to a 1–10 scale, is the main quantitative output and the basis for the alignment metrics: exact match with at least one human reviewer and average absolute distance from the human reviewers' mean.

What would settle it

A blinded study in which expert reviewers rate the factual correctness and technical depth of OpenReviewer's criticisms against those of GPT-4o on the same 400 papers would decide whether the higher alignment with human ratings reflects genuine critical quality or stylistic mimicry.

Watch

Extended reading notes

Core claim

The paper establishes that review generation is a task where specialized fine-tuning beats scale. On a test set of 400 papers from NeurIPS 2024 and ICLR 2025, OpenReviewer's recommendations match the distribution of human reviewers almost exactly: its average rating is 5.4 out of 10, identical to the human average, while Llama-3.1-8B, Llama-3.1-70B, Claude-3.5-Sonnet, and GPT-4o produce averages between 6.9 and 8.1. In an LLM-as-a-judge preference test, OpenReviewer is judged to align better with expert reviews than GPT-4o in 60% of comparisons and better than Llama-3.1-70B in 76% of comparisons. The authors conclude that general-purpose LLMs are insufficiently critical, and that training on expert review data shifts both the tone and the calibration of generated reviews toward human judgments.

Load-bearing premise

The evaluation treats similarity to human-written reviews as equivalent to review quality, so if human reviews are noisy, biased, or mistaken, OpenReviewer could reproduce those flaws while appearing realistic.

Editorial extensions

If this is right

  • OpenReviewer's recommendation distribution matches the human average (5.4 vs. 5.4), while general-purpose LLMs average between 6.9 and 8.1, so authors get realistic expectations about how their paper will be received.
  • The exact-match rate of 55.5% suggests that a fine-tuned open 8B model can approach human reviewer calibration, making large closed models unnecessary for this task.
  • Because the model was trained only on ICLR and NeurIPS reviews from 2022 onward, its behavior on other venues or fields is unproven until retrained.
  • The arena evaluation, with win rates of 60–76% against the baselines, indicates that the preference signal is consistent across different comparison partners.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If review alignment is treated as a trainable calibration property, the same fine-tuning recipe could be applied to other judgment tasks that currently suffer from LLM positivity bias, such as grant review or hiring recommendation.
  • Because the training reviews come from a small set of elite venues, the model may encode those venues' reviewing norms; a testable prediction is that its recommendations would shift if trained on reviews from journals or lower-tier conferences with different acceptance cultures.
  • The 55.5% exact-match ceiling may reflect inherent human reviewer disagreement; a useful extension would be to measure how often two human reviewers agree on the same paper, providing an upper bound for any automated system.
  • A concrete deployment risk the paper does not test: authors who act on OpenReviewer feedback without human oversight may over-trust its suggestions; a user study measuring manuscript improvement with and without the tool would quantify this.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper presents OpenReviewer, an open-source system that converts a PDF paper into a structured review using a fine-tuned Llama-3.1-8B-Instruct model trained on roughly 79,000 human reviews from ICLR and NeurIPS. The central claim is that OpenReviewer produces considerably more critical and realistic reviews than general-purpose LLMs, supported by two evaluations on 400 test papers: (1) recommendation matching, where OpenReviewer achieves 55.5% exact match (i.e., matching at least one human reviewer's rating) and an average absolute error of 0.96 on a 1–10 scale, versus 23.8% and 2.34 for GPT-4o; and (2) an LLM-as-a-judge arena where GPT-4o prefers OpenReviewer's reviews over baselines in 60–76% of cases. The paper also describes the system architecture, prompt design, training details, and a live demo.

Significance. If the reported alignment is robust and content-dependent, the contribution is significant: an open-source, fine-tuned 8B model that can provide pre-submission feedback with critical realism, potentially useful for authors and as a research platform. The paper's open release of the model, prompts, and training configuration is a strength, as is its candid discussion of limitations. However, the evaluation currently leaves open the possibility that the recommendation-matching results reflect only the marginal rating distribution, not paper-specific judgment. The arena evaluation is also limited by the reliability of the GPT-4o judge. The core methodology is transparent and reproducible, but the evidence for the central claim is not yet conclusive without additional baseline and correlation analyses.

major comments (3)
  1. [Section 4.1, Tables 1 and 2]
  2. [Section 4 (test set construction)]
  3. [Section 4.2 (Review Arena)]
minor comments (7)
  1. [Table 1]
  2. [Section 4.1]
  3. [Section 5, Ethical Considerations]
  4. [Section 3.4]
  5. [Section 3.2.1]
  6. [Section 4]
  7. [Figure 2]

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: fine-tuning on human reviews and evaluating against held-out human reviews is a standard supervised setup, though the headline metric lacks a content-free baseline.

full rationale

The paper does not exhibit a circular derivation chain. OpenReviewer is fine-tuned on 79K human reviews and then evaluated on 400 held-out papers from the same venues, comparing its generated recommendations to the human reviewers' recommendations. This is a standard supervised learning evaluation, not a case where a fitted parameter is renamed as a prediction or where a result reduces to its input by construction. The paper explicitly acknowledges the key assumption: "This approach assumes that similarity to human reviews equals quality, which may not always be accurate as the quality control for human-written reviews is limited" (Section 4). That is a validity limitation, not circularity. The skeptical concern that a constant predictor outputting the human mean (5.4) would achieve the same average error (0.96) is a serious evaluation-design gap: the reported metrics do not demonstrate content-dependent alignment, and the paper lacks a trivial-baseline comparison or per-paper correlation. However, this is a missing-baseline / statistical-validity issue, not a circularity of the kind defined by the analysis protocol. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The central comparative claim against general-purpose LLMs is an empirical result that could have failed, so the paper is not circular.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim relies on several domain assumptions about data quality, conversion fidelity, metric validity, and judge impartiality. These are reasonable starting points but are not independently verified, so the evidence for the claim is only as strong as these assumptions.

assumptions (5)
  • domain assumption The collected corpus of 79K reviews from ICLR and NeurIPS (2022 onward) is representative of expert review quality and contains the information needed to learn critical reviewing.
    The entire fine-tuning premise depends on this corpus being high-quality and representative; the paper filters by confidence threshold but does not validate the content of the reviews.
  • domain assumption The PDF-to-markdown conversion via Marker preserves enough technical content for review generation.
    Section 3.2.1 states they discard appendix content and rely on conversion accuracy; errors could distort generated reviews.
  • domain assumption Normalizing venue-specific recommendation scores to a common 1-10 scale preserves meaningful comparability.
    Section 4.1 normalizes scores but does not justify the mapping across venues with different scales.
  • domain assumption The 400 held-out test papers from NeurIPS 2024 and ICLR 2025 do not overlap with the training data and are representative of the target distribution.
    The paper states they are held out but does not describe the sampling or verify no leakage.
  • ad hoc to paper GPT-4o acting as a judge provides a valid and unbiased evaluation of review alignment with expert reviews.
    Section 4.2 uses GPT-4o, which is also a baseline model, as the judge; this could introduce systematic preference for its own style, and no validation of the judge is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews." pith.science (2026). https://pith.science/paper/ZJGAFUNW

@misc{pith2026241211948,
  author       = {Pith},
  title        = {Pith review of: OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZJGAFUNW}},
  note         = {Machine review of arXiv:2412.11948}
}
read the original abstract

We present OpenReviewer, an open-source system for generating high-quality peer reviews of machine learning and AI conference papers. At its core is Llama-OpenReviewer-8B, an 8B parameter language model specifically fine-tuned on 79,000 expert reviews from top conferences. Given a PDF paper submission and review template as input, OpenReviewer extracts the full text, including technical content like equations and tables, and generates a structured review following conference-specific guidelines. Our evaluation on 400 test papers shows that OpenReviewer produces considerably more critical and realistic reviews compared to general-purpose LLMs like GPT-4 and Claude-3.5. While other LLMs tend toward overly positive assessments, OpenReviewer's recommendations closely match the distribution of human reviewer ratings. The system provides authors with rapid, constructive feedback to improve their manuscripts before submission, though it is not intended to replace human peer review. OpenReviewer is available as an online demo and open-source tool.

Figures

Figures reproduced from arXiv: 2412.11948 by the authors.

Figure 1
Figure 1. Annotated screenshot of the OpenReviewer demo hosted on Huggingface Spaces, with slightly modified layout. 1) Dialogue for uploading a PDF file. 2) Once the user uploads a file, this text field will be populated with the papers’ full text in markdown format. The user can choose to edit the text to fix conversion errors. 3) An accordion element to show and optionally edit the review template used for generation. 4) B… view at source ↗
Figure 2
Figure 2. Preference evaluation using GPT-4o as the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. System prompt used by OpenReviewer. Fields in {} are placeholders. [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: System prompt for the LLM judge. Fields in {} are placeholders. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: User prompt for the LLM judge. Fields in {} are placeholders. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Example output from the LLM judge. Review A is generated by OpenReviewer, and Review B is from [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: The axolotl configuration used to train Llama-OpenReviewer-8B, including all hyperparameters. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. How Far Are AI Scientists from Changing the World?

    cs.AI 2025-07 conditional novelty 4.0 of 10

    This survey proposes a four-level capability framework for AI Scientist systems and, using an AI reviewer, finds that current systems produce papers rated well below normal scientific standards.

  2. AI and the Future of Academic Peer Review

    cs.CY 2025-09 conditional novelty 3.0 of 10

    AI-assisted peer review is plausible as a supervised, task-specific supplement to human review, provided transparent governance and human accountability are built in.

  3. Large language models for automated scholarly paper review: A survey

    cs.AI 2025-01 conditional novelty 3.0 of 10

    A survey of LLM-based automated scholarly paper review, cataloging models, datasets, methods, and publisher policies as of 2023-2024.

Reference graph

Works this paper leans on

15 extracted references · 11 canonical work pages · cited by 3 Pith papers

  1. [2]

    arXiv preprint arXiv:2402.10886

    Reviewer2: Optimizing review genera- tion through prompt generation. arXiv preprint arXiv:2402.10886. Bas Hofstra, Vivek V . Kulkarni, Sebastian Munoz-Najar Galvez, Bryan He, Dan Jurafsky, and Daniel A. Mc- Farland. 2020. The diversity–innovation paradox in science. Proceedings of the National Academy of Sciences, 117(17):9284–9291. Pin-Lun Hsu, Yun Dai, ...

    arXiv 2020

  2. [3]

    **Contribution**: - Rating: 3 - Matches the expert reviews, which consistently rated contribution as 3

    work page

  3. [4]

    However, Review A does not mention the gradient conflict resolution or the toy example, which were noted as strengths in the expert reviews

    **Strengths**: - Review A highlights the clarity of the paper, the theoretical rigor, and the motivation for the problem, which aligns with the expert reviews’ emphasis on the novelty of the epigraph form, theoretical guarantees, and empirical results. However, Review A does not mention the gradient conflict resolution or the toy example, which were noted...

    work page

  4. [6]

    These are relevant but do not fully align with the expert reviews, which also asked about the gap between transition kernels, assumptions, and scaling with state space size

    **Questions**: - The questions in Review A focus on extending the algorithm to continuous spaces, dealing with infinite uncertainty sets, and computational efficiency. These are relevant but do not fully align with the expert reviews, which also asked about the gap between transition kernels, assumptions, and scaling with state space size

    work page

  5. [7]

    The other two expert reviews rated it as 8 and 3, so this rating is within the range of expert opinions

    **Rating**: - Rating: 6 - Matches two of the expert reviews, which rated the paper as 6. The other two expert reviews rated it as 8 and 3, so this rating is within the range of expert opinions. #### **Review B** 1. **Soundness**: - Rating: 4 - Matches one expert review that rated soundness as 4, but is higher than the other three expert reviews, which rat...

    work page

  6. [8]

    However, it does not address the assumptions (e.g., Assumption 2) or the need for more complex real-world applications, which were highlighted in the expert reviews

    **Weaknesses**: - Review A mentions the focus on tabular MDPs and computational expense, which align with the expert reviews’ concerns about computational inefficiency and scalability. However, it does not address the assumptions (e.g., Assumption 2) or the need for more complex real-world applications, which were highlighted in the expert reviews

    work page

  7. [11]

    **Presentation**: - Rating: 3 - Matches the expert reviews, which consistently rated presentation as 3

    work page

  8. [12]

    This rating is more optimistic than the expert consensus

    **Contribution**: - Rating: 4 - Higher than the expert reviews, which consistently rated contribution as 3. This rating is more optimistic than the expert consensus

    work page

Show all 15 references
  1. [13]

    substantial

    **Strengths**: - Review B emphasizes the novelty of the epigraph form, theoretical rigor, and empirical validation, which align well with the expert reviews. It also mentions the clarity in problem formulation and the generality of the approach, which are consistent with the e...

  2. [14]

    These align partially with the expert reviews, which also noted computational inefficiency and scalability issues

    **Weaknesses**: - Review B mentions the complexity of presentation, computational efficiency, limited experimental scope, and lack of comparison to alternative methods. These align partially with the expert reviews, which also noted computational inefficiency and scalability i...

  3. [15]

    These are relevant but do not fully align with the expert reviews, which also asked about assumptions, scaling with state space size, and the gap between transition kernels

    **Questions**: - The questions in Review B focus on scalability, intuition for the epigraph form, scenarios where the Lagrangian approach might be preferable, and sensitivity to hyperparameters. These are relevant but do not fully align with the expert reviews, which also aske...

  4. [16]

    This rating is more optimistic than the majority of expert opinions

    **Rating**: - Rating: 8 - Matches one expert review that rated the paper as 8, but is higher than the other three expert reviews, which rated it as 6, 6, and 3. This rating is more optimistic than the majority of expert opinions. — ### Decision Both reviews align with the expe...

  5. [626]

    Miao Li, Eduard Hovy, and Jey Lau

    ACM. Miao Li, Eduard Hovy, and Jey Lau. 2023a. Summariz- ing multiple documents with conversational structure for meta-review generation. In Findings of the As- sociation for Computational Linguistics: EMNLP 2023, pages 7089–7112, Singapore. Association for Computational Lingu...

    2023 arXiv

  6. [2017]

    Review ”, “Review

    Reviewer bias in single- versus double- blind peer review. Proc. Natl. Acad. Sci. USA , 114(48):12708–12713. Susan van Rooyen, Nick Black, and Fiona Godlee. 1999. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. Journal of Clinical ...

    1999 arXiv

  7. [2024]

    OpenReview.net. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Aru...

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.