{"id":"1d68df9c-6361-432d-ac48-11aa82c4eca6","arxiv_id":"2412.11948","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"OpenReviewer, a fine-tuned Llama-3.1-8B model trained on 79,000 expert reviews, generates more critical and human-aligned paper reviews than GPT-4o and Claude-3.5.","lead":"This paper introduces OpenReviewer, an open-source system built on a fine-tuned 8-billion-parameter language model that generates structured peer reviews for AI and machine learning papers. The authors report that its ratings closely match human reviewer distributions, unlike general-purpose LLMs that tend to be overly positive, making it a potential pre-submission feedback tool.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OpenReviewer's recommendation error (0.96) matches the expected error of a constant predictor that always outputs the human mean (5.4); paper-level correlation is needed to support the central claim.","rationale":"The reader's identified weakest assumption was that similarity to human reviews equals review quality, which the paper itself acknowledges in Section 4. My review does not contest that assumption directly. The more load-bearing problem is that the reported similarity metric may not even capture paper-specific alignment: OpenReviewer's average error equals the optimal constant predictor's error under the reported human rating distribution, and no per-paper correlation analysis is provided. This is a different concern, and it is more damaging because it attacks the quantitative evidence for the central contribution. The paper is otherwise credible: the model is open, training details are transparent, and the limitation sections acknowledge many relevant caveats. The fix is straightforward: add a trivial baseline and a per-paper correlation analysis. Because the model and test data are available, the authors or a third party can run this check without new data collection. If the correlation turns out high (e.g., Spearman > 0.3) and the constant predictor is clearly worse, the central claim survives. Until that is shown, the recommendation-matching results should be interpreted as demonstrating calibration to the rating marginal, not paper-specific critical reviewing.","tokens_in":13226,"tokens_out":3929,"duration_ms":37164,"concrete_test":"For the 400 test papers, compute the per-paper mean and std of human recommendations. Then compute: (a) the Spearman correlation between OpenReviewer's recommendation and the per-paper human mean; (b) the average absolute error of a constant predictor that always outputs 5.4; (c) the exact-match rate of always predicting the modal human rating. If the correlation is near zero, or if the constant predictor's error is ≤ 0.96, or if the modal predictor's EM is near 55.5%, then the recommendation-matching results reflect distribution matching rather than paper-specific critical alignment. This single analysis would settle whether the central claim is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline evidence in Table 1 is that OpenReviewer achieves average recommendation error 0.96 and exact match 55.5%, versus much worse values for general-purpose LLMs. However, the paper never compares against a trivial baseline that ignores paper content. Given that human recommendations across the test set have mean 5.4 and standard deviation 1.2, a constant predictor that always outputs 5.4 has an expected absolute error of approximately 1.2 × sqrt(2/pi) ≈ 0.96. OpenReviewer's reported error of 0.96 is therefore indistinguishable from predicting the global mean for every paper. Similarly, always predicting the modal rating (e.g., 5 or 6) could match at least one human reviewer in a large fraction of papers, because each paper has multiple reviewers and the rating distribution is concentrated. The paper reports only EM and average error against other LLMs; it reports no per-paper correlation between OpenReviewer's recommendation and the human reviewers' average recommendation. Without that, the result does not show that OpenReviewer makes paper-specific critical judgments. It may only have learned the venue's marginal rating distribution. This directly undercuts the central claim that specialized training produces reviews that 'closely match human reviewer judgments' in a content-dependent way. The concern is concrete and testable, which is why the paper should be conditional on such an analysis.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents OpenReviewer, an open-source system that converts a PDF paper into a structured review using a fine-tuned Llama-3.1-8B-Instruct model trained on roughly 79,000 human reviews from ICLR and NeurIPS. The central claim is that OpenReviewer produces considerably more critical and realistic reviews than general-purpose LLMs, supported by two evaluations on 400 test papers: (1) recommendation matching, where OpenReviewer achieves 55.5% exact match (i.e., matching at least one human reviewer's rating) and an average absolute error of 0.96 on a 1–10 scale, versus 23.8% and 2.34 for GPT-4o; and (2) an LLM-as-a-judge arena where GPT-4o prefers OpenReviewer's reviews over baselines in 60–76% of cases. The paper also describes the system architecture, prompt design, training details, and a live demo.","tokens_in":13435,"tokens_out":6678,"duration_ms":57780,"significance":"If the reported alignment is robust and content-dependent, the contribution is significant: an open-source, fine-tuned 8B model that can provide pre-submission feedback with critical realism, potentially useful for authors and as a research platform. The paper's open release of the model, prompts, and training configuration is a strength, as is its candid discussion of limitations. However, the evaluation currently leaves open the possibility that the recommendation-matching results reflect only the marginal rating distribution, not paper-specific judgment. The arena evaluation is also limited by the reliability of the GPT-4o judge. The core methodology is transparent and reproducible, but the evidence for the central claim is not yet conclusive without additional baseline and correlation analyses.","major_comments":[{"comment":"","section":"Section 4.1, Tables 1 and 2"},{"comment":"","section":"Section 4 (test set construction)"},{"comment":"","section":"Section 4.2 (Review Arena)"}],"minor_comments":[{"comment":"","section":"Table 1"},{"comment":"","section":"Section 4.1"},{"comment":"","section":"Section 5, Ethical Considerations"},{"comment":"","section":"Section 3.4"},{"comment":"","section":"Section 3.2.1"},{"comment":"","section":"Section 4"},{"comment":"","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is best viewed as a systems/demonstration paper; its contribution is largely empirical and engineering-oriented. The open-source release is a clear strength. The trivial-baseline issue is easily addressed with additional analysis: if per-paper correlation between OpenReviewer ratings and human mean ratings is high (e.g., above 0.5) and clearly exceeds the constant-predictor baseline, the central claim would be substantially supported. If the authors provide such evidence, the paper could become acceptable. The current version, however, does not yet rule out the possibility that the model has only learned the marginal rating distribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear X,\n\nOpenReviewer is a fine-tuned 8B model for generating peer reviews, trained on 79K ICLR/NeurIPS reviews, with an open-source demo and model. That part is real and useful. The authors also show that general-purpose LLMs give inflated ratings, and that fine-tuning shifts the rating distribution toward the human marginal distribution (5.4 vs 5.4). If you need a pre-submission feedback tool that sounds critical and produces structured, template-following reviews, this is a credible contribution.\n\nThe evaluation, however, does not support the stronger claim that the model's recommendations match human judgments per paper. The headline number, average error 0.96, is exactly the expected absolute error of a constant predictor that always outputs the global mean (5.4), given the reported human standard deviation of ~1.2 (1.2 * sqrt(2/pi) ≈ 0.96). Without a per-paper correlation between model recommendation and the mean human recommendation, or a comparison against a constant baseline, the result could just mean the model learned the venue's marginal rating distribution. The exact-match metric is also fragile: with several reviewers per paper and a concentrated rating distribution, a constant predictor (e.g., always 5) could match at least one reviewer in a large fraction of papers. So the central claim is currently under-supported.\n\nThat said, the paper is transparent about its main weakness. It explicitly notes the similarity-to-human-reviews assumption, acknowledges the small test set, the lack of human evaluation, and the limited venues. The training details, prompts, and hyperparameters are fully documented, and the model is released, so the claims are independently checkable. The LLM-as-a-judge arena is the weakest part: GPT-4o is also a baseline, and bias there is expected.\n\nMy recommendation: send it to peer review, but the revision must add the trivial baselines and per-paper correlation analysis, and ideally a small human evaluation. The system itself is worth having, but the current evidence for 'closely matches human judgments' is not there.\n\nBest","headline":"Useful open system, but the headline metric does not beat a constant predictor, so the central claim is unsupported as written.","tokens_in":13991,"tokens_out":2675,"would_cite":true,"duration_ms":25689,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OpenReviewer, an open-source system built on a fine-tuned 8B model, produces reviews whose recommendations match human reviewers far more closely than GPT-4o does.","keywords":["automated peer review","LLM fine-tuning","review recommendation calibration","critical feedback generation","OpenReview dataset","Llama-3.1","LLM-as-a-judge","PDF-to-markdown conversion"],"falsifier":"A blinded study in which expert reviewers rate the factual correctness and technical depth of OpenReviewer's criticisms against those of GPT-4o on the same 400 papers would decide whether the higher alignment with human ratings reflects genuine critical quality or stylistic mimicry.","tokens_in":12977,"feed_emoji":"📝","tokens_out":7725,"duration_ms":56878,"temperature":0.7,"pith_summary":"OpenReviewer is an open-source system that generates peer reviews of machine-learning papers. The paper's central claim is that an 8-billion-parameter model fine-tuned on 79,000 expert reviews from top conferences produces markedly more critical and realistic reviews than general-purpose models such as GPT-4o and Claude-3.5. On 400 held-out papers, OpenReviewer's recommendation matched at least one human reviewer 55.5% of the time, with an average error of 0.96 on a 1–10 scale, whereas GPT-4o matched only 23.8% with an average error of 2.34. The authors position the system as a pre-submission feedback tool, not a replacement for human peer review. If the result holds, it shows that specialized fine-tuning on peer-review data can overcome the overly positive bias of general-purpose LLMs.","feed_headline":"Fine-tuned 8B model beats GPT-4o at matching human reviews","feed_subtitle":"OpenReviewer's recommendations match human reviewers 55.5% of the time; GPT-4o manages 23.8%.","key_machinery":"The central object is Llama-OpenReviewer-8B, a full fine-tune of Llama-3.1-8B-Instruct on roughly 79,000 high-confidence reviews of ICLR and NeurIPS papers from 2022 onward. The supporting pipeline extracts paper text from PDFs using the Marker converter, keeps the main text and references, and conditions generation with a system prompt based on the ICLR 2024 reviewer guide plus a venue-specific review template. The recommendation score, normalized to a 1–10 scale, is the main quantitative output and the basis for the alignment metrics: exact match with at least one human reviewer and average absolute distance from the human reviewers' mean.","core_discovery":"The paper establishes that review generation is a task where specialized fine-tuning beats scale. On a test set of 400 papers from NeurIPS 2024 and ICLR 2025, OpenReviewer's recommendations match the distribution of human reviewers almost exactly: its average rating is 5.4 out of 10, identical to the human average, while Llama-3.1-8B, Llama-3.1-70B, Claude-3.5-Sonnet, and GPT-4o produce averages between 6.9 and 8.1. In an LLM-as-a-judge preference test, OpenReviewer is judged to align better with expert reviews than GPT-4o in 60% of comparisons and better than Llama-3.1-70B in 76% of comparisons. The authors conclude that general-purpose LLMs are insufficiently critical, and that training on expert review data shifts both the tone and the calibration of generated reviews toward human judgments.","pith_inferences":["If review alignment is treated as a trainable calibration property, the same fine-tuning recipe could be applied to other judgment tasks that currently suffer from LLM positivity bias, such as grant review or hiring recommendation.","Because the training reviews come from a small set of elite venues, the model may encode those venues' reviewing norms; a testable prediction is that its recommendations would shift if trained on reviews from journals or lower-tier conferences with different acceptance cultures.","The 55.5% exact-match ceiling may reflect inherent human reviewer disagreement; a useful extension would be to measure how often two human reviewers agree on the same paper, providing an upper bound for any automated system.","A concrete deployment risk the paper does not test: authors who act on OpenReviewer feedback without human oversight may over-trust its suggestions; a user study measuring manuscript improvement with and without the tool would quantify this."],"forward_implications":["OpenReviewer's recommendation distribution matches the human average (5.4 vs. 5.4), while general-purpose LLMs average between 6.9 and 8.1, so authors get realistic expectations about how their paper will be received.","The exact-match rate of 55.5% suggests that a fine-tuned open 8B model can approach human reviewer calibration, making large closed models unnecessary for this task.","Because the model was trained only on ICLR and NeurIPS reviews from 2022 onward, its behavior on other venues or fields is unproven until retrained.","The arena evaluation, with win rates of 60–76% against the baselines, indicates that the preference signal is consistent across different comparison partners."],"supporting_citations":[{"why":"Supplies the base model Llama-3.1-8B-Instruct that is fine-tuned to create Llama-OpenReviewer-8B.","marker":"Dubey et al., 2024"},{"why":"Provides the LLM-as-a-judge evaluation methodology (MT-bench) that the arena comparison follows.","marker":"Zheng et al., 2023"},{"why":"AlpacaEval is the second LLM-judge benchmark cited as the template for the preference evaluation.","marker":"Li et al., 2023b"},{"why":"Earlier study of GPT-4 for paper reviewing that found limited technical depth, which the paper positions against.","marker":"Liu and Shah, 2023"},{"why":"Shows LLM feedback tends to focus on writing rather than technical content, motivating the need for review-specific training.","marker":"Liang et al., 2023"},{"why":"Evaluation showing LLMs are weak at critical feedback, the gap the paper directly targets.","marker":"Zhou et al., 2024"},{"why":"Survey of NLP for peer review that frames the field and the opportunity for an end-to-end review generation system.","marker":"Kuznetsov et al., 2024"},{"why":"Nougat, the PDF-to-markdown baseline that the Marker pipeline is chosen for improving upon.","marker":"Blecher et al., 2024"}],"fun_headline_variants":["Specialized 8B model matches human reviewers, beats GPT-4o","OpenReviewer: fine-tuned 8B rivals human review calibration","Critical review AI: small model outdoes GPT-4o at calibration","8B model trained on expert reviews matches human ratings","Fine-tuning beats scale: 8B model tops GPT-4o at reviews"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats similarity to human-written reviews as equivalent to review quality, so if human reviews are noisy, biased, or mistaken, OpenReviewer could reproduce those flaws while appearing realistic.","fun_headline_variants_meta":{"raw":{"variants":["Specialized 8B model matches human reviewers, beats GPT-4o","OpenReviewer: fine-tuned 8B rivals human review calibration","Critical review AI: small model outdoes GPT-4o at calibration","8B model trained on expert reviews matches human ratings","Fine-tuning beats scale: 8B model tops GPT-4o at reviews"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000804,"raw_usage":{"total_tokens":3524,"prompt_tokens":926,"completion_tokens":2598,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":2503}},"tokens_in":542,"tokens_out":2598,"duration_ms":15654,"temperature":1.0,"reasoning_tokens":2503,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:25:11.805778+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A blinded study in which expert reviewers rate the factual correctness and technical depth of OpenReviewer's criticisms against those of GPT-4o on the same 400 papers would decide whether the higher alignment with human ratings reflects genuine critical quality or stylistic mimicry.","supporting_citations":[],"review_version":1}