Pith. sign in

REVIEW 4 major objections 5 minor 28 references

AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM-generated peer reviews are fluent but systematically over-positive and poorly calibrated.

desk verdict A genuinely useful venue-level policy survey and a carefully documented LLM-review evaluation whose headline coverage numbers rest on a matcher tuned on the same data; the core descriptive findings survive, the coverage claims are fragile. read the letter →

arxiv 2608.03581 v1 pith:4RW7YV2C submitted 2026-08-04 cs.CY cs.AI

Alexander M. Fichtl , Lukas Ellinger , Josefin Kelber , Kryštof Olík , Georg Groh This is my paper · ORCID
classification cs.CYcs.AI
keywords peerreviewlargelanguagemodelsAI-assistedreviewingreviewerAIpolicyqualityevaluationscorealignmentconcernoverlapLLM-as-a-Judge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Using 50 original ICLR 2026 submissions and 31 Nature Communications papers with matched human and machine reviews, the paper asks whether current LLMs can produce genuinely useful peer reviews. It finds that GPT-5, Llama-4, and Qwen3 write long, fluent, detailed reviews that receive high LLM-judge quality scores, but all three are systematically over-positive: they average 6.8–7.9 on a 10-point scale where human reviewers average 4.3, and only GPT's scores track human rankings (correlation 0.62; Llama and Qwen are statistically indistinguishable from zero). The central contention is that review quality is multi-dimensional, so aggregate quality scores alone overestimate AI review usefulness; the real value of current AI reviews is breadth — surfacing more candidate concerns — not judgment. Separately, a policy survey of 111 venues finds AI/NLP conferences mostly permit partial AI assistance while medical journals mostly prohibit reviewer AI use, so regulation is community-specific and still unsettled.

What carries the argument

The argument is carried by a four-part evaluation suite applied to identical manuscript-and-review pairs. The centerpiece is the overlap score: reviews are decomposed into atomic concerns by GPT-5.5, each tagged as strength or weakness with a seriousness level, and a paper-grounded LLM matcher links AI concerns to pooled human concerns for the same manuscript, producing raw and length-normalized coverage. Around this sit score alignment (correlating model and human paper scores on ICLR), Granuscore (a reference-free measure of semantic granularity, where lower means more specific feedback), and LLM-as-a-Judge (GPT-5 ratings on understanding, coverage, evidence support, constructiveness, conc

What would settle it

Take the same 50 ICLR papers and have independent expert reviewers, blind to source, judge whether each AI-identified concern expresses the same point as a pooled human concern. If their agreement falls well below the reported coverage (e.g., GPT's 46% for Essential concerns), the overlap-based conclusion loses support. A complementary check: run a GPT variant prompted to match the human score distribution; if it still receives near-maximum LLM-judge scores while failing to rank papers, that confirms aggregate quality scores ignore calibration.

Watch

Extended reading notes

Core claim

On the paper's own terms, current LLMs are useful reviewer assistants but not autonomous reviewers. The evidence has three parts. First, calibration fails: mean LLM scores are 6.8–7.9 out of 10 versus 4.3 for humans; GPT separates accepted from rejected papers by only about half the human gap (0.87 vs 1.53 points), and Llama and Qwen show no separation at all. Second, depth of critique is uneven: decomposing reviews into atomic concerns and matching them to pooled human concerns shows AI reviews cover more human concerns than a single reviewer in raw terms (GPT 0.34 vs human baseline 0.16), but this advantage vanishes when normalized for review length (all sources about 0.14–0.16), meaning t

Load-bearing premise

The result that AI reviews add breadth rather than better judgment rests on trusting the model that matches AI criticisms to human criticisms; if that matcher is biased toward AI phrasing, the coverage numbers overstate real overlap.

Editorial extensions

If this is right

  • A single LLM-as-a-Judge score is not a valid gate for deploying AI review tools; venues should require calibration, grounding, and overlap metrics alongside it.
  • Using current open-weight LLM review scores for accept/reject decisions would be close to random at the ranking level; even the best model tested is upward-biased and only moderately aligned.
  • AI reviews can serve as a second-pass checklist: they surface additional candidate concerns and attach proposed actions to almost every weakness, including more Essential concerns in raw coverage.
  • Reviewer-AI policy should be written for specific uses, not blanket rules: AI/NLP venues tend to permit language assistance while medical journals restrict manuscript uploads and emphasize confidentiality.
  • Human reviewers should form their own assessment before seeing AI output to avoid anchoring; this workflow matches the paper's evidence and its recommended role for AI as an assistant.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper: the overlap matcher was tuned until its outputs matched the authors' judgments, so the headline coverage numbers deserve an independent re-check; if blind expert raters call fewer AI-human pairs the same point, the breadth advantage shrinks.
  • Going beyond the paper: the Nature Communications sample contains only accepted papers, so the large overlap drop from ICLR to Nature may partly reflect selection, not just review format; rejected manuscripts would separate the two explanations.
  • Going beyond the paper: the paper's Limitations section notes that GPT-5 is both generator and judge; an independent human rating of the same reviews would test how much of GPT's near-maximum judge scores is stylistic affinity rather than quality.
  • Going beyond the paper: GPT's higher coverage of Essential concerns suggests a testable prompt intervention — asking models to name the single most important weakness — that might lift open-weight models without fixing score calibration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies AI-assisted peer review from two angles. First, it surveys reviewer-facing AI policies across 111 venues (63 AI/NLP conferences and 48 medical journals), finding that AI/NLP venues are predominantly permissive with 'Partial' policies while medical journals mostly prohibit AI use. Second, it constructs a dataset of original manuscripts and corresponding human and machine-generated reviews for ICLR 2026 (50 papers) and Nature Communications (31 papers), using GPT-5, Llama-4 Scout, and Qwen3-VL to generate reviews. The evaluation combines score alignment, an LLM-based overlap metric, GranuScore, and LLM-as-a-Judge. The reported findings are that LLM reviews are longer, more positive, and less calibrated than human reviews; GPT shows moderate correlation with human scores (r=0.62) while Llama and Qwen show none; AI reviews achieve high aggregate judge scores but exhibit generic criticism and uneven evidence grounding; and AI reviews cover a nontrivial fraction of human concerns, especially 'Essential' ones, which the authors interpret as breadth rather than superior judgment.

Significance. If the results hold, the paper makes a useful contribution to the current debate on AI-assisted peer review. The venue-level policy survey is novel and valuable, and the release of a dataset with original submission versions and human/AI review pairs addresses a real gap in the literature. The multi-metric evaluation, including direct qualitative evidence of positive bias, generic critique, and generation degeneracy, is a strength: these findings are human-auditable and do not depend on the more fragile parts of the pipeline. The paper is also commendable for making code and data available and for transparently discussing limitations. However, two load-bearing evaluation components—the LLM-based overlap metric and the GPT-5-as-judge—have serious validity concerns that directly affect the headline conclusions about coverage and about aggregate scores overestimating review quality. These concerns need to be addressed before the paper's central claims can be accepted as stated.

major comments (4)
  1. [§5.3, Appendix E.1/E.5] The overlap score, which supports the 'breadth not judgment' conclusion and the seriousness-stratified coverage numbers (e.g., GPT covers 46% of Essential concerns vs. 12% of Minor concerns), rests entirely on a GPT-5.5-based concern extraction and matching pipeline. Appendix E.1 states that prompts were 'iteratively refined until the inspected outputs aligned with our judgments' on the same corpus, with no held-out validation, human gold standard, or inter-annotator agreement reported. The sensitivity check in Appendix E.5 only varies the matcher (embedding vs. LLM) while reusing the same extracted concerns; it reports per-paper correlation of only 0.50 and pairwise agreement of 0.39, which is not strong evidence of robustness. Without independent human annotation of extracted concerns and matched pairs, the specific coverage numbers and the related conclusion in §6 that 'the main value
  2. [§5.5, Table 4, Limitations] The LLM-as-a-Judge evaluation uses GPT-5 to rate reviews that include GPT-5's own output. This creates a same-model self-preference bias, which the authors acknowledge in the Limitations. The judge scores are not merely incidental: they directly feed the paper's claim that 'aggregate quality scores alone can overestimate review quality' and the observation that GPT and Qwen receive near-maximum scores on most dimensions. The acknowledgment that the paper 'does not rely on LLM-as-a-Judge as the only evaluation' mitigates the concern but does not remove it for the aggregate-quality claim. I recommend either using a judge from a different model family, adding a human evaluation of a subset of reviews, or explicitly reporting a sensitivity analysis that quantifies the self-preference effect (e.g., by also running the judge on human reviews of varying style).
  3. [§4.1.2, Appendix E.4] The Nature Communications component includes only accepted papers, because rejected submissions are not public. The paper acknowledges this in the Limitations, but the cross-venue comparison in Appendix E.4—coverage drops by roughly one third to one half from ICLR to Nature Communications for all sources—is affected by this selection bias: human reviews of accepted papers may differ systematically from reviews of a mixed set, and the AI reviews are generated for the same accepted papers. This makes the venue comparison difficult to interpret as a pure effect of review setting. The authors should either restrict cross-venue claims to within-venue relative comparisons or explicitly discuss the direction of the bias for each metric.
  4. [§6, 'AI as Review Support'] The recommendation that AI should be used as a 'second-pass assistant' is reasonable and well-hedged, but the manuscript goes further in saying AI reviews 'may also support the formulation of more actionable feedback' based on the recommendation-category analysis in Appendix E.3. That analysis is also produced by the same unvalidated GPT-5.5 extraction pipeline, and the categories are assigned by the extractor rather than by human readers. Since this is a secondary point, it can be fixed by softening the language or by providing a small human-coded verification of the recommendation categories.
minor comments (5)
  1. [General] There is inconsistent capitalization and naming: 'GranuScore' vs. 'Granuscore' in the text; 'Qwen3-VL-235B' is sometimes referred to as 'Qwen3'; and 'Nature Comm.' vs. 'Nature Communications' are used interchangeably. Please standardize.
  2. [Figure 6 caption] Typo: 'indiviual' should be 'individual'.
  3. [§1] The sentence 'Every main-track submission at AAAI 2026 received one clearly identified AI review...' cites Biswas et al. (2026), but the reference entry does not clarify whether this was an official pilot or an external experiment. Please add a sentence describing the nature of the deployment, since it is used to motivate the paper.
  4. [Table 4] The standard deviations for GPT dimensions in Table 4 are reported as 0.0 for many entries (e.g., Understanding, Coverage, Evidence Support, Constructiveness for ICLR). This suggests a ceiling effect. It would be informative to report the full distribution or the number of papers at the maximum score, so readers can judge the discriminative power of the metric.
  5. [Appendix E.2] The explanation for excluding strengths is clear and sensible, but the example with paper OWHKdYwYiF would benefit from a citation to the specific review excerpt so that the reader can verify the point.

Circularity Check

2 steps flagged · score 6.0 of 10

Overlap metric's concern-extraction/matching pipeline was iteratively tuned on the same corpus and lacks independent validation; headline coverage numbers and the 'breadth not judgment' conclusion are partly fitted to the authors' judgments. LLM-as-Judge self-preference adds a second-order confound.

  1. fitted input called prediction [Appendix E.1 (Concern Extraction and Matching); used in Section 5.3, Table 3, and Section 6]
    "During development, we manually inspected extraction and matching outputs for outlier papers and iteratively refined the prompts until the inspected outputs aligned with our judgments."

    The headline coverage results (e.g., GPT covers 46% of Essential human concerns, Table 3) are produced by the GPT-5.5 extraction/matching pipeline whose prompts were tuned on this same corpus to match the authors' judgments. No held-out validation or independent gold-standard annotation is reported. The only sensitivity check (Appendix E.5) swaps the matcher for sentence embeddings but reuses the same extracted concerns, so it cannot detect extraction bias; per-paper correlation is 0.50 and pair-level agreement is 0.39. Thus the coverage-by-seriousness numbers and the 'breadth rather than superior judgment' conclusion (Section 6) are not independent of the authors' tuning: the metric is effectively calibrated to their judgments.

  2. other [Section 5.5, Table 4; Limitations]
    "This is prevalent especially in our setting, where GPT-5 is used both as a review generator and evaluator, which may introduce bias toward GPT-like outputs."

    The LLM-as-a-Judge scores (Table 4) are produced by GPT-5, the same model used to generate the GPT reviews, and these scores are used in the Discussion to argue that aggregate quality scores alone can overestimate review quality. Because the judge and the evaluated GPT reviews share the same model, the high aggregate scores for GPT/Qwen (e.g., 6.7 and 5.8 vs 3.5 for human on ICLR) partly reflect self-preference rather than independent assessment. The paper acknowledges this but still uses the scores as one of the complementary metrics, so the 'overestimate' demonstration is partially dependent on the judge choice. This is a confound rather than a definitional reduction, but it weakens the independence of the claim.

full rationale

The paper's core evaluation is not wholly self-contained. The overlap metric—the basis for the distinctive 'breadth rather than judgment' claim—uses a GPT-5.5 concern-extraction and matching pipeline whose prompts were iteratively refined on the same corpus until they matched the authors' judgments (Appendix E.1). The sensitivity analysis (Appendix E.5) changes only the matcher, not the extraction, so it cannot rule out extraction bias; the reported per-paper correlation of 0.50 and pair-level agreement of 0.39 between matchers indicate that individual matches are not stable. This makes the coverage-by-seriousness numbers (Table 3) and the Discussion's 'breadth' conclusion partly an output of the authors' tuning rather than an external measurement. Separately, the LLM-as-a-Judge stage uses GPT-5 to rate GPT-5-generated reviews, an acknowledged self-preference confound (Limitations) that feeds the 'aggregate quality scores overestimate review quality' argument, though this claim is also supported by direct score-alignment results (Section 5.2) and qualitative analysis (Section 5.6). Other findings—score inflation, generic criticism, evidence-grounding gaps, and generation degeneracy—are based on direct human-auditable counts and correlations, so they are not circular. The policy survey is independent. Overall, the paper has substantial independent content, but the headline overlap-based claim reduces in part to a tuned measurement instrument, warranting a partial-circularity score of 6 rather than a clean bill.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities. The central empirical claims rest on a curated dataset of original submissions and human reviews, on LLM-mediated concern matching, and on an unpublished in-group granularity metric. The main burden the reader pays for upstream is trust in the human review labels and in the LLM pipeline that extracts and matches concerns.

assumptions (5)
  • domain assumption Human peer reviews at ICLR 2026 and Nature Communications constitute a valid reference standard for review quality and reviewer concerns.
    The overlap and score-alignment metrics treat aggregated human reviews as ground truth (Sections 4.3.1, 4.3.2). If human reviews are themselves noisy or incomplete, AI-vs-human alignment is an imperfect proxy for review quality, a point the paper itself concedes.
  • domain assumption The manuscript versions used are the exact versions reviewers evaluated.
    Section 4.1.1 relies on capturing ICLR submissions before revised versions appeared; Section 4.1.2 relies on Research Square postings being the version seen by reviewers at Nature Communications. No verification against reviewer-held copies is possible.
  • domain assumption GPT-5.5-based concern extraction and matching correctly identifies semantic equivalence between AI and human concerns.
    Appendix E.1 defines the pipeline and the matching criterion 'same underlying point'. The validity of all coverage numbers depends on this LLM-mediated judgment, which is not independently verified.
  • domain assumption The LLM-as-a-Judge rubric scores review quality as intended, without systematic model-family bias.
    Section 5.5 and Appendix D use GPT-5 with the stated rubric. The authors acknowledge potential bias toward GPT-like outputs in the Limitations; absolute judge scores therefore cannot be read as unbiased measures.
  • domain assumption Granuscore is a valid reference-free measure of review granularity.
    Sections 4.3.3 and 5.4 rely on Granuscore (Ellinger et al., 2026), a metric from the same research group; its validity is assumed from the cited paper, which is not available to readers here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality." pith.science (2026). https://pith.science/paper/4RW7YV2C

@misc{pith2026260803581,
  author       = {Pith},
  title        = {Pith review of: AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4RW7YV2C}},
  note         = {Machine review of arXiv:2608.03581}
}
read the original abstract

AI-assisted peer review is increasingly discussed and adopted as a tool to support the scientific publishing process, yet there is little systematic understanding of how publication venues regulate its use or of how capable current AI review systems are. We address these questions by first surveying reviewer-facing AI policies across 111 leading AI/NLP conferences and medical journals, revealing substantial regulation differences between the two communities. Second, we evaluate AI-generated peer reviews at ICLR 2026 and Nature Communications using a novel dataset comprising original manuscript submissions and several hundred human- and machine-generated reviews. We compare reviews produced by open-source and proprietary models using complementary evaluation metrics, including LLM-as-a-Judge, score alignment, granularity, and overlap with human reviewers' concerns. Our results show that current LLMs can generate detailed and fluent reviews but exhibit systematic weaknesses, such as overly positive recommendations, generic criticism, and uneven evidence grounding. We demonstrate that aggregate quality scores alone can overestimate review quality and argue for multi-dimensional evaluation of AI-generated peer reviews.

Figures

Figures reproduced from arXiv: 2608.03581 by the authors.

Figure 1
Figure 1. Reviewer-AI policies across 63 AI/NLP conferences and 48 medical journals. Left: top-level policy [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Raw and length-normalized weakness cov￾erage across review sources. Confidence intervals are obtained via a paper-level bootstrap (2,000 resamples). cerns generated by AI reviewers: GPT produces a median of 36 extracted concerns per review com￾pared with 11 for humans. When controlling for this difference in review length, the gap is much smaller: all sources fall into a narrow range, with GPT becoming comparable to… view at source ↗
Figure 3
Figure 3. Per-paper GranuScore for ICLR reviews. Gray points show human reviews, with their median and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Overall score given by LLM-as-a-Judge for the peer-reviews of ICLR papers. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Weakness coverage by venue and source. The embedding matcher closely reproduces the aggregate coverage obtained with the paper-aware LLM matcher (0.36 vs. 0.37), suggesting that the headline coverage estimate is robust to the matching method. Agreement is lower, howeve…
Figure 6
Figure 6. Figure 6: Per-paper weakness coverage obtained with the LLM matcher (x-axis) and the sentence-embedding [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Scores assigned to individual ICLR submissions. Gray points show scores from individual human [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: Per-paper GranuScore for Nature Communications reviews. Gray points show human reviews, with their [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Overall score given by LLM-as-a-Judge for the peer-reviews of Nature Communication papers. [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: Understanding score given by LLM-as-a-Judge for the peer-reviews of ICLR papers. [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: Coverage score given by LLM-as-a-Judge for the peer-reviews of ICLR papers. [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Evidence support score given by LLM-as-a-Judge for the peer-reviews of ICLR papers. [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Constructiveness score given by LLM-as-a-Judge for the peer-reviews of ICLR papers. [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: Conciseness score given by LLM-as-a-Judge for the peer-reviews of ICLR papers. [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: Understanding score given by LLM-as-a-Judge for the peer-reviews of Nature Communications papers. [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]
Figure 16
Figure 16. Figure 16: Coverage score given by LLM-as-a-Judge for the peer-reviews of Nature Communications papers. [PITH_FULL_IMAGE:figures/full_fig_p027_16.png]
Figure 17
Figure 17. Figure 17: Evidence support score given by LLM-as-a-Judge for the peer-reviews of Nature Communications [PITH_FULL_IMAGE:figures/full_fig_p028_17.png]
Figure 18
Figure 18. Figure 18: Constructiveness score given by LLM-as-a-Judge for the peer-reviews of Nature Communications papers. [PITH_FULL_IMAGE:figures/full_fig_p028_18.png]
Figure 19
Figure 19. Figure 19: Conciseness score given by LLM-as-a-Judge for the peer-reviews of Nature Communications papers. [PITH_FULL_IMAGE:figures/full_fig_p028_19.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 27 canonical work pages

  1. [3]

    No boilerplate

    Write like a human reviewer: Be concise, relevant, specific, and non-repetitive. No boilerplate

    work page

  2. [6]

    For rating/scoring questions (Soundness, Presentation, Contribution, Confidence, Overall): Reason briefly, then give the numeric rating in the form of ’Rating: <value>’

    work page

  3. [7]

    Keep the amount of points reasonable

    For Strengths/Weaknesses/Suggestions prefer numbered lists (1., 2., ...). Keep the amount of points reasonable. Only keep the least generic points

    work page

  4. [9]

    Briefly summarize the paper and its contributions. Do not critique the paper here. The authors should agree with a well-written summary. Typical range: 3-8 sentences

    A total review has typically at least 300 and maximal 2000 words. Prefer your reviews to have between 500 and 1500 words. [Reviewer Guide] [Area Chair Guide] [Code of Ethics] [Code of Conduct] [Previous Year Conference Statistics] Review Field Prompts: Review Generation for ICLR Q1 (Summary):"Briefly summarize the paper and its contributions. Do not criti...

    work page 2000

  5. [12]

    Do not hallucinate

    Use only the provided context. Do not hallucinate

    work page

  6. [13]

    No chain-of-thought in the final answer

    work page

  7. [14]

    Do not use section headers or a rigid structure

    Write in natural flowing prose like a human expert reviewer. Do not use section headers or a rigid structure. Weave validity, significance, methodology, and suggestions naturally into your text. Be concise, relevant, specific, and non-repetitive. No boilerplate. Not unnecessarily long

    work page

  8. [15]

    No restating of the questions or instructions

    work page

Show all 28 references
  1. [16]

    Cite evidence from context. E.g. the section title or figure id or table id. You can also quote short sentence parts

  2. [17]

    Keep the amount of points reasonable

    For listing strengths/weaknesses/suggestions prefer numbered lists (1., 2., ...). Keep the amount of points reasonable. Only keep the least generic points

  3. [18]

    You can incorporate Markdown and Latex into your review

  4. [19]

    items" containing a list of decomposed items. CHANNEL CLASSIFICATION (every item gets exactly one) •

    A total review has typically at least 300 and maximal 2000 words. Prefer your reviews to have between 500 and 1500 words. [image standards] [research ethics] [reviewer report guidelines] User Prompt: Review Generation for Nature Communications [paper text and referenced images...

    2000

  5. [20]

    Do not emit any item that comes from the review’s summary

    SKIP THE SUMMARY ENTIRELY. Do not emit any item that comes from the review’s summary. For structured reviews, this is the field labelled “Summary”; for narrative journal reviews, it is an opening paragraph that merely restates or paraphrases the paper without evaluation. By ve...

  6. [21]

    Do not bundle two distinct concerns or two distinct strengths into one item

    Each item makes EXACTLY ONE point. Do not bundle two distinct concerns or two distinct strengths into one item

  7. [22]

    this issue

    Each item must be understandable on its own. Spell out implicit references (e.g., replace “this issue” with the specific issue)

  8. [23]

    Do not invent points the reviewer did not raise

    Extract only what the reviewer actually wrote. Do not invent points the reviewer did not raise. Do not soften or sharpen the framing

  9. [24]

    If a single sentence contains multiple distinct points, split it into multiple items

  10. [25]

    If multiple sentences make the same point with different wording, merge them into one item with a single combined formulation

  11. [26]

    Soundness: 4

    Skip boilerplate: greetings, signoffs, conference-system scaffolding, numerical score lines (e.g., “Soundness: 4”, “Confidence: 3”, “Rating: 6”)

  12. [27]

    critique

    Keep the item text concise (one to three sentences). The verbatim quote is not required; a faithful paraphrase is fine as long as the point is preserved. TYPE CLASSIFICATION (every item gets exactly one) •Experimental design: experiments, baselines, ablations, controls, hyperp...

  13. [28]

    NEVER pair a critique with a strength

    POLARITY: Match a critique only to a critique and a strength only to a strength. NEVER pair a critique with a strength

  14. [29]

    experiments

    SAME POINT: A pair is valid when both concerns address the same substantive issue, gap, claim, contribution, or strength of the paper. Vague thematic overlap (e.g., both concerns mention “experiments”) is NOT sufficient. They must express the same specific point

  15. [30]

    GRANULARITY: Matches may be many-to-many. One AI concern may match several human concerns (e.g., when an AI concern combines two points raised separately by human reviewers), and one human concern may match several AI concerns. Emit each valid pair separately

  16. [31]

    When in doubt, do not emit the pair

    CONFIDENCE: Emit a pair only when you are reasonably confident that the concerns express the same point. When in doubt, do not emit the pair. Unmatched concerns do not appear in the output

  17. [32]

    Provide a one-sentence justification for each pair that identifies the shared point

  18. [33]

    matches", containing a list of objects with the fields

    If no concerns match, return an empty list. OUTPUT Reply with a JSON object that matches the provided schema. The object has a single field, "matches", containing a list of objects with the fields "ai_number", "human_number", and "justification". The value of "ai_number" must ...

    2019

  19. [2024]

    Sihong Wu, Owen Jiang, Yilun Zhao, Tiansheng Hu, Yiling Ma, Kaiyan Zhang, Manasi Patwardhan, and Arman Cohan

    Ai-driven review systems: evaluating llms in scalable and bias-aware academic reviews.arXiv preprint arXiv:2408.10365. Sihong Wu, Owen Jiang, Yilun Zhao, Tiansheng Hu, Yiling Ma, Kaiyan Zhang, Manasi Patwardhan, and Arman Cohan. 2026. Can ai be a good peer reviewer? a survey o...

    2026 arXiv

  20. [2025]

    Proceedings of the ACM on Human-Computer Inter- action

    The AI review lottery: Widespread AI-assisted peer reviews boost paper scores and acceptance rates. Proceedings of the ACM on Human-Computer Inter- action. Z. Sun. 2025. Large language models in peer re- view: challenges and opportunities.Scientometrics, 130:5503–5546. Nitya T...

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.