Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RealCritic argues that a critique should be scored by whether the correction it produces is correct, rather than by whether its verdict matches the solution's truth.

desk verdict RealCritic's closed-loop metric is a useful step past verdict-matching, but the joint critique-correction sampling leaves a real confound that the sampled post-check doesn't resolve. read the letter →

arxiv 2501.14492 v1 pith:OSGRG4IR submitted 2025-01-24 cs.CL cs.AIcs.LG

Zhengyang Tang , Ziniu Li , Zhenyang Xiao , Tian Ding , Ruoyu Sun , Benyou Wang , Dayiheng Liu , Fei Huang
show 3 more authors
Tianyu Liu Bowen Yu Junyang Lin
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords critiqueevaluationclosed-loopbenchmarkingself-critiquecross-critiqueiterativeLLMreasoningcorrectionaccuracy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that existing critique benchmarks mis-measure quality because they score the critique itself, typically by verdict matching, rather than its downstream effect. It introduces RealCritic, a closed-loop benchmark on eight challenging reasoning tasks in which a model critiques a solution and then produces a corrected solution, with critique quality defined as the accuracy of that correction. On this metric, o1-mini is the only tested model that improves through self-critique (+3.3% on average), while LLaMA-3.1-70B, Mistral-Large, Qwen2.5-72B variants, and GPT-4o all decline relative to their direct chain-of-thought baselines; o1-mini also shows the largest cross-critique gain (+15.6% over the 50% baseline). A human evaluation on CriticBench supports the critique of open-loop scoring: roughly a third of critiques that pass verdict-matching are judged low-quality by human evaluators.

What carries the argument

The closed-loop critique-and-correction pipeline, with the scoring identity performance $=\mathbb{E}_{x\sim\rho}[\mathbb{I}(y_{\mathrm{correction}})]$: after a model emits a critique, the same model must produce a corrected solution, and only the correctness of that corrected solution counts as the critique score. A post-check prompt verifies that the model critiques before solving (98% compliance on a sampled hundred instances, but the check is optional in the main pipeline), while an I-to-C / C-to-I decomposition separates a model's ability to turn wrong solutions into right ones from its tendency to break right ones.

What would settle it

Compare each model's correction accuracy in the critique condition against a control condition where the same prompt asks for a correction with no critique at all; if accuracy does not drop when the critique is removed, the benchmark is measuring solution ability rather than critique quality.

Watch

Extended reading notes

Core claim

RealCritic formalizes critique quality as performance $=\mathbb{E}_{x\sim\rho}[\mathbb{I}(y_{\mathrm{correction}})]$ where $(y_{\mathrm{critic}}, y_{\mathrm{correction}})$ is drawn from the critic model conditioned on the question, the input solution, and a critique instruction, so a critique is good exactly when it produces a correct corrected solution. The benchmark covers self-critique (critiquing one's own generated solution), cross-critique (critiquing pre-collected solutions, balanced equally between correct and incorrect), and iterative critique (multi-round refinement). Its central empirical finding is that only o1-mini improves under self-critique, with large gains on College Math (+24.0%) and ARC (+19.4%), while classical LLMs underperform their direct chain-of-thought baselines; o1-mini also posts the largest cross-critique improvement (+15.6%), and all tested models lose accuracy on GPQA and MMLU-STEM under self-critique. The paper interprets this as a capability separation: comparable direct chain-of-thought accuracy does not imply comparable critique ability.

Load-bearing premise

The score presumes that the corrected answer is correct because of the critique; a model that can solve the problem directly can earn a high critique score without producing a useful critique.

Editorial extensions

If this is right

  • Verdict-matching benchmarks can award high scores to critiques whose reasoning would lead to wrong answers; RealCritic's correction-based score would re-rank those cases.
  • Self-critique is not a universal skill: among tested models only o1-mini reliably turns self-feedback into correctness gains, so critique progress should be measured as a delta over direct reasoning, not as absolute accuracy.
  • Iterative critique exposes different dynamics across models, with some models' gains decaying across rounds while o1-mini sustains them, making iteration count a useful stress test for long-horizon reasoning.
  • Because all tested models degrade on GPQA and MMLU-STEM under self-critique, critique-driven refinement can hurt when domain expertise is thin, so benchmark results should be reported by task family rather than only as an average.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If correction accuracy is the standard, critique quality and problem-solving ability are not cleanly separable in the benchmark: the same model both critiques and corrects, so a strong solver can score high even with an uninformative critique.
  • A cleaner test would feed the critique to a fixed, weaker solver and measure the change in that solver's accuracy, isolating the critique's contribution from the critic's own solving skill.
  • The instrumental definition of critique quality extends beyond math: any task with a verifiable output (code, formal proofs, data extraction) could reuse the same closed-loop protocol.
  • o1-mini's positive self-critique delta, contrasted with classical models' negative deltas, suggests that training on one's own generated corrections rather than on external critique corpora may be the lever that instills self-critique; this is a trainable hypothesis the benchmark could support.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. RealCritic proposes a closed-loop benchmark for evaluating LLM critique quality: instead of judging verdicts or critique texts, the quality of a critique is measured by the correctness of the solution produced after the critique is applied (Eq. (1), Section 3.1). The benchmark covers self-critique, cross-critique, and iterative critique on eight reasoning datasets, with solutions generated by a model pool and evaluated on six models. The headline findings are that o1-mini is the only tested model that improves under self-critique (+3.3%), while classical instruction-tuned models generally decline, and that o1-mini also gains the most in cross-critique (+15.6%). The paper also reports a human evaluation suggesting that CriticBench's verdict-based evaluation misclassifies about 30% of low-quality critiques as high-quality.

Significance. If the metric's assumption is valid, RealCritic would provide an automatic, objective, and scalable alternative to open-loop critique evaluation, and it would support the practically relevant claim that reasoning-optimized models differ fundamentally from classical LLMs in self-critique capability. The paper is strong on execution: it publicly releases code and data, documents its data-collection and filtering protocol carefully, and includes a human evaluation that directly tests a competing benchmark's construct validity. The core technical idea is timely and likely to influence subsequent work on critique evaluation. However, the central validity assumption is not yet established, and the headline comparisons lack statistical grounding. The benchmark is therefore potentially valuable, but its current evidence base does not support the strength of the paper's conclusions.

major comments (3)
  1. [Section 3.1, Eq. (1); Appendix B] The central claim that correction accuracy measures critique quality is not yet supported because (y_critic, y_correction) are drawn jointly from PCritic-LLM(x, s, Icritic), so a model can score highly by ignoring the critique and directly solving the problem or repairing the solution from x and s alone. The 50% baseline used for cross-critique is not an appropriate null, since the model has access to the question and can solve it independently; indeed, direct CoT accuracies in Table 4 are 56.0-67.6% for the classical models, which alone can approximately reproduce several reported after-critique scores. The post-check described in Appendix B only verifies format-level compliance (critiquing before solving and not overtly circumventing identified errors), it was run on a sample of 100 instances (98% compliance), and the authors explicitly state that it was not enforced in the pipeline; it does not establish that the correction tokens are causally derived from the content of the critique. This is load-bearing because it is exactly the premise that distinguishes RealCritic from open-loop evaluation. I recommend adding a control condition where the critique text is replaced by a generic prompt or by a critique of a different solution, or otherwise demonstrating that correction accuracy drops when the critique content is removed.
  2. [Table 4; Figure 6] The headline findings rest on single-run accuracy numbers without variance, confidence intervals, or significance tests. The per-dataset sample sizes are roughly 190-300 items, so differences of a few percentage points (e.g., o1-mini self-critique +3.3% vs. LLaMA-3.1 -4.3%) may be within sampling noise, and some large deltas come from tasks where the base performance is low (e.g., o1-mini College Math direct CoT is 20.1% but self-critique is 44.1%). Moreover, o1-mini uses its official default sampling settings rather than temperature 0, which may add stochasticity. Repeating runs, reporting bootstrap confidence intervals, or performing paired tests across the 8 tasks would materially strengthen the claim that classical LLMs 'significantly lag' o1-mini and that self-critique underperformance is a systematic phenomenon rather than noise.
  3. [Section 3.1 and Appendix B] The manuscript frames the 'shortcut' problem as addressed by the post-check, but the post-check is described as optional and was not applied to the entire pipeline. The text in Appendix B states: 'Given this high compliance rate, we did not implement post-check as a mandatory step in our evaluation pipeline.' Since the authors themselves identify this as a risk to validity, the released benchmark should either enforce the post-check for all generated samples or exclude unqualified outputs; otherwise, the publicly released numbers and the benchmark protocol include a known confound that the authors have not eliminated.
minor comments (5)
  1. [General] The phrase 'significantly lag' appears in the abstract and conclusion, but no statistical test is reported; please qualify the language until uncertainty is quantified.
  2. [Section 3.1] The notation P_Critic-LLM(x, s, I_critic) is used for a joint distribution over both y_critic and y_correction; please state explicitly that it is a joint (or factored) distribution to avoid ambiguity.
  3. [Section 3.3 / Table 3] The paper does not describe how final answers are extracted or matched for open-ended math tasks (e.g., exact string match, numeric equivalence, or rule-based extraction); this detail is needed for reproducibility and for interpreting accuracy numbers.
  4. [Figure 1 caption] The word 'seperation' should be 'separation'.
  5. [References] The reference list contains duplicate entries for Clark et al. 2018 (arXiv:1803.05457 and arXiv:1803.05457v1); please merge them and use the consistent citation key.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the closed-loop metric is an explicit operational definition, and the self-critique confound in Appendix B is a validity limitation, not a by-construction identity.

full rationale

RealCritic's derivation chain contains no fitted parameters, no imported uniqueness theorems, and no load-bearing self-citation chain. The central metric in Sec. 3.1 (performance = E[I(y_correction)] with (y_critic, y_correction) ~ P_Critic-LLM(x, s, I_critic)) is an explicit operational definition of critique quality as correction effectiveness, not a derivation that secretly presupposes its conclusion. The reported findings, such as o1-mini's +3.3 self-critique delta and the large negative deltas for other models, are empirical measurements under that definition and are not forced by any equation: the observed deltas vary from -35.6 to +24.0 points across models and tasks, so the results are not tautological. The self-critique scenario does sample the initial solution and the correction from the same model distribution, creating a genuine construct-validity confound: a model could score well by direct solving or repair without using the critique's identified errors. The authors acknowledge exactly this in Sec. 3.1 ('potential shortcuts...models may directly generate a new solution instead of critiquing the original one') and Appendix B designs a post-check for it, but then states 'we did not implement post-check as a mandatory step in our evaluation pipeline.' That is a limitation in the causal interpretation of the metric, not an instance of Eq. X = Eq. Y by construction. The only self-citation is the authors' College Math/MathScale dataset (Tang et al., 2024), used as one of eight task sources; it is not load-bearing because the main conclusions are not derived from that dataset alone. Therefore the paper is best scored 0 for circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The benchmark is an evaluation artifact, not a derivation, so there are no fitted free parameters or invented entities. It rests on three domain assumptions: correction accuracy is a valid proxy for critique quality, the prompts and post-check prevent bypass behavior, and the eight selected datasets are representative. These are reasonable but not independently established, which drives the soundness and correctness-risk scores.

assumptions (3)
  • domain assumption Correction accuracy after critique is a valid proxy for critique quality.
    Section 3.1 defines performance as the expected accuracy of y_correction, making this equivalence the foundational premise of the benchmark. It is not independently validated.
  • domain assumption The prompts and post-check prevent models from bypassing the critique and solving directly.
    Section 3.1 and Appendix B claim a post-check mitigates shortcut behavior, but it was validated on 100 samples and not applied as a mandatory filter to the reported results.
  • domain assumption The eight selected datasets represent the reasoning-critique ability of LLMs.
    Section 3.3 and Appendix C describe a subjective filtering process from 20 candidate datasets to 8, with balanced correct/incorrect splits, which shapes the measured findings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques." pith.science (2026). https://pith.science/paper/OSGRG4IR

@misc{pith2026250114492,
  author       = {Pith},
  title        = {Pith review of: RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OSGRG4IR}},
  note         = {Machine review of arXiv:2501.14492}
}
read the original abstract

Critiques are important for enhancing the performance of Large Language Models (LLMs), enabling both self-improvement and constructive feedback for others by identifying flaws and suggesting improvements. However, evaluating the critique capabilities of LLMs presents a significant challenge due to the open-ended nature of the task. In this work, we introduce a new benchmark designed to assess the critique capabilities of LLMs. Unlike existing benchmarks, which typically function in an open-loop fashion, our approach employs a closed-loop methodology that evaluates the quality of corrections generated from critiques. Moreover, the benchmark incorporates features such as self-critique, cross-critique, and iterative critique, which are crucial for distinguishing the abilities of advanced reasoning models from more classical ones. We implement this benchmark using eight challenging reasoning tasks. We have several interesting findings. First, despite demonstrating comparable performance in direct chain-of-thought generation, classical LLMs significantly lag behind the advanced reasoning-based model o1-mini across all critique scenarios. Second, in self-critique and iterative critique settings, classical LLMs may even underperform relative to their baseline capabilities. We hope that this benchmark will serve as a valuable resource to guide future advancements. The code and data are available at \url{https://github.com/tangzhy/RealCritic}.

Figures

Figures reproduced from arXiv: 2501.14492 by the authors.

Figure 1
Figure 1. Benchmark results for self-critique and cross-critique abilities of representative [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Examples illustrating the limitations of CriticBench ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison between the evaluation method used in CriticBench and our frame [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Data collection process for constructing solutions to assess the critique abilities of [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Performance of C→I and I→C in self-critique and cross-critique scenarios. Here, “C” denotes correct, “I” denotes incorrect, and the arrow indicates the accuracy change from the input solution to the correction after critique. A striking observation is the consistent as…
Figure 6
Figure 6. Figure 6: Performance of iterative critique. The performance is averaged over 8 tasks. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Examples comparing CriticBench and RealCritic. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Examples comparing CriticBench and RealCritic. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Examples comparing CriticBench and RealCritic. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback

    cs.CL 2025-07 conditional novelty 6.0 of 10

    RefCritic uses reinforcement learning with two rule-based rewards, judgment accuracy and refinement accuracy, to train long-chain-of-thought critic models that improve LLM math reasoning and detect erroneous steps.

  2. MCTS-Refined CoT: High-Quality Fine-Tuning Data for LLM-Based Repository Issue Resolution

    cs.SE 2025-06 conditional novelty 5.0 of 10

    MCTS-REFINE uses tree search plus strict ground-truth matching to build chain-of-thought training data that lifts open-source LLM issue-resolution scores on SWE-bench.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages · cited by 2 Pith papers

  1. [1]

    The correctness of the student’s solution (compared to ground truth)

    work page

  2. [2]

    14 Preprint

    The verdict provided in the critique (”correct” or ”wrong”) Based on these dimensions, we developed a comprehensive argument mapping framework to evaluate critique quality, as shown in Table 5. 14 Preprint. Work in progress. Scenario Condition Low-Quality Argument 1 Solution: Wrong, Verdict: Wrong The critique correctly identifies the solution is wrong, b...

    work page

  3. [3]

    First, evaluators thoroughly reviewed the mathematical problem and its ground truth solution to establish a clear understanding of the correct approach and key concepts involved

    work page

  4. [4]

    They then examined the student’s attempted solution in detail, comparing it with the ground truth solution to identify any discrepancies or errors

    work page

  5. [5]

    Next, they analyzed the generated critique, paying particular attention to both its verdict (”correct” or ”wrong”) and the specific reasoning provided in the analysis

    work page

  6. [6]

    Based on the solution’s correctness and the critique’s verdict, evaluators were presented with the corresponding argument from Table 5

    work page

  7. [7]

    See Figure 2, 7, 8, and 9 for more concrete examples

    Finally, evaluators provided: A binary judgment: ”True” (indicating the argument applies and the critique is low-quality) or ”False” (indicating the argument doesn’t apply and the critique is high-quality) The results of this systematic evaluation process were used to create the confusion matrix presented in Table 2 in the main text, which demonstrates th...

    work page

  8. [8]

    The model does not adhere to the paradigm of critiquing before providing its own solution; instead, it solves the problem first and provides a critique afterward

    work page

Show all 21 references
  1. [9]

    If any of the above non-compliant behaviors are present, the final answer should be Unquali f ied

    During the critique, the model is capable of identifying errors but fails to correct the flawed reasoning based on the identified mistakes; instead, it circumvents the pointed-out issues in a deceptive manner, providing its own solution. If any of the above non-compliant behav...

    2017

  2. [10]

    Outcomes with zero heads: There is only one outcome with zero heads, which is TTT

  3. [11]

    Outcomes with one head: There are three outcomes with one head, which are HTT, THT, and TTH

  4. [12]

    Outcomes with exactly two heads: There are three outcomes with exactly two heads, which are HHT, HTH, and THH. Now, we add up the number of outcomes for each case: • Outcomes with zero heads: 1 • Outcomes with one head: 3 • Outcomes with exactly two heads: 3 The total number o...

  5. [13]

    - **Edward:** 120 minutes 48 minutes/lawn = 2.5 lawns Similarly, Edward mows **2 lawns**

    **Calculate Lawns Mowed Individually in 120 Minutes:** - **Dennis:** 120 minutes 48 minutes/lawn = 2.5 lawns Since partial lawns aren’t counted until completed, Dennis mows **2 lawns**. - **Edward:** 120 minutes 48 minutes/lawn = 2.5 lawns Similarly, Edward mows **2 lawns**. -...

  6. [14]

    Conduct a step-by-step critique of the student’s solution

    **Total Lawns Mowed Together:** 2 (Dennis) + 2 (Edward) + 5 (Shawn) = 9 lawns **Answer:** 9 Step 2: Self-Critique System Prompt: I will provide a math problem along with a student solution. Conduct a step-by-step critique of the student’s solution. For each step, use proper ve...

  7. [15]

    - **Edward’s Rate:** 1 48 lawns per minute

    **Calculate Combined Mowing Rate:** - **Dennis’s Rate:** 1 48 lawns per minute. - **Edward’s Rate:** 1 48 lawns per minute. - **Shawn’s Rate:** 1 24 lawns per minute. **Total Combined Rate:** 1 48 + 1 48 + 1 24 = 1 + 1 + 2 48 = 4 48 = 1 12 lawns per minute

  8. [16]

    22 Preprint

    **Calculate Total Lawns Mowed in 120 Minutes:** 120 minutes × 1 12 lawns per minute = 10 lawns **Answer:** 10 Table 11: A concrete example of Self-Critique. 22 Preprint. Work in progress. System Prompt: I will provide a math problem along with a student solution. Conduct a ste...

  9. [18]

    Each hole has 4 sides, and each side is a rectangle with dimensions 1 × 37

    **Calculate the surface area added by the holes:** Each hole is a square with side length 1, and it passes through the entire cube, so it has a length of 37. Each hole has 4 sides, and each side is a rectangle with dimensions 1 × 37. The area of one side of a hole is 37, so th...

  10. [19]

    Each face of the cube has an area of 372 = 1369, and with 6 faces, the total surface area is 6 × 1369 = 8214

    **Calculate the surface area of the resulting solid:** The surface area of the resulting solid is the sum of the original surface area of the cube and the additional surface area created by the holes: 8214 + 888 = 9102 Thus, the surface area of the resulting solid is: 9102 Ass...

  11. [20]

    The area of one face is 372 = 1369

    **Calculate the original surface area of the cube:** The cube has 6 faces, and each face is a square with side length T = 37. The area of one face is 372 = 1369. Therefore, the original surface area of the cube is: 6 × 1369 = 8214

  12. [21]

    Since there are 3 pairs of opposite faces, there are 3 distinct square holes

    **Calculate the surface area removed by the holes:** Each square hole removes a 1 × 1 square from each face it penetrates. Since there are 3 pairs of opposite faces, there are 3 distinct square holes. Each hole removes 2 squares (one from each of the two opposite faces), so th...

  13. [22]

    23 Preprint

    **Calculate the total surface area of the resulting solid:** Add the adjusted original surface area and the net additional surface area from the holes: 8208 + 432 = 8640 Therefore, the surface area of the resulting solid is: 8640 Table 12: A concrete example of Cross-Critique....

    2016

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.