{"id":"22c0f8a6-1f14-41af-9547-6629cba9fb9d","arxiv_id":"2501.14492","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A closed-loop benchmark that scores LLM critiques by the correctness of the corrections they produce finds o1-mini clearly outperforms classical LLMs, which often get worse after self-critique.","lead":"RealCritic tests how well large language models critique solutions by measuring whether the corrections produced after their critiques are correct, rather than judging the critique text alone. It adds self-critique, cross-critique, and iterative critique modes, and finds that OpenAI's o1-mini improves from its own critiques while most classical models do not.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The metric equates correction accuracy with critique quality, but since critique and correction are sampled jointly, high scores can come from direct problem-solving rather than from the critique; the paper's post-check was sampled and not enforced.","rationale":"In good faith, the paper's contribution is a closed-loop benchmark with a sensible taxonomy and a substantial evaluation; the human-evaluation evidence against verdict matching (Table 2) and the public code/data are real. My concern is not disagreement with the closed-loop philosophy but with an unvalidated causal identification inside its own metric. The reader's weakest-assumption statement matches this exactly. The paper itself flags the shortcut risk in Section 3.1 ('models may directly generate a new solution instead of critiquing the original one') and relies on prompt tuning plus the Appendix B post-check. But the post-check is a sampled LLM-based format/compliance check, not a causal intervention, and it was explicitly not enforced. A two-stage correction experiment with controlled critique content would settle the question. If the control arms reproduce the reported ordering, the benchmark's construct validity is established and the conditional acceptance should stand; if not, the metric needs revision or the central claim should be withdrawn. Given that the reader already imposed the condition, I leave the verdict unchanged rather than moving it.","tokens_in":20215,"tokens_out":5733,"duration_ms":53577,"concrete_test":"Conduct a two-stage causal control on the same 8 tasks and models: (1) generate y_critic alone; (2) in a fresh generation with no simultaneous critique, produce y_correction conditioned on (x, s, y_critic). Add two control arms: (a) the same correction prompt with a neutral placeholder in place of y_critic, and (b) a deliberately wrong critique (e.g., the model's critique with its conclusion flipped). If correction accuracy in the real-critique arm is not significantly higher than in arms (a) and (b) with per-task confidence intervals, then Eq. (3.1) largely reflects direct solving/repair ability rather than critique quality. Report the delta and variance; this directly tests causality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines performance = E[I(y_correction)] with (y_critic, y_correction) ~ P_Critic-LLM(x, s, I_critic). Because critique and correction are produced in the same generation under one prompt, a model that ignores the critique and solves (or repairs) the problem directly from x and s can score well. The 50% baseline in cross-critique does not remove this confound: direct CoT accuracy on the same questions is 56-68% for the tested models, so independent solving alone can produce the reported after-critique scores. Appendix B's post-check asks a second LLM whether the model critiqued before solving and whether the correction circumvented identified errors, but it was applied to a sample (98% compliance on 100 instances) and the authors explicitly state it was not made mandatory in the pipeline. Format-level compliance also does not establish that the correction tokens were causally derived from the critique content. Consequently, the central construct-validity assumption—that E[I(y_correction)] measures critique quality—is not yet supported; the o1-mini self-critique advantage could reflect better behavior under the critique prompt rather than better critiques.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RealCritic proposes a closed-loop benchmark for evaluating LLM critique quality: instead of judging verdicts or critique texts, the quality of a critique is measured by the correctness of the solution produced after the critique is applied (Eq. (1), Section 3.1). The benchmark covers self-critique, cross-critique, and iterative critique on eight reasoning datasets, with solutions generated by a model pool and evaluated on six models. The headline findings are that o1-mini is the only tested model that improves under self-critique (+3.3%), while classical instruction-tuned models generally decline, and that o1-mini also gains the most in cross-critique (+15.6%). The paper also reports a human evaluation suggesting that CriticBench's verdict-based evaluation misclassifies about 30% of low-quality critiques as high-quality.","tokens_in":20394,"tokens_out":4041,"duration_ms":40067,"significance":"If the metric's assumption is valid, RealCritic would provide an automatic, objective, and scalable alternative to open-loop critique evaluation, and it would support the practically relevant claim that reasoning-optimized models differ fundamentally from classical LLMs in self-critique capability. The paper is strong on execution: it publicly releases code and data, documents its data-collection and filtering protocol carefully, and includes a human evaluation that directly tests a competing benchmark's construct validity. The core technical idea is timely and likely to influence subsequent work on critique evaluation. However, the central validity assumption is not yet established, and the headline comparisons lack statistical grounding. The benchmark is therefore potentially valuable, but its current evidence base does not support the strength of the paper's conclusions.","major_comments":[{"comment":"The central claim that correction accuracy measures critique quality is not yet supported because (y_critic, y_correction) are drawn jointly from PCritic-LLM(x, s, Icritic), so a model can score highly by ignoring the critique and directly solving the problem or repairing the solution from x and s alone. The 50% baseline used for cross-critique is not an appropriate null, since the model has access to the question and can solve it independently; indeed, direct CoT accuracies in Table 4 are 56.0-67.6% for the classical models, which alone can approximately reproduce several reported after-critique scores. The post-check described in Appendix B only verifies format-level compliance (critiquing before solving and not overtly circumventing identified errors), it was run on a sample of 100 instances (98% compliance), and the authors explicitly state that it was not enforced in the pipeline; it does not establish that the correction tokens are causally derived from the content of the critique. This is load-bearing because it is exactly the premise that distinguishes RealCritic from open-loop evaluation. I recommend adding a control condition where the critique text is replaced by a generic prompt or by a critique of a different solution, or otherwise demonstrating that correction accuracy drops when the critique content is removed.","section":"Section 3.1, Eq. (1); Appendix B"},{"comment":"The headline findings rest on single-run accuracy numbers without variance, confidence intervals, or significance tests. The per-dataset sample sizes are roughly 190-300 items, so differences of a few percentage points (e.g., o1-mini self-critique +3.3% vs. LLaMA-3.1 -4.3%) may be within sampling noise, and some large deltas come from tasks where the base performance is low (e.g., o1-mini College Math direct CoT is 20.1% but self-critique is 44.1%). Moreover, o1-mini uses its official default sampling settings rather than temperature 0, which may add stochasticity. Repeating runs, reporting bootstrap confidence intervals, or performing paired tests across the 8 tasks would materially strengthen the claim that classical LLMs 'significantly lag' o1-mini and that self-critique underperformance is a systematic phenomenon rather than noise.","section":"Table 4; Figure 6"},{"comment":"The manuscript frames the 'shortcut' problem as addressed by the post-check, but the post-check is described as optional and was not applied to the entire pipeline. The text in Appendix B states: 'Given this high compliance rate, we did not implement post-check as a mandatory step in our evaluation pipeline.' Since the authors themselves identify this as a risk to validity, the released benchmark should either enforce the post-check for all generated samples or exclude unqualified outputs; otherwise, the publicly released numbers and the benchmark protocol include a known confound that the authors have not eliminated.","section":"Section 3.1 and Appendix B"}],"minor_comments":[{"comment":"The phrase 'significantly lag' appears in the abstract and conclusion, but no statistical test is reported; please qualify the language until uncertainty is quantified.","section":"General"},{"comment":"The notation P_Critic-LLM(x, s, I_critic) is used for a joint distribution over both y_critic and y_correction; please state explicitly that it is a joint (or factored) distribution to avoid ambiguity.","section":"Section 3.1"},{"comment":"The paper does not describe how final answers are extracted or matched for open-ended math tasks (e.g., exact string match, numeric equivalence, or rule-based extraction); this detail is needed for reproducibility and for interpreting accuracy numbers.","section":"Section 3.3 / Table 3"},{"comment":"The word 'seperation' should be 'separation'.","section":"Figure 1 caption"},{"comment":"The reference list contains duplicate entries for Clark et al. 2018 (arXiv:1803.05457 and arXiv:1803.05457v1); please merge them and use the consistent citation key.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is appealing and the public release is a plus, but the construct-validity concern is genuinely load-bearing. I would not reject outright because the issue can be addressed with additional experiments (e.g., critique-ablation controls, enforced post-checking, significance testing), but in its current form the benchmark's main claim is not yet proven. The authors should also be encouraged to make the raw outputs and post-check annotations available so that reviewers and users can independently verify the critique-correction link."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RealCritic is a useful benchmark paper, and the closed-loop idea is the right instinct, but the central metric measures something looser than critique quality. The headline finding—o1-mini is the only model that improves via self-critique—is plausible but not yet proven.\n\nWhat's new and good: the taxonomy of self-, cross-, and iterative critique is a genuine organizational contribution. The closed-loop framing (score the correction, not the verdict) is a real improvement over CriticBench's verdict accuracy, and the human evaluation showing roughly 30% misclassification on CriticBench is concrete. The benchmark covers eight tasks with balanced correct/incorrect solutions, and the code and data are released. That's solid, reproducible work.\n\nThe soft spot is load-bearing. Section 3.1 defines performance as the expectation of the correction's correctness, where critique and correction are sampled jointly from the same model under one prompt. A model can score well by ignoring the critique and solving the problem directly. The authors are aware and designed a post-check, but Appendix B admits it was applied to 100 instances and not made mandatory in the pipeline. Format compliance (critique before correction) doesn't establish that the correction tokens were causally derived from the critique content. The 50% baseline in cross-critique doesn't fix this: direct CoT accuracy on these tasks is 56–68%, so independent solving alone can produce the reported after-critique scores. That said, the self-critique results for classical models are negative, which is hard to explain by a re-solving shortcut—if models were just re-solving, you'd expect accuracy near direct CoT, not lower. So the shortcut is more likely inflating some cross-critique numbers and confounds the o1-mini advantage, rather than invalidating the entire framework.\n\nOther soft spots are minor: no error bars or significance tests (single temperature-0 runs), and the abstract says 'all critique scenarios' but the benchmark is confined to reasoning tasks. The novelty is real but incremental; CriticEval already had a correction dimension.\n\nWho it's for: people building critique and self-correction benchmarks, and anyone studying o1-style self-improvement. It deserves a serious referee. The revision should add a causal control (e.g., compare correction accuracy with versus without the critique, or corrupt the critique and show corrections degrade), enforce the post-check, and report variance. I'd engage with it.","headline":"RealCritic's closed-loop metric is a useful step past verdict-matching, but the joint critique-correction sampling leaves a real confound that the sampled post-check doesn't resolve.","tokens_in":20977,"tokens_out":3476,"would_cite":true,"duration_ms":31132,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RealCritic argues that a critique should be scored by whether the correction it produces is correct, rather than by whether its verdict matches the solution's truth.","keywords":["critique evaluation","closed-loop benchmarking","self-critique","cross-critique","iterative critique","LLM reasoning","correction accuracy"],"falsifier":"Compare each model's correction accuracy in the critique condition against a control condition where the same prompt asks for a correction with no critique at all; if accuracy does not drop when the critique is removed, the benchmark is measuring solution ability rather than critique quality.","tokens_in":19994,"feed_emoji":"🔁","tokens_out":8843,"duration_ms":70534,"temperature":0.7,"pith_summary":"The paper argues that existing critique benchmarks mis-measure quality because they score the critique itself, typically by verdict matching, rather than its downstream effect. It introduces RealCritic, a closed-loop benchmark on eight challenging reasoning tasks in which a model critiques a solution and then produces a corrected solution, with critique quality defined as the accuracy of that correction. On this metric, o1-mini is the only tested model that improves through self-critique (+3.3% on average), while LLaMA-3.1-70B, Mistral-Large, Qwen2.5-72B variants, and GPT-4o all decline relative to their direct chain-of-thought baselines; o1-mini also shows the largest cross-critique gain (+15.6% over the 50% baseline). A human evaluation on CriticBench supports the critique of open-loop scoring: roughly a third of critiques that pass verdict-matching are judged low-quality by human evaluators.","feed_headline":"Judge LLM critiques by the corrections they produce","feed_subtitle":"Closed-loop tests show only o1-mini gains from self-critique; other models drop.","key_machinery":"The closed-loop critique-and-correction pipeline, with the scoring identity performance $=\\mathbb{E}_{x\\sim\\rho}[\\mathbb{I}(y_{\\mathrm{correction}})]$: after a model emits a critique, the same model must produce a corrected solution, and only the correctness of that corrected solution counts as the critique score. A post-check prompt verifies that the model critiques before solving (98% compliance on a sampled hundred instances, but the check is optional in the main pipeline), while an I-to-C / C-to-I decomposition separates a model's ability to turn wrong solutions into right ones from its tendency to break right ones.","core_discovery":"RealCritic formalizes critique quality as performance $=\\mathbb{E}_{x\\sim\\rho}[\\mathbb{I}(y_{\\mathrm{correction}})]$ where $(y_{\\mathrm{critic}}, y_{\\mathrm{correction}})$ is drawn from the critic model conditioned on the question, the input solution, and a critique instruction, so a critique is good exactly when it produces a correct corrected solution. The benchmark covers self-critique (critiquing one's own generated solution), cross-critique (critiquing pre-collected solutions, balanced equally between correct and incorrect), and iterative critique (multi-round refinement). Its central empirical finding is that only o1-mini improves under self-critique, with large gains on College Math (+24.0%) and ARC (+19.4%), while classical LLMs underperform their direct chain-of-thought baselines; o1-mini also posts the largest cross-critique improvement (+15.6%), and all tested models lose accuracy on GPQA and MMLU-STEM under self-critique. The paper interprets this as a capability separation: comparable direct chain-of-thought accuracy does not imply comparable critique ability.","pith_inferences":["If correction accuracy is the standard, critique quality and problem-solving ability are not cleanly separable in the benchmark: the same model both critiques and corrects, so a strong solver can score high even with an uninformative critique.","A cleaner test would feed the critique to a fixed, weaker solver and measure the change in that solver's accuracy, isolating the critique's contribution from the critic's own solving skill.","The instrumental definition of critique quality extends beyond math: any task with a verifiable output (code, formal proofs, data extraction) could reuse the same closed-loop protocol.","o1-mini's positive self-critique delta, contrasted with classical models' negative deltas, suggests that training on one's own generated corrections rather than on external critique corpora may be the lever that instills self-critique; this is a trainable hypothesis the benchmark could support."],"forward_implications":["Verdict-matching benchmarks can award high scores to critiques whose reasoning would lead to wrong answers; RealCritic's correction-based score would re-rank those cases.","Self-critique is not a universal skill: among tested models only o1-mini reliably turns self-feedback into correctness gains, so critique progress should be measured as a delta over direct reasoning, not as absolute accuracy.","Iterative critique exposes different dynamics across models, with some models' gains decaying across rounds while o1-mini sustains them, making iteration count a useful stress test for long-horizon reasoning.","Because all tested models degrade on GPQA and MMLU-STEM under self-critique, critique-driven refinement can hurt when domain expertise is thin, so benchmark results should be reported by task family rather than only as an average."],"supporting_citations":[{"why":"CriticBench's verdict-matching protocol is the open-loop baseline RealCritic argues against, and its MATH-based test set is used in the human misjudgment study.","marker":"Lin et al. (2024)"},{"why":"CriticEval's separation of feedback and correction dimensions motivates RealCritic's decision to score critique quality through the correction.","marker":"Lan et al. (2024)"},{"why":"Earlier critique-ability work that established the critique-with-verdict generation format that RealCritic replaces with its closed-loop format.","marker":"Luo et al. (2023)"},{"why":"Supplies the premise that critique ability drives LLM self-improvement and bug-catching, motivating the focus on downstream effectiveness.","marker":"McAleese et al. (2024)"},{"why":"Documents that LLMs cannot reliably self-correct without external feedback, the background against which o1-mini's positive self-critique delta is interpreted.","marker":"Huang et al. (2023)"},{"why":"o1-mini is the reasoning-based model whose self-critique and cross-critique results carry the paper's main empirical claims.","marker":"OpenAI (2024)"}],"fun_headline_variants":["Only o1-mini gains from self-critique","Judge LLM critiques by the fixes they produce","Self-critique backfires for most LLMs","Closed-loop test separates critics from imitators"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The score presumes that the corrected answer is correct because of the critique; a model that can solve the problem directly can earn a high critique score without producing a useful critique.","fun_headline_variants_meta":{"raw":{"variants":["Only o1-mini gains from self-critique","Judge LLM critiques by the fixes they produce","Self-critique backfires for most LLMs","Closed-loop test separates critics from imitators"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1244,"prompt_tokens":1003,"completion_tokens":241,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":179}},"tokens_in":619,"tokens_out":241,"duration_ms":3172,"temperature":1.0,"reasoning_tokens":179,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:05:44.075138+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare each model's correction accuracy in the critique condition against a control condition where the same prompt asks for a correction with no critique at all; if accuracy does not drop when the critique is removed, the benchmark is measuring solution ability rather than critique quality.","supporting_citations":[],"review_version":1}