{"id":"071e030b-7fcd-4768-b821-b2d08a85af01","arxiv_id":"2506.15894","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"When a math reasoning chain is perturbed mid-way, several LLMs, including non-reasoning models, can detect the error and complete the solution correctly in the same utterance, with recovery varying strongly by model size and stub source.","lead":"This paper tests whether language models can notice and fix mistakes that are secretly inserted into their own step-by-step reasoning. The results show that several open-weight models, including non-reasoning ones, sometimes catch these errors and recover the correct answer within the same response.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Recovery rate S measures final-answer accuracy after a perturbed stub, not whether the model detected and corrected the injection; without a stub-ignoring control or process annotation, the 'self-correction' numbers are an upper bound.","rationale":"The reader's conditional verdict already identifies protocol measurement as the weak assumption; this pass agrees and isolates the most damaging variant. Grader accuracy is a real but lesser risk, since the grader only checks final answers against a ground-truth string, a comparatively tractable judgment. The assistant-prefill doubt is acknowledged by the authors and mitigated by their heuristic pre-screening, though not fully resolved. The unaddressed confound is that S does not measure detection: it measures any route to a correct final answer. This matters most for the small non-reasoning models, whose reported 'recovery' rates are low enough that chance re-solving could account for a large share; their Appendix D evidence is labeled 'implicit', which is precisely the ambiguous category. The proposed control and annotation would settle the issue: if the recovery rate above an irrelevant-stub baseline is zero, the paper's quantitative claims collapse to the selected examples; if the annotated correction rate is substantial, the central claim survives. The recommendation remains conditional, consistent with the reader, because the qualitative examples provide partial support for an existential reading of the claim; the missing evidence is needed before the broader 'robust... across a range' and 'not finetuned for long CoT' generalizations can be accepted.","tokens_in":11956,"tokens_out":9219,"duration_ms":99584,"concrete_test":"Run the same on-policy perturbed-stub protocol for 100 problems per model, with two additions. (1) Control condition: replace the perturbed stub with an equally long, fluent but task-irrelevant reasoning stub (e.g., a correct solution to a different math problem) before the same completion call; measure final-answer accuracy. (2) Process annotation: have two independent annotators label each completion in the perturbed condition for whether the model explicitly or unambiguously implicitly signals detection of the specific injected error (e.g., a pivot token plus recalculation restoring the original value) before the final answer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline conclusion is that models 'can, in a single utterance, detect and recover from errors in their own reasoning chains' (Section 3.4). The experimental quantity used to support the aggregate version of this claim is final-answer accuracy after the model continues from a perturbed stub, S = c/N (Section 3.2), judged by an LLM grader. A correct final answer does not imply that the model detected the injected error: the model may discard the stub's faulty premise and re-derive the answer from the problem statement, or reach the correct answer by a route that never engages with the perturbation. The paper reports explicit pivot-token corrections in Appendix D, but those are selected examples; the quantitative recovery rates in Table 1 (e.g., Command R7B 10.8%, Nemo 12B 9.3%, Gemma 2 27B 12.1% on GSM8K on-policy, versus 88–91% direct) are not restricted to, or validated against, examples where a correction signal is present. The Section 4 admission that perturbations are 'likely to be significantly off-policy... potentially making the recovery task artificially simple' addresses difficulty, not the re-solving confound. As published, the metric conflates 'correct continuation despite perturbation' with 'self-correction of the perturbation', so the central claim is supported only by anecdotal examples for the models where explicit corrections are shown.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces an experimental protocol for measuring single-utterance intrinsic self-correction in LLMs. A model first generates a reasoning stub, an external LLaMA 3.1 405B model applies a synthetic perturbation to that stub, the same model then completes the generation from the perturbed stub, and a grader checks whether the final answer is correct. The authors evaluate seven models of varying size on GSM8K, GSM-Symbolic, GSM8K Matched, and MATH-500, reporting recovery rates for on-policy and off-policy stubs and providing qualitative examples of implicit and explicit corrections. The central claim is that LLMs, including models not explicitly trained as reasoning models, possess latent single-utterance self-correction capabilities that RL-based reasoning training may amplify.","tokens_in":12172,"tokens_out":5411,"duration_ms":58338,"significance":"If established, the finding would be relevant to current debates about whether RL-trained reasoning models acquire self-correction from scratch or amplify pre-existing latent abilities. The paper has notable strengths: it evaluates multiple model families and datasets, uses an external grader and an external perturbation generator, makes no fitted parameters define the outcome, and includes candid limitations. The qualitative examples in Appendix D are valuable evidence that the phenomenon exists. However, the current quantitative protocol does not cleanly separate 'recovery of the final answer' from 'detection and correction of the injected error,' and the aggregate numbers for most models are much weaker than the word 'robust' suggests. The contribution is therefore promising but needs measurement refinements before the central claim is supported.","major_comments":[{"comment":"The success metric S = c/N measures only final-answer accuracy after completion from a perturbed stub, not whether the model detected and corrected the injected error. A correct final answer can be obtained by ignoring the faulty stub and re-solving from the problem statement, or by an unrelated chain of reasoning, without any internal detection of the perturbation. The central claim in Section 3.4 that 'models can, in a single utterance, detect and recover from errors in their own reasoning chains' is therefore stronger than what this metric supports. I recommend adding a control condition (e.g., completing from the unperturbed stub, or from a stub whose perturbation is not actually task-relevant) and reporting the fraction of completions that contain explicit correction signals or pivot tokens, at least on a labeled subset, so that the quantitative recovery rates can be interpreted as bounds on true corrective behavior.","section":"Section 3.2, Table 1"},{"comment":"The word 'robust' overstates the measured phenomenon. Excluding R1, the average absolute on-policy drop is 61.6%, and on GSM8K the three smallest models recover only a small fraction of their direct accuracy: Command R7B reaches 10.8% versus 88.0% direct, Nemo 12B 9.3% versus 87.7%, and Gemma 2 27B 12.1% versus 90.8%, roughly 10-13% of direct performance. The evidence supports a latent but often weak and highly variable capability, not the 'robust single-utterance intrinsic self-correction behavior' claimed in the abstract. I recommend rewording the high-level claims to distinguish the qualitative existence of correction behavior (supported by Appendix D) from the aggregate robustness claim (not supported by the table).","section":"Section 3.3, Table 1, Abstract"},{"comment":"The LLaMA 3.1 405B grader is used to decide correctness of every model output without any human validation or agreement analysis, and the grader prompt explicitly instructs it to consider only whether the final answer matches the ground truth. Since all absolute values of S and all derived percentages depend on this grader, grader leniency or systematic bias could affect the reported recovery rates. I recommend reporting a human-validated sample of grader judgments or a second independent grader, with agreement statistics, at least on a representative subset of the main table.","section":"Section 3.2, Appendix C"},{"comment":"Two self-acknowledged limitations are load-bearing for the central claim. First, the authors note that the assistant-prefill feature 'cannot be guaranteed' to work as advertised for each model/provider; if the completion does not actually begin from the specified perturbed stub, the experiment does not measure continuation of that stub. Second, they concede that the perturbations are 'likely to be significantly off-policy' and 'potentially making the recovery task artificially simple,' which directly weakens the generalization from synthetic perturbations to natural sampling-induced reasoning errors. I recommend either providing per-model evidence that the prefill feature works as intended, or explicitly restricting the conclusions to the synthetic, best-case setting and removing the broader framing about real reasoning errors.","section":"Section 4, Limitations"}],"minor_comments":[{"comment":"The abstract refers to 'open-weight models,' but Table A2 states that Command R7B is not open-weight; please qualify the claim to avoid an internal inconsistency.","section":"Abstract, Table A2"},{"comment":"The order of scenarios is inconsistent: Table 1 uses 'Direct Off On' while the text and Figure A1 use 'Direct, On-Policy, Off-Policy'; please standardize the column ordering and terminology.","section":"Table 1, Figure 2"},{"comment":"The paper does not report confidence intervals or significance levels for the recovery rates, which is important because the GSM-Symbolic and GSM8K Matched subsets contain only 100 problems; please add uncertainty estimates or state the sample sizes explicitly.","section":"Figure 2, Table 1"},{"comment":"There are several typographical issues, including 'significantly effect' and 'do not meaningfully effect' where 'affect' is intended, and 'problem-solvingtrajectory' missing a space; a light copyedit would fix these.","section":"Section 4"},{"comment":"The appendix shows selected positive examples and one failure case (Figure D15); including a broader sample of failures or a quantitative breakdown of correction-signal frequency would help calibrate how representative the positive examples are.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable empirical contribution to the self-correction debate, but the current experimental quantity does not support the aggregate version of the central claim. The recommended revision is feasible within the paper's scope: add a control or process-level annotation, tighten the language from 'robust' to a capability-level claim, and validate or caveat the grader and prefill assumptions. I also note that the perturbation recipe is closely related to the co-authored Sun et al. (2024) work; the authors should make the boundary of the novel contribution explicit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead 2506.15894. The new thing here is the measurement protocol: take a model's own 100-token CoT stub, inject a localized math error with LLaMA 405B, let the same model continue in one utterance, and score the final answer. That design is a real departure from the three-turn generate-critique-correct setups in Huang et al., and it gives a cleaner view of what happens when an error is already in the context. The model sweep is useful—seven open-weight models, three datasets, on- and off-policy stubs—and the finding that R1 recovers ~90% of direct accuracy while Command R7B recovers ~10% is a real empirical contrast. The explicit correction examples in Appendix D show that at least some non-reasoning models do produce 'Wait' / 'However' pivot tokens and fix their arithmetic mid-generation. I believe those examples; they are the best evidence in the paper.\n\nThe soft spot is the gap between the aggregate success rate S and the word 'self-correction.' S is final-answer accuracy after continuing from a perturbed stub. A correct final answer can also occur if the model simply ignores the faulty stub, re-derives from the problem statement, or never engages with the injected error. Without a control that removes the stub or a process-level annotation on a sample of completions, S is an upper bound on correction, not a measured correction rate. The paper's own examples are hand-picked; the Tables report S for all completions. The authors also acknowledge two additional measurement risks in Section 4—off-policy perturbations may make recovery artificially easy, and assistant-prefill behavior across OpenRouter providers is not guaranteed. Those are honestly stated, but the re-solving confound is not addressed there, and it is the more serious one.\n\nThe 'robust' characterization in the abstract is too strong. Non-R1 models lose on average 61.6% absolute in the on-policy condition; small models lose ~78% and recover only about a tenth of direct accuracy. The data support 'some models can sometimes detect and correct injected reasoning errors in a single utterance,' not 'robust self-correction across models.' That weaker claim is still new and worth having.\n\nVerdict: deserves a serious referee. I would ask for (a) a stub-only control or human annotation of correction events, (b) human-validated grader accuracy, and (c) error bars or multiple seeds. With those, this could be a solid empirical paper. I'd cite it once the control is in. Bring it to reading group if you want a good methods discussion.","headline":"A genuinely new protocol for measuring single-utterance recovery from perturbed reasoning, but the headline claim overstates what the metric shows: final-answer accuracy after a perturbed stub is an upper bound, not proof, of self-correction.","tokens_in":12774,"tokens_out":2945,"would_cite":false,"duration_ms":30164,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Even plain LLMs fix planted math errors mid-answer, without being asked to check their work.","keywords":["self-correction","chain-of-thought","mathematical reasoning","single-utterance generation","perturbed reasoning","latent capability","reinforcement learning","LLM evaluation"],"falsifier":"Human-annotate a random sample of grader verdicts on perturbed-completion outputs and compare final answers to ground truth; if the grader marks wrong final answers as correct, or if models given a stub with no reasoning content complete it with the same accuracy as when given a planted error, the single-utterance recovery result would not demonstrate self-correction.","tokens_in":11680,"feed_emoji":"🧮","tokens_out":6221,"duration_ms":56809,"temperature":0.7,"pith_summary":"This paper asks whether language models can notice and fix errors in their own step-by-step reasoning while they are still generating the answer, without being told to review their work. The authors introduce a protocol in which a model writes the opening of a solution, a separate model sneaks a targeted mathematical mistake into that opening, and the original model then finishes the solution in one continuous utterance. Across seven open-weight models and three math benchmarks, they find that even models not advertised as reasoning models recover from the planted errors a large fraction of the time, sometimes with explicit pivot phrases such as \"Wait\" or \"However.\" The authors read this as evidence that intrinsic self-correction is a latent capability of strong models, and that recent RL-trained reasoning models may be amplifying a pre-existing trait rather than creating it from scratch.","feed_headline":"Even plain LLMs fix planted math errors mid-answer","feed_subtitle":"Even non-reasoning models correct their own math mistakes in one continuous answer, hinting RL amplifies a latent skill.","key_machinery":"The load-bearing mechanism is a four-phase stub-perturb-complete-verify protocol. A model generates the first 100 tokens of a solution; a held-out model rewrites that stub with one targeted mathematical or reasoning error; the original model completes the perturbed stub as a single uninterrupted utterance; and a grader checks the final answer against ground truth. Because the stub and completion are presented as one turn, any correction must be intrinsic: there is no second pass, no critique prompt, and no external verifier inside the generation. The paper's evidence is the collection of completions containing pivot tokens, corrected arithmetic, and explicit acknowledgments of mistakes.","core_discovery":"The central claim is that current language models, including models not finetuned for long chain-of-thought, can perform single-utterance intrinsic self-correction: after an error is inserted into their own reasoning stub, they detect it, acknowledge it implicitly or explicitly, and produce a corrected continuation, all within one uninterrupted generation. The authors report recovery across Command R7B, Nemo 12B, Gemma 2 27B, QwQ 32B Preview, LLaMA 3.3 70B, Qwen 2.5 72B, and R1 on GSM8K, GSM-Symbolic, and MATH-500, with larger models recovering better and R1 nearly unaffected. What matters for the paper's thesis is that non-reasoning models show the behavior in a weaker form; the conclusion is that RL-based reasoning training amplifies latent self-correction rather than supplying it as a new skill.","pith_inferences":["If the latent trait is real, failures in multi-turn self-critique setups may be elicitation failures rather than absence of ability; a direct test would run the same perturbed stubs under a three-turn critique prompt and compare recovery.","A stronger test would inject perturbations sampled from the model's own high-probability wrong continuations; the paper notes its perturbations are likely off-policy, so reported recovery rates may overstate correction of natural errors.","Because the grader's verdicts are not human-validated, a small human audit of final answers on perturbed completions would bound how much of the reported recovery is genuine correction rather than grader leniency."],"forward_implications":["If the latent-capability claim holds, intrinsic self-correction should be measurable in standard instruction-tuned models before any RL-for-reasoning training, and it should appear without explicit critique prompts.","Reasoning-model training can be understood as amplification: gains from RL should correlate with the base model's perturbed-stub recovery rate.","Perturbed-stub recovery offers a lightweight robustness probe that costs one generation and needs no extra self-critique tokens.","Model capability and generation style can be coupled, since QwQ's recovery drops sharply when it must continue an off-policy reasoning stub, warning that reasoning performance may not transfer across formats."],"supporting_citations":[{"why":"Supplies the R1 model and the RL-trained \"aha moment\" behavior that the paper's pivot-token observations are compared against.","marker":"DeepSeek (2025)"},{"why":"Provides the three-turn self-correction negative result that motivates testing single-utterance intrinsic correction.","marker":"Huang et al. (2024)"},{"why":"Provides the taxonomy of perturbation types, including operator swaps and phrase changes, adapted for stub perturbation.","marker":"Sun et al. (2024)"},{"why":"Provides the GSM8K benchmark used for stub generation and recovery evaluation.","marker":"Cobbe et al. (2021)"},{"why":"Provides the MATH benchmark from which the MATH-500 subset is drawn.","marker":"Hendrycks et al. (2021)"},{"why":"Provides GSM-Symbolic, used to check that recovery behavior is not dataset-specific.","marker":"Mirzadeh et al. (2024)"},{"why":"Defines the 500-problem MATH subset used to keep evaluation costs feasible.","marker":"Lightman et al. (2023)"},{"why":"Supplies QwQ 32B Preview, one of the reasoning-style models whose off-policy stub sensitivity is analyzed.","marker":"Qwen Team (2024)"}],"fun_headline_variants":["LLMs self-correct reasoning errors in one uninterrupted pass","Even non-reasoning models fix their own math slip-ups mid-answer","Single-utterance self-correction: LLMs catch their own CoT mistakes","Self-correction is latent in LLMs, RL just amplifies it","Models correct planted reasoning errors without extra tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the experiment is actually measuring self-correction: that each completion genuinely continues the perturbed stub, that the grader's correct and incorrect verdicts are trustworthy, and that the planted errors resemble the mistakes models really make.","fun_headline_variants_meta":{"raw":{"variants":["LLMs self-correct reasoning errors in one uninterrupted pass","Even non-reasoning models fix their own math slip-ups mid-answer","Single-utterance self-correction: LLMs catch their own CoT mistakes","Self-correction is latent in LLMs, RL just amplifies it","Models correct planted reasoning errors without extra tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000388,"raw_usage":{"total_tokens":2021,"prompt_tokens":895,"completion_tokens":1126,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":1036}},"tokens_in":511,"tokens_out":1126,"duration_ms":9531,"temperature":1.0,"reasoning_tokens":1036,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:28:52.597487+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Human-annotate a random sample of grader verdicts on perturbed-completion outputs and compare final answers to ground truth; if the grader marks wrong final answers as correct, or if models given a stub with no reasoning content complete it with the same accuracy as when given a planted error, the single-utterance recovery result would not demonstrate self-correction.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the R1 model and the RL-trained \"aha moment\" behavior that the paper's pivot-token observations are compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MATH benchmark from which the MATH-500 subset is drawn."}],"review_version":1}