Language Models can perform Single-Utterance Self-Correction of Perturbed Reasoning

Eddie Kim; Ivan Zhang; Jimin Sun; Sam Silver; Sara Hooker

REVIEW 4 major objections 5 minor 1 cited by

Language Models can perform Single-Utterance Self-Correction of Perturbed Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Even plain LLMs fix planted math errors mid-answer, without being asked to check their work.

desk verdict A genuinely new protocol for measuring single-utterance recovery from perturbed reasoning, but the headline claim overstates what the metric shows: final-answer accuracy after a perturbed stub is an upper bound, not proof, of self-correction. read the letter →

arxiv 2506.15894 v1 pith:O4CBYOB5 submitted 2025-06-18 cs.CL cs.AI

Sam Silver , Jimin Sun , Ivan Zhang , Sara Hooker , Eddie Kim This is my paper · ORCID

classification cs.CLcs.AI

keywords self-correctionchain-of-thoughtmathematicalreasoningsingle-utterancegenerationperturbedlatentcapabilityreinforcementlearningLLMevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether language models can notice and fix errors in their own step-by-step reasoning while they are still generating the answer, without being told to review their work. The authors introduce a protocol in which a model writes the opening of a solution, a separate model sneaks a targeted mathematical mistake into that opening, and the original model then finishes the solution in one continuous utterance. Across seven open-weight models and three math benchmarks, they find that even models not advertised as reasoning models recover from the planted errors a large fraction of the time, sometimes with explicit pivot phrases such as "Wait" or "However." The authors read this as evidence that intrinsic self-correction is a latent capability of strong models, and that recent RL-trained reasoning models may be amplifying a pre-existing trait rather than creating it from scratch.

What carries the argument

The load-bearing mechanism is a four-phase stub-perturb-complete-verify protocol. A model generates the first 100 tokens of a solution; a held-out model rewrites that stub with one targeted mathematical or reasoning error; the original model completes the perturbed stub as a single uninterrupted utterance; and a grader checks the final answer against ground truth. Because the stub and completion are presented as one turn, any correction must be intrinsic: there is no second pass, no critique prompt, and no external verifier inside the generation. The paper's evidence is the collection of completions containing pivot tokens, corrected arithmetic, and explicit acknowledgments of mistakes.

What would settle it

Human-annotate a random sample of grader verdicts on perturbed-completion outputs and compare final answers to ground truth; if the grader marks wrong final answers as correct, or if models given a stub with no reasoning content complete it with the same accuracy as when given a planted error, the single-utterance recovery result would not demonstrate self-correction.

Watch

Extended reading notes

Core claim

The central claim is that current language models, including models not finetuned for long chain-of-thought, can perform single-utterance intrinsic self-correction: after an error is inserted into their own reasoning stub, they detect it, acknowledge it implicitly or explicitly, and produce a corrected continuation, all within one uninterrupted generation. The authors report recovery across Command R7B, Nemo 12B, Gemma 2 27B, QwQ 32B Preview, LLaMA 3.3 70B, Qwen 2.5 72B, and R1 on GSM8K, GSM-Symbolic, and MATH-500, with larger models recovering better and R1 nearly unaffected. What matters for the paper's thesis is that non-reasoning models show the behavior in a weaker form; the conclusion is that RL-based reasoning training amplifies latent self-correction rather than supplying it as a new skill.

Load-bearing premise

The load-bearing premise is that the experiment is actually measuring self-correction: that each completion genuinely continues the perturbed stub, that the grader's correct and incorrect verdicts are trustworthy, and that the planted errors resemble the mistakes models really make.

Editorial extensions

If this is right

If the latent-capability claim holds, intrinsic self-correction should be measurable in standard instruction-tuned models before any RL-for-reasoning training, and it should appear without explicit critique prompts.
Reasoning-model training can be understood as amplification: gains from RL should correlate with the base model's perturbed-stub recovery rate.
Perturbed-stub recovery offers a lightweight robustness probe that costs one generation and needs no extra self-critique tokens.
Model capability and generation style can be coupled, since QwQ's recovery drops sharply when it must continue an off-policy reasoning stub, warning that reasoning performance may not transfer across formats.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the latent trait is real, failures in multi-turn self-critique setups may be elicitation failures rather than absence of ability; a direct test would run the same perturbed stubs under a three-turn critique prompt and compare recovery.
A stronger test would inject perturbations sampled from the model's own high-probability wrong continuations; the paper notes its perturbations are likely off-policy, so reported recovery rates may overstate correction of natural errors.
Because the grader's verdicts are not human-validated, a small human audit of final answers on perturbed completions would bound how much of the reported recovery is genuine correction rather than grader leniency.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Read 2506.15894. The new thing here is the measurement protocol: take a model's own 100-token CoT stub, inject a localized math error with LLaMA 405B, let the same model continue in one utterance, and score the final answer. That design is a real departure from the three-turn generate-critique-correct setups in Huang et al., and it gives a cleaner view of what happens when an error is already in the context. The model sweep is useful—seven open-weight models, three datasets, on- and off-policy stubs—and the finding that R1 recovers ~90% of direct accuracy while Command R7B recovers ~10% is a real empirical contrast. The explicit correction examples in Appendix D show that at least some non-reasoning models do produce 'Wait' / 'However' pivot tokens and fix their arithmetic mid-generation. I believe those examples; they are the best evidence in the paper.

The soft spot is the gap between the aggregate success rate S and the word 'self-correction.' S is final-answer accuracy after continuing from a perturbed stub. A correct final answer can also occur if the model simply ignores the faulty stub, re-derives from the problem statement, or never engages with the injected error. Without a control that removes the stub or a process-level annotation on a sample of completions, S is an upper bound on correction, not a measured correction rate. The paper's own examples are hand-picked; the Tables report S for all completions. The authors also acknowledge two additional measurement risks in Section 4—off-policy perturbations may make recovery artificially easy, and assistant-prefill behavior across OpenRouter providers is not guaranteed. Those are honestly stated, but the re-solving confound is not addressed there, and it is the more serious one.

The 'robust' characterization in the abstract is too strong. Non-R1 models lose on average 61.6% absolute in the on-policy condition; small models lose ~78% and recover only about a tenth of direct accuracy. The data support 'some models can sometimes detect and correct injected reasoning errors in a single utterance,' not 'robust self-correction across models.' That weaker claim is still new and worth having.

Verdict: deserves a serious referee. I would ask for (a) a stub-only control or human annotation of correction events, (b) human-validated grader accuracy, and (c) error bars or multiple seeds. With those, this could be a solid empirical paper. I'd cite it once the control is in. Bring it to reading group if you want a good methods discussion.

Referee Report

4 major / 5 minor

Summary. The paper introduces an experimental protocol for measuring single-utterance intrinsic self-correction in LLMs. A model first generates a reasoning stub, an external LLaMA 3.1 405B model applies a synthetic perturbation to that stub, the same model then completes the generation from the perturbed stub, and a grader checks whether the final answer is correct. The authors evaluate seven models of varying size on GSM8K, GSM-Symbolic, GSM8K Matched, and MATH-500, reporting recovery rates for on-policy and off-policy stubs and providing qualitative examples of implicit and explicit corrections. The central claim is that LLMs, including models not explicitly trained as reasoning models, possess latent single-utterance self-correction capabilities that RL-based reasoning training may amplify.

Significance. If established, the finding would be relevant to current debates about whether RL-trained reasoning models acquire self-correction from scratch or amplify pre-existing latent abilities. The paper has notable strengths: it evaluates multiple model families and datasets, uses an external grader and an external perturbation generator, makes no fitted parameters define the outcome, and includes candid limitations. The qualitative examples in Appendix D are valuable evidence that the phenomenon exists. However, the current quantitative protocol does not cleanly separate 'recovery of the final answer' from 'detection and correction of the injected error,' and the aggregate numbers for most models are much weaker than the word 'robust' suggests. The contribution is therefore promising but needs measurement refinements before the central claim is supported.

major comments (4)

[Section 3.2, Table 1] The success metric S = c/N measures only final-answer accuracy after completion from a perturbed stub, not whether the model detected and corrected the injected error. A correct final answer can be obtained by ignoring the faulty stub and re-solving from the problem statement, or by an unrelated chain of reasoning, without any internal detection of the perturbation. The central claim in Section 3.4 that 'models can, in a single utterance, detect and recover from errors in their own reasoning chains' is therefore stronger than what this metric supports. I recommend adding a control condition (e.g., completing from the unperturbed stub, or from a stub whose perturbation is not actually task-relevant) and reporting the fraction of completions that contain explicit correction signals or pivot tokens, at least on a labeled subset, so that the quantitative recovery rates can be interpreted as bounds on true corrective behavior.
[Section 3.3, Table 1, Abstract] The word 'robust' overstates the measured phenomenon. Excluding R1, the average absolute on-policy drop is 61.6%, and on GSM8K the three smallest models recover only a small fraction of their direct accuracy: Command R7B reaches 10.8% versus 88.0% direct, Nemo 12B 9.3% versus 87.7%, and Gemma 2 27B 12.1% versus 90.8%, roughly 10-13% of direct performance. The evidence supports a latent but often weak and highly variable capability, not the 'robust single-utterance intrinsic self-correction behavior' claimed in the abstract. I recommend rewording the high-level claims to distinguish the qualitative existence of correction behavior (supported by Appendix D) from the aggregate robustness claim (not supported by the table).
[Section 3.2, Appendix C] The LLaMA 3.1 405B grader is used to decide correctness of every model output without any human validation or agreement analysis, and the grader prompt explicitly instructs it to consider only whether the final answer matches the ground truth. Since all absolute values of S and all derived percentages depend on this grader, grader leniency or systematic bias could affect the reported recovery rates. I recommend reporting a human-validated sample of grader judgments or a second independent grader, with agreement statistics, at least on a representative subset of the main table.
[Section 4, Limitations] Two self-acknowledged limitations are load-bearing for the central claim. First, the authors note that the assistant-prefill feature 'cannot be guaranteed' to work as advertised for each model/provider; if the completion does not actually begin from the specified perturbed stub, the experiment does not measure continuation of that stub. Second, they concede that the perturbations are 'likely to be significantly off-policy' and 'potentially making the recovery task artificially simple,' which directly weakens the generalization from synthetic perturbations to natural sampling-induced reasoning errors. I recommend either providing per-model evidence that the prefill feature works as intended, or explicitly restricting the conclusions to the synthetic, best-case setting and removing the broader framing about real reasoning errors.

minor comments (5)

[Abstract, Table A2] The abstract refers to 'open-weight models,' but Table A2 states that Command R7B is not open-weight; please qualify the claim to avoid an internal inconsistency.
[Table 1, Figure 2] The order of scenarios is inconsistent: Table 1 uses 'Direct Off On' while the text and Figure A1 use 'Direct, On-Policy, Off-Policy'; please standardize the column ordering and terminology.
[Figure 2, Table 1] The paper does not report confidence intervals or significance levels for the recovery rates, which is important because the GSM-Symbolic and GSM8K Matched subsets contain only 100 problems; please add uncertainty estimates or state the sample sizes explicitly.
[Section 4] There are several typographical issues, including 'significantly effect' and 'do not meaningfully effect' where 'affect' is intended, and 'problem-solvingtrajectory' missing a space; a light copyedit would fix these.
[Appendix D] The appendix shows selected positive examples and one failure case (Figure D15); including a broader sample of failures or a quantitative breakdown of correction-signal frequency would help calibrate how representative the positive examples are.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: recovery-rate S is an external, unfitted measurement; the self-correction interpretation is a construct-validity concern, not a definitional reduction.

full rationale

The claimed derivation is an empirical measurement chain, not a formal derivation: each model generates a stub, a held-out LLM perturbs it, the evaluated model continues in a single assistant turn, and a grader LLM scores final-answer correctness. The headline capability ('models can, in a single utterance, detect and recover from errors in their own reasoning chains') is an interpretation of the recovery rate S = c/N. S is not constructed from detection events; the paper does not define 'self-correction' as 'final answer correct after perturbation,' so the conclusion is not equivalent to the metric by definition. The gap is construct validity—a correct completion could arise from ignoring the stub and re-solving, or from grader leniency—which the paper partly acknowledges in its limitations, not circularity. No parameter is fitted to the target result, and no uniqueness theorem or load-bearing prior claim by the same authors is invoked. The perturbation recipe is attributed to Sun et al. (2024), and Jimin Sun is a co-author of the present paper, but the perturbation protocol is fully specified in Appendix C and the cited work is not used as an unverified premise; the citation is therefore not load-bearing. The Section 4 admissions that perturbations are 'likely to be significantly off-policy... potentially making the recovery task artificially simple' and that assistant-prefill behavior 'cannot be guaranteed' weaken external validity but do not make the outcome equivalent to the input. Accordingly, no circular step can be exhibited under the required reduction standard, and the score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical measurement, so the ledger records the experiment-design choices that the conclusions depend on: grader accuracy, perturbation realism, assistant-prefill reliability, and the use of final-answer accuracy as a proxy for self-correction. No free parameters are fitted in the derivation sense, but several hyperparameters and subset sizes are hand-chosen and affect the reported recovery rates.

free parameters (4)

Stub length = 100 tokens
Chosen by hand as a balance between enough progress to perturb and enough headroom to recover; not varied or justified.
Sampling temperature = 0.2
Fixed for all solution generation; affects diversity and recovery difficulty, not swept.
Top-P = 0.8
Nucleus sampling threshold fixed for completion generation.
GSM-Symbolic subset size = 100 problems
A hand-chosen sample derived from 100 unique GSM8K problems; not a full benchmark.

assumptions (4)

domain assumption LLaMA 3.1 405B grader verdicts are accurate
The correctness metric Sa = c/N is defined by LLM grader output with no human validation or calibration check, and the grader is from the same family as some evaluated models.
domain assumption Perturbed stubs are realistic reasoning errors
The authors state in Section 4 that injected errors are "likely to be significantly off-policy with respect to the models under evaluation, potentially making the recovery task artificially simple."
domain assumption Assistant prefill works as intended for all providers
Section 4: "we cannot guarantee with certainty that the assistant-prefill feature functions as advertised for each model/provider combination."
domain assumption Final-answer accuracy is a valid proxy for self-correction
The grader only checks the final answer and explicitly ignores reasoning errors, so a model could succeed by ignoring the perturbed stub rather than correcting it; the aggregate metric cannot distinguish the two.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Language Models can perform Single-Utterance Self-Correction of Perturbed Reasoning." pith.science (2026). https://pith.science/paper/O4CBYOB5

@misc{pith2026250615894,
  author       = {Pith},
  title        = {Pith review of: Language Models can perform Single-Utterance Self-Correction of Perturbed Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O4CBYOB5}},
  note         = {Machine review of arXiv:2506.15894}
}

read the original abstract

Large Language Models (LLMs) have demonstrated impressive mathematical reasoning capabilities, yet their performance remains brittle to minor variations in problem description and prompting strategy. Furthermore, reasoning is vulnerable to sampling-induced errors which autoregressive models must primarily address using self-correction via additionally-generated tokens. To better understand self-correction capabilities of recent models, we conduct experiments measuring models' ability to self-correct synthetic perturbations introduced into their Chain of Thought (CoT) reasoning. We observe robust single-utterance intrinsic self-correction behavior across a range of open-weight models and datasets, ranging from subtle, implicit corrections to explicit acknowledgments and corrections of errors. Our findings suggest that LLMs, including those not finetuned for long CoT, may possess stronger intrinsic self-correction capabilities than commonly shown in the literature. The presence of this ability suggests that recent "reasoning" model work involves amplification of traits already meaningfully present in models.

Figures

Figures reproduced from arXiv: 2506.15894 by the authors.

**Figure 2.** Perturbation recovery success rate in on-policy (orange) and off-policy (blue) reasoning stub scenarios [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents
cs.CL 2026-08 conditional novelty 5.0 of 10

A survey of 1,547 papers defines the 'horizon gap' and documents that long-horizon agent research is converging on trajectory-level process signals instead of outcome-only scores.

Reference graph

Works this paper leans on

46 extracted references · 21 canonical work pages · cited by 1 Pith paper

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Mistral AI. 2024. https://mistral.ai/news/mistral-nemo/ Mistral NeMo : A state-of-the-art 12b model . Blog post

work page 2024
[4]

Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

arXiv 2022
[5]

Chi, Xuezhi Wang, and Denny Zhou

Xinyun Chen, Ryan A. Chi, Xuezhi Wang, and Denny Zhou. 2024. Premise order matters in reasoning with large language models. In International Conference on Machine Learning

work page 2024
[6]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. http://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems

arXiv 2021
[7]

Cohere. 2024. https://cohere.com/blog/command-r7b Introducing Command R7B : Fast and efficient generative AI . Blog post

work page 2024
[8]

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. http://arxiv.org/abs/2310.01377 Ultrafeedback: Boosting language models with scaled ai feedback

arXiv 2024

Show all 46 references

[9]

DeepSeek . 2025. https://github.com/deepseek-ai/DeepSeek-R1/blob/main/DeepSeek_R1.pdf Deepseek-r1: Incentivizing reasoning capability in LLM s via reinforcement learning . Research@deepseek.com

2025
[10]

Tenenbaum, and Igor Mordatch

Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023. http://arxiv.org/abs/2305.14325 Improving factuality and reasoning in language models through multiagent debate

2023 arXiv
[11]

Aaron Grattafiori et al. 2024. http://arxiv.org/abs/2407.21783 The llama 3 herd of models

2024 arXiv
[12]

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024. CRITIC : Large language models can self-correct with tool-interactive critiquing. In International Conference on Learning Representations

2024
[13]

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track

2021
[14]

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations

2020
[15]

Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large language models cannot self-correct reasoning yet. In International Conference on Learning Representations

2024
[16]

Pei Ke, Bosi Wen, Andrew Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, Jie Tang, and Minlie Huang. 2024. https://doi.org/10.18653/v1/2024.acl-long.704 C ritique LLM : Towards an informative critique generation model for evalua...

2024 doi
[17]

Kimi Team . 2025. https://github.com/MoonshotAI/Kimi-k1.5/blob/main/Kimi_k1.5.pdf Kimi k1.5: Scaling reinforcement learning with LLM s

2025
[18]

Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust. 2024...

2024 arXiv
[19]

Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luc...

2024 arXiv
[20]

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving quantitative reasoning problems with language mo...

2022
[21]

Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. 2024. Numinamath. [https://huggingface.co/AI-MO/Numi...

2024
[22]

Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2024. Generative judge for evaluating alignment. In International Conference on Learning Representations

2024
[23]

Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. http://arxiv.org/abs/2305.19118 Encouraging divergent thinking in large language models through multi-agent debate

2024 arXiv
[24]

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. http://arxiv.org/abs/2305.20050 Let's verify step by step

2023 arXiv
[25]

Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. 2025. http://arxiv.org/abs/2502.01100 Zebralogic: On the scaling limits of llms for logical reasoning

2025 arXiv
[26]

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. https://proceed...

2023
[27]

Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2024. http://arxiv.org/abs/2410.05229 Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models

2024 arXiv
[28]

Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2022. Show your work: Scratchpads for intermediate computation with language models. ...

2022
[29]

OpenAI . 2024. https://openai.com/index/learning-to-reason-with-llms/ Learning to Reason with LLMs . OpenAI Blog

2024
[30]

Sundar Pichai, Demis Hassabis, and Koray Kavukcuoglu. 2024. https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/#project-astra Introducing Gemini 2.0: Our new AI model for the agentic era . The Keyword (Google Blog)

2024
[31]

Shuofei Qiao, Honghao Gui, Chengfei Lv, Qianghuai Jia, Huajun Chen, and Ningyu Zhang. 2024. Making language models better tool learners with execution feedback. In Annual Meeting of the Association for Computational Linguistics

2024
[32]

Qwen Team . 2024. https://qwenlm.github.io/blog/qwq-32b-preview/ Qwq: Reflect deeply on the boundaries of the unknown . Blog post

2024
[33]

William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. http://arxiv.org/abs/2206.05802 Self-critiquing models for assisting human evaluators

2022 arXiv
[34]

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Sch \"a rli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning

2023
[35]

Jimin Sun, So Yeon Min, Yingshan Chang, and Yonatan Bisk. 2024. http://arxiv.org/abs/2406.19228 Tools fail: Detecting silent errors in faulty tools

2024 arXiv
[36]

Gemma Team. 2024. http://arxiv.org/abs/2408.00118 Gemma 2: Improving open language models at a practical size

2024 arXiv
[37]

Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang...

2024 arXiv
[38]

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. http://arxiv.org/abs/2211.14275 Solving math word problems with process- and outcome-based feedback

2022 arXiv
[39]

Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O'Brien, Ramakanth Pasunuru, Jane Dwivedi-Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023. http://arxiv.org/abs/2308.04592 Shepherd: A critic for language model generation

2023 arXiv
[40]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf Chain-of-thought prompting elicits reasoning...

2022
[41]

Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2023. Generating sequences by learning to self-correct. In International Conference on Learning Representations

2023
[42]

An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024. http://arxiv.org/abs/2409.12122 Qwen2.5-math technical report: T...

2024 arXiv
[43]

Seonghyeon Ye, Yongrae Jo, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, and Minjoon Seo. 2023. https://kaistai.github.io/SelFee/ SelFee : Iterative self-revising LLM empowered by self-feedback generation . Blog post

2023
[44]

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. In International Conference on Machine Learning

2024
[45]

Chi, Quoc V Le, and Denny Zhou

Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V Le, and Denny Zhou. 2024. Take a step back: Evoking reasoning via abstraction in large language models. In International Conference on Learning Representations

2024
[46]

Denny Zhou, Nathanael Sch \"a rli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. In International Conference on Learning Repre...

2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Mistral AI. 2024. https://mistral.ai/news/mistral-nemo/ Mistral NeMo : A state-of-the-art 12b model . Blog post

work page 2024

[4] [4]

Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

arXiv 2022

[5] [5]

Chi, Xuezhi Wang, and Denny Zhou

Xinyun Chen, Ryan A. Chi, Xuezhi Wang, and Denny Zhou. 2024. Premise order matters in reasoning with large language models. In International Conference on Machine Learning

work page 2024

[6] [6]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. http://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems

arXiv 2021

[7] [7]

Cohere. 2024. https://cohere.com/blog/command-r7b Introducing Command R7B : Fast and efficient generative AI . Blog post

work page 2024

[8] [8]

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. http://arxiv.org/abs/2310.01377 Ultrafeedback: Boosting language models with scaled ai feedback

arXiv 2024