Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

On the Emergence of Thinking in LLMs I: Searching for the Right Intuition

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper's central claim is that decoupling an exploration reward from a correctness reward in PPO training induces emergent search behaviors—backtracking, verification, and exploration of alternatives—in LLMs and improves mathematical…

desk verdict A clear, honest proposal for decoupling exploration from correctness in RL post-training; the main tables miss the outcome-only PPO control, so the causal claim is promising but not yet proven. read the letter →

arxiv 2502.06773 v1 pith:WKBGRUX4 submitted 2025-02-10 cs.AI cs.CLcs.LG

Guanghao Ye , Khiem Duc Pham , Xinzhi Zhang , Sivakanth Gopi , Baolin Peng , Beibin Li , Janardhan Kulkarni , Huseyin A. Inan This is my paper · ORCID
classification cs.AIcs.CLcs.LG
keywords reinforcementlearninglargereasoningmodelsemergentsearchbehaviorchain-of-thoughtbacktrackingself-verificationPPOmathematical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement Learning via Self-Play (RLSP) is a post-training recipe that separates the reward for thinking effort from the reward for a correct answer. The paper's central claim is that a minimal exploration reward—even simply rewarding longer responses that spell out intermediate steps—makes search behaviors such as backtracking, verification, and exploring alternative approaches emerge during PPO training, in some cases with no supervised fine-tuning at all. If correct, this offers a scalable path toward 'thinking' language models: instead of engineering search logic, one needs only an outcome verifier and a weak signal that effort is valued. Empirically, RLSP lifts MATH-500 accuracy by 23% for an 8B model and AIME 2024 accuracy by 10% for a 32B model, and matches or beats majority voting under the same token budget.

What carries the argument

The load-bearing mechanism is the two-component RLSP reward combined with the PPO update that propagates it. The exploration reward $R_{\mathrm{ex}}(q,o) = -C/|o|$ is the simplest instance: longer responses that explicitly write out intermediate steps receive higher reward, under the assumption—made precise by the chain-of-thought expressivity results the paper cites—that more intermediate steps increase a transformer's computational power. The outcome-verifier component keeps the search tethered to correct answers and, because it carries 80% of the weight, counteracts reward hacking. The KL penalty to the reference policy is also part of the machinery: since the KL term suppresses long responses, the exploration reward partly exists to counteract that suppression so that long reasoning traces can survive training.

What would settle it

Train an 8B model with RLSP's length reward and, in parallel, with a control reward that pays for long but repetitive output while keeping the outcome-verifier component identical. If the control model matches RLSP in response length and shows the same backtracking, verification, and accuracy gains, then length rather than search-oriented exploration is the active ingredient and the emergence claim would be falsified. A second check is to evaluate RLSP's MATH-500 gains only on problems whose trajectories contain at least one backtrack or verification event; if the gains are not concentrated there, the proposed causal link between emergent search and improved reasoning is weakened.

Watch

Extended reading notes

Core claim

The paper's central discovery is that emergent reasoning behavior can be induced by decoupling exploration from correctness in the reward signal. During PPO training, the total reward is $R(q,o) = \alpha \cdot \mathbf{1}[\mathrm{Ver}(q,o)=\mathrm{True}] + (1-\alpha) R_{\mathrm{ex}}(q,o)$, with $R_{\mathrm{ex}}(q,o) = -C/|o|$ in the simplest implementation. With $\alpha=0.8$, most of the reward still comes from a binary outcome verifier, but the model also receives a dense, correctness-independent incentive to produce longer, more explicit reasoning trajectories. Across two model families and both math and coding domains, models trained this way produced trajectories that abandon a failed approach, re-examine earlier steps, verify an answer through multiple routes, and try several rationales before concluding—behaviors that did not appear under outcome-reward-only PPO except in one model-and-domain combination. The paper treats these behaviors as emergent relative to standard chain-of-thought, while acknowledging that they are implicitly present, less frequently, in pretraining data, and it interprets RLSP as enabling the model to synthesize novel chain-of-thought traces through self-play.

Load-bearing premise

The load-bearing premise is that response length is a valid proxy for reasoning effort, so rewarding longer outputs teaches genuine search rather than repetition; the paper itself concedes that models can hack this signal by repeating responses, and that the behaviors it calls emergent are already present, less frequently, in pretraining data.

Editorial extensions

If this is right

  • Even a crude length-based exploration reward, applied without SFT, elicits backtracking, verification, and alternative-approach search across several base models and domains.
  • Under a fixed token budget, RLSP-trained models beat self-consistency majority voting: 73.4% versus 61.8% on MATH-500 for the 8B model, and 23.3% versus 20% on AIME 2024 for the 32B model.
  • The recipe needs only an outcome verifier and an exploration signal, so it can transfer to any domain with verifiable answers without process reward models or hand-constructed search trees.
  • Because the model generates its own chain-of-thought traces during training, the framework predicts continued self-improvement as long as sufficiently diverse new problems are available.
  • The decoupled reward structure suggests a scaling recipe: with more compute and more problems, the length signal can be refined into richer effort signals for further gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A sharper test of the length-reward theory: stratify MATH-500 by problem difficulty and check whether RLSP's accuracy gain grows with the number of search steps the trajectory actually contains; the chain-of-thought expressivity argument predicts exactly that monotonic relationship.
  • The paper's own Remark 2 suggests a confound: if backtracking is already present in pretraining data, then the observed behaviors could be unmasked rather than newly created. Comparing models pretrained on corpora with different rates of self-correction, while holding the RLSP reward fixed, would separate these explanations.
  • The decoupling principle extends naturally to other effort signals—diversity of sampled rationales, number of distinct methods tried, or a learned judge of creativity—and the framework predicts each would induce similar search behavior; this is testable by replacing $-C/|o|$ with each signal and measuring emergent backtracking.
  • If the theory's self-play story is right, RLSP should compound: iterative rounds of RLSP on the model's own successful long trajectories should keep improving accuracy beyond a single round, which the current single-round experiments do not yet test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes RLSP (Reinforcement Learning via Self-Play), a three-stage post-training recipe: (1) optional SFT on reasoning traces, (2) an exploration reward independent of answer correctness (e.g., a length bonus), and (3) PPO with an outcome verifier, with the total reward a convex combination of correctness and exploration terms (Eq. (1)). The authors report that on Llama-3.1-8B-Instruct, SFT plus exploration-reward RL reaches 73.4% on MATH-500 versus 70.2% for SFT alone, and on Qwen2.5-32B-Instruct, exploration-reward RL improves AIME 2024 pass@1 from 13.3% to 23.3%. They further claim that even without SFT, the length reward induces emergent search behaviors (backtracking, self-correction, verification, alternative methods), illustrated with qualitative traces in Figures 1–3 and Appendix A. The paper ends with a heuristic argument, grounded in chain-of-thought expressivity results, for why rewarding longer trajectories should create self-play over reasoning traces.

Significance. If the causal role of the exploration reward were established, this would be a useful design principle for reasoning post-training, and the public code plus decontamination efforts are valuable assets. The strengths of the paper are its clear framework, its honesty about limitations (Remark 2, Section 6), and concrete benchmark comparisons. However, the central attribution of performance gains to the exploration term is not yet supported: the headline improvement on Llama is mostly from SFT (50.6→70.2), and the marginal 3.2-point gain from ER-RLSP lacks an outcome-only PPO control and error bars. Similarly, the emergence claim relies on selected traces rather than quantitative behavior measurement, and the length reward partially builds the observed behavior into the objective. These are omitted controls and measurement gaps, not contradictions, so the paper remains promising, but the framework's distinctive contribution requires further evidence.

major comments (3)
  1. [§4.1, §4.2, Tables 1–2; Eq. (1)] The performance claims attribute the gains to the exploration reward in Eq. (1), but the comparison arms are SFT vs. SFT+ER-RLSP and Base vs. Base+ER-RLSP. No arm trains with only the outcome verifier (α=1). Because §3 demonstrates that outcome-only PPO can increase response length for Qwen2.5-7B, the 10-point AIME gain for Qwen2.5-32B could plausibly be due to outcome reward alone. Please add SFT+PPO(α=1) and Base+PPO(α=1) controls, and report multiple seeds or confidence intervals, especially since the Llama gain (70.2→73.4) is only 3.2 points and may be within run-to-run noise.
  2. [§1.1, Remark 2, §3, Figures 1–3, Appendix A] The emergence claim is supported only by qualitatively selected traces. The paper defines emergent as 'the model does not exhibit that behavior in regular CoT without any special prompting' (Remark 2), but no systematic measurement of backtracking, verification, or alternative-method frequencies in the base/SFT model versus the ER-RLSP model is provided. Since Remark 2 and Section 6 concede these behaviors are 'indirectly present in the pretraining data,' the claim that RLSP induces them requires a quantitative behavioral protocol (e.g., code or template matching for search patterns) rather than illustrative examples, one of which the paper itself reports as a failure case.
  3. [§2.2, Eq. (1); §3] The length reward R_ex = -C/|o| directly incentivizes longer outputs, and longer outputs are then interpreted as 'search behavior.' The paper acknowledges in §2.2 that models can hack this signal by repeating responses. The assumption that output length is a valid proxy for reasoning effort is load-bearing but not validated; the observed behaviors could reflect reward hacking or amplification of pretraining patterns. Please provide evidence that the longer traces contain diverse intermediate steps rather than repetition, or measure search behaviors independently of length.
minor comments (5)
  1. [Abstract, §4.1] The abstract states RLSP 'can boost performance by 23%' on MATH-500 for Llama-3.1-8B, but 22.8 points of the 50.6→73.4 gain come from SFT; the exploration-reward RL contributes 3.2 points. Clarify this decomposition to avoid overstating the novel component.
  2. [Figure 4 caption] The caption claims the increase in response length is 'necessary for search behavior and better reasoning, even in a strict theoretical sense [MS23].' The cited results show that longer chain-of-thought can increase expressivity, not that length is necessary for reasoning or search; please rephrase to match what the citations actually establish.
  3. [Appendix B.4] There are typos in the hyperparameters paragraph: 'value clippoing' should be 'value clipping'.
  4. [References] The bibliography entry [MS23] contains the misspelling 'expresssive power'; also consider providing volume/page details for the published version, if available.
  5. [§3, Figures 6–8] The response-length plots are informative, but the section claims 'search behavior' based solely on length; adding example outputs or behavior counts for the pure-RL arms would make the comparison more convincing.

Circularity Check

1 steps flagged · score 3.0 of 10

Partial circularity: 'search behavior' is measured by the exact response length that the exploration reward optimizes; benchmark gains remain externally grounded.

  1. self definitional [Section 3 'Can Pure RL Lead to Thinking Behavior?' (Figures 6–8), with the reward defined in Section 2.2 and Eq. (2)]
    "In these experiments we use response length as an objective metric to measure the search behavior. ... On the other hand, as expected, reward with the response length leads to an increase on the search behavior."

    Section 2.2 defines the exploration reward as R_ex(q,o) = -C/|o|, so longer responses receive higher exploration reward by construction. Section 3 then defines 'search behavior' as response length and concludes that the response-length reward increases search behavior. The quantitative claim in this ablation is therefore a restatement of the reward objective: the metric being measured is the same quantity the reward directly optimizes. The paper's genuinely nontrivial observations are the qualitative backtracking/verification traces, which are separate evidence, but the section's stated metric-based conclusion is tautological. This is a localized circularity in the motivational ablation, not in the central external-benchmark comparisons.

full rationale

The central performance claims are checked against external benchmarks (MATH-500 and AIME 2024), so those numbers are not circular: RLSP-trained models are evaluated on held-out test sets, and no fitted parameter is renamed as a prediction. The theory section cites external CoT-complexity results [LLZM24, MS23], not self-citations, and no uniqueness theorem is imported from the authors' prior work. The one genuine circular step is in Section 3, where 'search behavior' is operationally defined as response length while the exploration reward directly rewards response length; the conclusion that the length reward increases search behavior is then true by definition. This does not invalidate the benchmark results, but it means the ablation's quantitative evidence for 'search behavior' is weaker than it appears. The absence of outcome-only PPO control arms in Tables 1–2 is a real attribution gap, but it is a missing control rather than a circular reduction; the reported gains could come from the verifier component, yet they remain externally measured. Overall, the paper is not fundamentally circular; the definitional shortcut is confined to one auxiliary metric.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper contributes no new physical or formal entities. It relies on two hand-chosen numeric parameters (alpha and C) and on several assumptions about the relationship between length, reasoning effort, and emergent behavior. The theoretical grounding is borrowed from cited CoT expressivity results, which are treated as background axioms.

free parameters (2)
  • alpha (reward mixing weight) = 0.8
    Equation (1) sets R = alpha * correctness + (1-alpha) * exploration reward. alpha = 0.8 was chosen for all experiments without tuning, and the paper notes it should be decayed over time, so the results depend on this choice.
  • C (length reward constant) = 1000
    Appendix B.4 defines R_ex = -C/|o| with C = 1000. This scales the length penalty and directly affects the balance between exploration and correctness.
assumptions (4)
  • standard math Chain-of-thought provably increases the computational power of transformers, with power growing with the number of steps (Merrill and Sabharwal 2023; Li et al. 2024).
    Section 1.2 bases the exploration reward design on this theoretical result. The paper assumes the theorem is correct and applicable to neural networks trained in practice.
  • ad hoc to paper Response length is a valid proxy for reasoning effort: longer responses that output more intermediate steps correspond to harder reasoning.
    Section 2.2 defines the exploration reward based on length, citing the CoT expressivity results. This is an assumption about the correlation between token count and genuine reasoning effort, acknowledged as an intuition.
  • domain assumption The outcome verifier provides an unambiguous binary correctness signal for math problems (exact answer match).
    Section 2.2 assumes the verifier is objective. In math this is plausible, but edge cases such as equivalent answer forms are not discussed.
  • domain assumption Behaviors labeled 'emergent' are rare enough in standard CoT to be considered new, and the selected examples are representative.
    Remark 2 admits these behaviors are implicitly present in pretraining data. The claim that RLSP 'leads to' their emergence depends on the representativeness of the chosen examples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Emergence of Thinking in LLMs I: Searching for the Right Intuition." pith.science (2026). https://pith.science/paper/WKBGRUX4

@misc{pith2026250206773,
  author       = {Pith},
  title        = {Pith review of: On the Emergence of Thinking in LLMs I: Searching for the Right Intuition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WKBGRUX4}},
  note         = {Machine review of arXiv:2502.06773}
}
read the original abstract

Recent AI advancements, such as OpenAI's new models, are transforming LLMs into LRMs (Large Reasoning Models) that perform reasoning during inference, taking extra time and compute for higher-quality outputs. We aim to uncover the algorithmic framework for training LRMs. Methods like self-consistency, PRM, and AlphaZero suggest reasoning as guided search. We ask: what is the simplest, most scalable way to enable search in LLMs? We propose a post-training framework called Reinforcement Learning via Self-Play (RLSP). RLSP involves three steps: (1) supervised fine-tuning with human or synthetic demonstrations of the reasoning process, (2) using an exploration reward signal to encourage diverse and efficient reasoning behaviors, and (3) RL training with an outcome verifier to ensure correctness while preventing reward hacking. Our key innovation is to decouple exploration and correctness signals during PPO training, carefully balancing them to improve performance and efficiency. Empirical studies in the math domain show that RLSP improves reasoning. On the Llama-3.1-8B-Instruct model, RLSP can boost performance by 23% in MATH-500 test set; On AIME 2024 math problems, Qwen2.5-32B-Instruct improved by 10% due to RLSP. However, a more important finding of this work is that the models trained using RLSP, even with the simplest exploration reward that encourages the model to take more intermediate steps, showed several emergent behaviors such as backtracking, exploration of ideas, and verification. These findings demonstrate that RLSP framework might be enough to enable emergence of complex reasoning abilities in LLMs when scaled. Lastly, we propose a theory as to why RLSP search strategy is more suitable for LLMs inspired by a remarkable result that says CoT provably increases computational power of LLMs, which grows as the number of steps in CoT \cite{li2024chain,merrill2023expresssive}.

Figures

Figures reproduced from arXiv: 2502.06773 by the authors.

Figure 1
Figure 1. The emergence of backtracking, self-correction in Llama-3.1-8B. During PPO training we incentivize [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The emergence of backtracking and self-correction in Qwen-2.5-32B-Instruct; full CoT reasoning [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. A side-by-side comparison of regular CoT and CoT after RLSP training. Both experiments [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Reward, response length and AIME24 accuracy during RL training with the PPO algorithm using [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: A possible training paradigm for the emergence of complex reasoning process. In this work, we [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Response length during RL training of Llama-3.1-8B-Instruct on MATH dataset. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Response length during RL training of Qwen2.5-7B-Instruct model with only outcome reward in [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Response length during RL training of Qwen2.5-7B-Instruct model in coding domain. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reasoning-Finetuning Repurposes Latent Representations in Base Models

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A steering vector computed from base model activations induces backtracking in DeepSeek-R1-Distill-Llama-8B, indicating reasoning fine-tuning repurposes pre-existing representations.

  2. LLM-First Search: Self-Guided Exploration of the Solution Space

    cs.AI 2025-06 conditional novelty 6.0 of 10

    LLM-First Search, where the model itself decides whether to continue or backtrack during reasoning, outperforms MCTS, BestFS, and ToT-BFS on harder Countdown and Sudoku tasks while using fewer tokens.

Reference graph

Works this paper leans on

87 extracted references · 22 canonical work pages · cited by 2 Pith papers

  1. [1]

    Phi-4 technical report

    Marah Abdin, Jyoti Aneja, Harkirat Behl, S \'e bastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905 , 2024

    arXiv 2024

  2. [2]

    Amc 2023

    AI-MO. Amc 2023. https://huggingface.co/datasets/AI-MO/aimo-validation\\ -amc, 2023

    2023

  3. [3]

    Aime 2024

    AI-MO. Aime 2024. https://huggingface.co/datasets/AI-MO/aimo-validation\\ -aime, 2024

    2024

  4. [4]

    Numinamath-cot

    AI-MO. Numinamath-cot. https://huggingface.co/datasets/AI-MO/NuminaMath-\ , 2024

    2024

  5. [5]

    Large language monkeys: Scaling inference compute with repeated sampling

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R \'e , and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787 , 2024

    arXiv 2024

  6. [6]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 , 2022

    arXiv 2022

  7. [7]

    Scaling test-time compute with open models, 2024

    Edward Beeching, Lewis Tunstall, and Sasha Rush. Scaling test-time compute with open models, 2024

    2024

  8. [8]

    Open-r1: a fully open reproduction of deepseek-r1

    Elie Bakouch, Leandro von Werra, and Lewis Tunstall. Open-r1: a fully open reproduction of deepseek-r1. https://huggingface.co/blog/open-r1, 2025

    2025

Show all 87 references
  1. [9]

    Longwriter: Unleashing 10,000+ word generation from long context llms

    Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longwriter: Unleashing 10,000+ word generation from long context llms. arXiv preprint arXiv:2408.07055 , 2024

    2024 arXiv

  2. [10]

    Self-improving robust preference optimization

    Eugene Choi, Arash Ahmadian, Matthieu Geist, Oilvier Pietquin, and Mohammad Gheshlaghi Azar. Self-improving robust preference optimization. arXiv preprint arXiv:2406.01660 , 2024

    2024 arXiv

  3. [11]

    Alphamath almost zero: process supervision without process

    Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: process supervision without process. arXiv preprint arXiv:2405.03553 , 2024

    2024 arXiv

  4. [12]

    Codeforces dataset

    CodeForces. Codeforces dataset. https://codeforces.com/blog/entry/136853, 2024

    2024

  5. [13]

    Process reinforcement through implicit rewards

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinfor...

    2024

  6. [14]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. https://github.com/deepseek-ai/DeepSeek-R1/blob/main/DeepSeek_R1.pdf, 2025

    2025

  7. [15]

    Flash A ttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flash A ttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR) , 2024

    2024

  8. [16]

    Ai achieves silver-medal standard solving international mathematical olympiad problems, 2024

    Google Deepmind. Ai achieves silver-medal standard solving international mathematical olympiad problems, 2024

    2024

  9. [17]

    Alphazero-like tree-search can guide large language model decoding and training

    Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309.17179 , 2023

    2023 arXiv

  10. [18]

    Introducing gemini 2.0: our new ai model for the agentic era

    Google. Introducing gemini 2.0: our new ai model for the agentic era. https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/#ceo-message, 2024

    2024

  11. [19]

    rstar-math: Small llms can master math reasoning with self-evolved deep thinking

    Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519 , 2025

    2025 arXiv

  12. [20]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020

    2009 arXiv

  13. [21]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 , 2021

    2021 arXiv

  14. [22]

    Large language models cannot self-correct reasoning yet

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798 , 2023

    2023 arXiv

  15. [23]

    Teaching large language models to reason with reinforcement learning

    Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.04642 , 2024

    2024 arXiv

  16. [24]

    O1 replication journey -- part 3: Inference-time scaling for medical reasoning

    Zhongzhen Huang, Gui Geng, Shengyi Hua, Zhen Huang, Haoyang Zou, Shaoting Zhang, Pengfei Liu, and Xiaofan Zhang. O1 replication journey -- part 3: Inference-time scaling for medical reasoning. arXiv preprint arXiv:2501.06458 , 2025

    2025 arXiv

  17. [25]

    Reasoning with language model is planning with world model

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992 , 2023

    2023 arXiv

  18. [26]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 , 2024

    2024 arXiv

  19. [27]

    Advancing language model reasoning through reinforcement learning and inference scaling

    Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao, Juanzi Li, Jie Tang, and Yuxiao Dong. Advancing language model reasoning through reinforcement learning and inference scaling. arXiv preprint arXiv:2501.11651 , 2025

    2025 arXiv

  20. [28]

    Openrlhf: An easy-to-use, scalable and high-performance rlhf framework

    Jian Hu, Xibin Wu, Weixun Wang, Dehao Zhang, Yu Cao, et al. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143 , 2024

    2024 arXiv

  21. [29]

    O1 replication journey--part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson? arXiv preprint arXiv:2411.16489 , 2024

    Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey--part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson? arXiv preprint arXiv:2411.16489 , 2024

    2024 arXiv

  22. [30]

    Technical report: Enhancing llm reasoning with reward-guided tree search

    Jinhao Jiang, Zhipeng Chen, Yingqian Min, Jie Chen, Xiaoxue Cheng, Jiapeng Wang, Yiru Tang, Haoxiang Sun, Jia Deng, Wayne Xin Zhao, et al. Technical report: Enhancing llm reasoning with reward-guided tree search. arXiv preprint arXiv:2411.11694 , 2024

    2024 arXiv

  23. [31]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 , 2024

    2024 arXiv

  24. [32]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720 , 2024

    2024 arXiv

  25. [33]

    Reinforcement learning with unsupervised auxiliary tasks

    Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397 , 2016

    2016 arXiv

  26. [34]

    Kimi k1.5: Scaling reinforcement learning with llms

    Kimi. Kimi k1.5: Scaling reinforcement learning with llms. https://github.com/MoonshotAI/Kimi-k1.5/blob/main/Kimi_k1.5.pdf, 2025

    2025

  27. [35]

    Mindstar: Enhancing math reasoning in pre-trained llms at inference time

    Jikun Kang, Xin Zhe Li, Xi Chen, Amirreza Kazemi, Qianyi Sun, Boxing Chen, Dong Li, Xu He, Quan He, Feng Wen, et al. Mindstar: Enhancing math reasoning in pre-trained llms at inference time. arXiv preprint arXiv:2405.16265 , 2024

    2024 arXiv

  28. [36]

    Training language models to self-correct via reinforcement learning

    Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917 , 2024

    2024 arXiv

  29. [37]

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face reposito...

    2024

  30. [38]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050 , 2023

    2023 arXiv

  31. [39]

    Chain of thought empowers transformers to solve inherently serial problems

    Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transformers to solve inherently serial problems. arXiv preprint arXiv:2402.12875 , 2024

    2024 arXiv

  32. [40]

    Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct

    Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583 , 2023

    2023 arXiv

  33. [41]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55(9):1--35, 2023

    2023

  34. [42]

    Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems

    Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv:2412.09413 , 2024

    2024 arXiv

  35. [43]

    Llama-3.1-8b

    Meta. Llama-3.1-8b. https://huggingface.co/meta-llama/Llama-3.1-8B, 2024

    2024

  36. [44]

    Ray: A distributed framework for emerging \ AI \ applications

    Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. Ray: A distributed framework for emerging \ AI \ applications. In 13th USENIX symposium on operating systems design a...

    2018

  37. [45]

    The expresssive power of transformers with chain of thought

    William Merrill and Ashish Sabharwal. The expresssive power of transformers with chain of thought. arXiv preprint arXiv:2310.07923 , 2023

    2023 arXiv

  38. [46]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393 , 2025

    2025 arXiv

  39. [47]

    Sky-t1: Train your own o1 preview model within \ 450

    NovaSky. Sky-t1: Train your own o1 preview model within \ 450. https://novasky-ai.github.io/posts/sky-t1/, 2025

    2025

  40. [48]

    OpenAI. Gsm8k. https://huggingface.co/datasets/openai/gsm8k, 2022

    2022

  41. [49]

    Learning to reason with llms

    OpenAI. Learning to reason with llms. https://openai.com/index/learning-to-\\ reason-with-llms/, 2024

    2024

  42. [50]

    Math-500

    OpenAI. Math-500. https://huggingface.co/datasets/HuggingFaceH4/MATH-500, 2024

    2024

  43. [51]

    Openai humaneval

    OpenAI. Openai humaneval. https://huggingface.co/datasets/openai/openai_humaneval, 2024

    2024

  44. [52]

    Openai o1-mini advancing cost-efficient reasoning

    OpenAI. Openai o1-mini advancing cost-efficient reasoning. https://openai.com/index/openai-o1-mini-advancing-cost-\\ efficient-reasoning/, 2024

    2024

  45. [53]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:277...

    2022

  46. [54]

    O1 replication journey: A strategic progress report--part 1

    Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, et al. O1 replication journey: A strategic progress report--part 1. arXiv preprint arXiv:2410.18982 , 2024

    2024 arXiv

  47. [55]

    Mutual reasoning makes smaller llms stronger problem-solvers

    Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller llms stronger problem-solvers. arXiv preprint arXiv:2408.06195 , 2024

    2024 arXiv

  48. [56]

    Qwen-2.5-32b

    Qwen. Qwen-2.5-32b. https://huggingface.co/Qwen/Qwen2.5-32B, 2024

    2024

  49. [57]

    Qwq-32b-preview

    Qwen. Qwq-32b-preview. https://huggingface.co/Qwen/QwQ-32B-Preview, 2024

    2024

  50. [58]

    Qwq-longcot-130k-cleaned

    Qwen. Qwq-longcot-130k-cleaned. https://huggingface.co/datasets/gghfez/QwQ-LongCoT-130K-cleaned, 2024

    2024

  51. [59]

    Qwq: Reflect deeply on the boundaries of the unknown

    Qwen. Qwq: Reflect deeply on the boundaries of the unknown. https://qwenlm.github.io/blog/qwq-32b-preview, 2024

    2024

  52. [60]

    Recursive introspection: Teaching language model agents how to self-improve

    Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. arXiv preprint arXiv:2407.18219 , 2024

    2024 arXiv

  53. [61]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022 , 2023

    2023 arXiv

  54. [62]

    A program for the machine translation of natural languages

    W Smoke and E Dubinsky. A program for the machine translation of natural languages. Mech. Transl. Comput. Linguistics , 6:2--10, 1961

    1961

  55. [63]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 , 2024

    2024 arXiv

  56. [64]

    Rewarding progress: Scaling automated process verifiers for llm reasoning

    Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for llm reasoning. arXiv preprint arXiv:2410.08146 , 2024

    arXiv

  57. [65]

    Dualformer: Controllable fast and slow thinking by learning with randomized reasoning traces

    DiJia Su, Sainbayar Sukhbaatar, Michael Rabbat, Yuandong Tian, and Qinqing Zheng. Dualformer: Controllable fast and slow thinking by learning with randomized reasoning traces. arXiv preprint arXiv:2410.09918 , 2024

    2024 arXiv

  58. [66]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017

    2017 arXiv

  59. [67]

    Toward self-improvement of llms via imagination, searching, and criticizing

    Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. Toward self-improvement of llms via imagination, searching, and criticizing. arXiv preprint arXiv:2404.12253 , 2024

    2024 arXiv

  60. [68]

    Solving olympiad geometry without human demonstrations

    Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature , 625(7995):476--482, 2024

    2024

  61. [69]

    Solving math word problems with process-and outcome-based feedback

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275 , 2022

    2022 arXiv

  62. [70]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442 , 2024

    2024 arXiv

  63. [71]

    Q*: Improving multi-step reasoning for llms with deliberative planning

    Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan, and Bo An. Q*: Improving multi-step reasoning for llms with deliberative planning. arXiv preprint arXiv:2406.14283 , 2024

    2024 arXiv

  64. [72]

    Openr: An open source framework for advanced reasoning with large language models

    Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M Ni, et al. Openr: An open source framework for advanced reasoning with large language models. arXiv preprint arXiv:2410.09671 , 2024

    arXiv

  65. [73]

    An empirical analysis of compute-optimal inference for problem-solving with language models

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. An empirical analysis of compute-optimal inference for problem-solving with language models. Under review , 2024

    2024

  66. [74]

    Towards self-improvement of llms via mcts: Leveraging stepwise knowledge with curriculum preference learning

    Xiyao Wang, Linfeng Song, Ye Tian, Dian Yu, Baolin Peng, Haitao Mi, Furong Huang, and Dong Yu. Towards self-improvement of llms via mcts: Leveraging stepwise knowledge with curriculum preference learning. arXiv preprint arXiv:2410.06508 , 2024

    arXiv

  67. [75]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 , 2022

    2022 arXiv

  68. [76]

    Llava-o1: Let vision language models reason step-by-step

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-o1: Let vision language models reason step-by-step. arXiv preprint arXiv:2411.10440 , 2024

    2024 arXiv

  69. [77]

    Towards system 2 reasoning in llms: Learning how to think with meta chain-of-though

    Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, et al. Towards system 2 reasoning in llms: Learning how to think with meta chain-of-though. arXiv preprint arXiv:2501.04682 , 2025

    2025 arXiv

  70. [78]

    Metamath: Bootstrap your own mathematical questions for large language models

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284 , 2023

    2023 arXiv

  71. [79]

    Demystifying long chain-of-thought reasoning in llms

    Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in llms. arXiv preprint arXiv:2502.03373 , 2025

    2025 arXiv

  72. [80]

    Scaling relationship on learning mathematical reasoning with large language models

    Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825 , 2023

    2023 arXiv

  73. [81]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems , 36, 2024

    2024

  74. [82]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122 , 2024

    2024 arXiv

  75. [83]

    7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient

    Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. https://hkust-nlp.notion.site/simplerl-reason#18439bdc1c6b8083ba31f9cc912cf7f0, 2025

    2025

  76. [84]

    Small language models need strong verifiers to self-correct reasoning

    Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. Small language models need strong verifiers to self-correct reasoning. arXiv preprint arXiv:2404.17140 , 2024

    2024 arXiv

  77. [85]

    Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b

    Di Zhang, Jiatong Li, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b. arXiv preprint arXiv:2406.07394 , 2024

    2024 arXiv

  78. [86]

    Llama-berry: Pairwise optimization for o1-like olympiad-level mathematical reasoning

    Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, et al. Llama-berry: Pairwise optimization for o1-like olympiad-level mathematical reasoning. arXiv preprint arXiv:2410.02884 , 2024

    arXiv

  79. [87]

    Rest-mcts*: Llm self-training via process reward guided tree search

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search. arXiv preprint arXiv:2406.03816 , 2024

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.