Pith. sign in

REVIEW 4 major objections 6 minor 6 cited by

BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Inserting short expert prefixes during reinforcement learning lets small language models solve reasoning problems that supervised fine-tuning plus RL cannot.

desk verdict A practical GRPO variant that gives struggling small models adaptive expert hints; the empirical gains look real, but the theory covers a different algorithm and the key transfer claim rests on thin, unreplicated evidence. read the letter →

arxiv 2506.17211 v1 pith:IYEQMZRZ submitted 2025-06-20 cs.LG

Xuechen Zhang , Zijian Huang , Yingcong Li , Chenshun Ni , Jiasi Chen , Samet Oymak This is my paper · ORCID
classification cs.LG
keywords smalllanguagemodelsreasoningreinforcementlearningGRPOexperttracesrewarddensificationcurriculumbranchedrollouts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the standard recipe for teaching small language models to reason—supervised fine-tuning on traces from a stronger model, then reinforcement learning—can fail outright when the expert traces are too hard to imitate or when the initial model almost never produces a correct rollout. It proposes BREAD, a modification of Group Relative Policy Optimization (GRPO) in which a short prefix of the expert trace is appended to the question whenever the model's own rollouts fail; the model then completes the reasoning, so every update includes at least one successful trace. This turns one long all-or-nothing task into a chain of easier completions, densifying the reward and creating a self-paced curriculum along the trace. The paper reports that BREAD needs fewer than 40 percent of the expert traces, beats standard GRPO (including by more than 15 percent accuracy on NuminaMath-CoT), solves hard questions that SFT and GRPO leave unsolved, and reaches baseline accuracy in about a quarter of the training steps.

What carries the argument

The mechanism is the expert anchor: a short prefix of the expert solution, appended to the question as a hint, chosen automatically by binary search (the Episode Anchor Search) so that the small model completes the rest of the trajectory with mixed success. The branched rollout is the complementary half of the machinery: instead of asking the model to generate the entire answer, the optimizer samples continuations from the anchored prefix, so the reward signal is evaluated on a short, feasible completion. Together they densify reward, guarantee at least one correct trace per update, and induce a curriculum in which the hint shortens as the model improves.

What would settle it

Take a dataset and model pair where even the full expert solution appended to the prompt does not increase the rollouts' correctness rate (the extreme case the paper concedes in its limitations). If BREAD shows no accuracy gain over GRPO there, the load-bearing premise is falsified. More cheaply, replicate the paper's correctness-ratio-versus-hint-ratio experiment and look for a flat curve across hint ratios 0 to 0.5, which would mean hints are not actually guiding the model.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that the two-stage SFT+RL paradigm has a structural blind spot: distillation-style SFT compresses the expert's reasoning into the small model, and if that compression fails—because the expert jumps several reasoning steps per token—RL inherits a policy that cannot generate a correct trajectory to get reward. BREAD removes the need for the small model to express the whole expert trace at once. When a batch of rollouts contains no correct answer, Episode Anchor Search runs a binary search over prefixes of the expert solution to find the longest hint the student can use; rollouts branch from that point, and the group-relative policy update runs on these branched trajectories. The paper argues that success probability per subproblem becomes $\epsilon^\tau$ instead of $\epsilon^T$, so the model can learn the remaining steps one component at a time, and it proves this advantage in a symmetric-random-walk model: SFT+RL needs trace length $\Omega(K^2)$ to have any chance, while the BREAD variant succeeds with trace length $O(K^2/T)$ using $O(T \log T)$ rollouts. The empirical claim is that this transfers to real math reasoning, where BREAD outperforms every baseline and solves problems the SFT+RL strategy cannot.

Load-bearing premise

The argument depends on the small model being able to extract and follow enough of a partial expert trace that appending it raises the chance of a correct rollout—and that this ability transfers to questions with no hint at test time; if the premise fails, BREAD degrades to GRPO with extra computation.

Editorial extensions

If this is right

  • On reasoning datasets where the base model can sometimes finish a trace, BREAD should beat GRPO with a fraction of the expert data, making distillation pipelines cheaper and removing the need for hand-filtered SFT sets.
  • The hard-question experiments imply BREAD can learn from questions where pass@3 is zero—standard SFT and GRPO show flat accuracy on such sets—so its value is concentrated exactly on the samples other pipelines discard.
  • Because BREAD's objective needs fewer rollouts (the paper shows parity with 5 instead of 8), its compute advantage compounds on longer reasoning traces.
  • The reward-densification argument predicts that BREAD's advantage grows with the number of compositional steps in the problem, since each branched completion avoids the exponential blowup of full-trajectory success.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The anchor-search idea is a general scaffolding principle: using a teacher at intermediate landmarks rather than imitating complete traces or full trajectories could apply to code generation, tool use, and multi-step planning, not just arithmetic reasoning.
  • One tension the paper leaves open is the test-time dependence on hints; the reported improvement at hint ratio 0 shows some transfer, so a natural extension is to anneal the anchor length during training and measure hint-free accuracy at each stage.
  • The random-walk model predicts that BREAD's advantage over SFT+RL should increase with problem depth; this is testable by binning benchmark questions by solution length and checking whether BREAD's relative gain rises with bin length.
  • The episode-splitting and binary-search heuristics could be replaced by a learned branching-point predictor or by token-level guidance, which would remove the sentence/paragraph splitting assumption and likely improve efficiency on long traces.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes BREAD, a GRPO variant for small language model reasoning that, when self-generated rollouts fail, appends a short prefix of an expert trace to the prompt and samples rollouts from the resulting branched context. The paper motivates the method with a Markov-chain model in which SFT cannot learn from expert traces that exceed the student's jump capacity and GRPO receives no reward, while a BREAD-like procedure can succeed; Lemma 1 and Theorem 1 formalize this for a symmetric random walk. Empirically, on MATH and NuminaMath-CoT with Qwen2.5-1.5B/3B, BREAD is reported to outperform GRPO, SFT+GRPO, GRPO with the expert trace as an extra rollout, and trace-budget-matched baselines, and to improve accuracy on a hard pass@3=0 subset where baselines fail.

Significance. If the empirical results hold up, BREAD is a practical and simple technique that could make RL-based post-training of small models feasible in settings where standard SFT+RL stalls, and the paper provides a useful comparison of trace-budget baselines. The theoretical analysis, however, concerns a memorization/replay surrogate rather than the policy-gradient objective, so the current manuscript does not provide a formal explanation of the actual algorithm; the empirical transfer claim also needs stronger evidence. Strengths: the paper reports reproducible experimental configurations, compares against trace-budget-matched baselines, and conducts a clean hard-subset experiment, although that experiment lacks seed variation.

major comments (4)
  1. [Section 2.2, Theorem 1] Theorem 1 is stated as "Success of BREAD" but is proved for a distinct memorization variant that records and replays stitched suffixes; the actual BREAD objective in Eq. (1) does not include any replay or memory mechanism, and the toy experiments in Figure 3 use Algorithm 4, not the policy-gradient BREAD. As a result, the central theoretical claim that BREAD succeeds where SFT+RL fails is not established for the algorithm evaluated in Section 4. Please either prove the guarantee for the actual objective or explicitly state and defend the surrogate relationship.
  2. [Section 2.2, Lemma 1] The statement "SFT and GRPO training have no impact on the student model with probability at least 2e^{-K^2/4L}" inverts the probability bound derived in the proof: the union bound gives that no sampled trace receives a reward with probability at least 1 - 2e^{-K^2/4L}, so the lemma should say "with probability at least 1 - 2e^{-K^2/4L}". As written, the lemma asserts the opposite of what is proved.
  3. [Section 3.2, Figure 6b] The claim that training with partial hints improves accuracy at hint ratio 0 (generalization from partial traces to full-length inference) is confounded: the orange RL model was trained on a mix containing 40% hinted prompts and 60% unhinted prompts, so the improvement at hint ratio 0 may be caused by direct RL on the unhinted prompts rather than by transfer from hinted training. A control with standard GRPO trained on the same unhinted subset (or an intervention that varies the hint fraction while holding the unhinted data fixed) is needed to support the transfer claim.
  4. [Section 4.1, Figure 8a] The hard-subset experiment, which is the primary evidence for the abstract's claim that BREAD solves problems unsolvable by SFT+RL, reports a single run on a roughly 100-question test set with no error bars, no multiple seeds, and no significance test; given that baselines earn near-zero reward and BREAD's final accuracy is about 14%, the gap could be within run-to-run variation. Please provide multiple seeds with confidence intervals, and consider a larger test set or a bootstrap analysis.
minor comments (6)
  1. [Section 2.2, Theorem 1] The symbol t is used both for the number of rollouts per round and for the time index of the random walk; this overloaded notation makes the theorem harder to read.
  2. [Section 5 and Introduction] Section 5 contains "dicusses" instead of "discusses", and the Introduction's "Section 1 explains our algorithm BREAD" appears to refer to Section 2.
  3. [Algorithm 1] The line "Update the policy model by maxmizing the BREAD objective" contains a typo: "maxmizing" should be "maximizing".
  4. [Section A.1] The sentence "For estimation, we define the average length of a single question in one inference time as Dsample" is repeated verbatim in the same paragraph.
  5. [Section 3.1] The citation "following [35]" for the S1K traces appears to be a mis-citation; the S1K dataset is introduced in [24].
  6. [Section B.3] The phrase "which may slow down the training sif there ised if there is no episode aggregation" is garbled and should be corrected; also "steps" here refers to solution steps and should be made explicit.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: central claims rest on external benchmarks and independent random-walk bounds.

full rationale

BREAD's central claims are not forced by construction. The algorithm is an RL objective (Eq. 1) plus an episode-anchor search that adaptively branches from expert traces; the main empirical comparisons (Figs. 7-8, Tables 1-2) are run against external baselines (GRPO, SFT+GRPO, GRPO w/ Expert Trace) on public MATH and NuminaMath-CoT benchmarks, so the reported gains are externally falsifiable rather than entailed by the method's definition. The theoretical section is a Markov-chain toy model: Lemma 1 proves SFT+RL cannot help because the expert's jumps exceed the student's allowed jump size by construction, and the RL phase gets no nonzero reward by a standard random-walk hitting-time bound; Theorem 1's success guarantee is derived from the same independent hitting-time bound (Lemma 2, citing Durrett/Feller), not from the conclusion being assumed. No fitted parameter is renamed as a prediction: the EAS hint selection is an algorithmic choice, and the trace-budget matched SFT(selected)/SFT(random) baselines are explicit attempts to control for the expert-trace budget. The paper's self-citations ([43], [44]) appear only in the related-work survey and do not carry the load-bearing argument. The empirical transfer claim from hinted training to no-hint inference (Fig. 6b) has a confound - the RL run used 40% hinted prompts and 60% unhinted prompts - but that is a validity or bias concern, not a derivation-by-construction circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central results rest on a Markov-chain model of student and expert behavior, standard random-walk hitting-time bounds, and the availability of expert traces that can be split into episodes. No free parameters are fitted to make the theory match the experiments; parameters such as the 10-episode aggregation and the EAS success range are design choices for the algorithm.

free parameters (3)
  • Episode count K=10 for EAS splitting = 10
    All expert traces are aggregated into 10 episodes before binary search, a hand-chosen granularity that determines hint length and EAS cost (Section A.2).
  • EAS success probability range = not specified numerically
    Episode Anchor Search stops when 0 < p_correct < 1; the range is a design choice that controls how often expert hints are used, and it determines the reported 36.7% and 19.1% trace usage fractions.
  • Number of rollouts G = 8
    The rollout count is set to 8 for all main experiments; it affects the solve-none ratio and the FLOPs estimate, and the paper shows BREAD is less sensitive to reducing it than GRPO.
assumptions (4)
  • domain assumption Small language models can be modeled as Markov chains with local state transitions, and expert generation as a Markov chain with larger jumps (Section 2.1).
    The theoretical results rely on this model to conclude SFT+RL fails and BREAD succeeds.
  • domain assumption The student model is initialized as a symmetric random walk (d=1, epsilon=0.5) and cannot learn transitions with jump size greater than 1 during SFT.
    Lemma 1 assumes this to show SFT has no impact; it is a toy model condition, not verified on real LLMs.
  • standard math Random walk hitting time bounds from Durrett and Feller are used as standard background.
    The proofs of Lemma 1 and Theorem 1 invoke classical concentration and hitting time results without proving them.
  • domain assumption Expert traces are available for every training question and can be split into meaningful episodes (sentences or paragraphs).
    BREAD requires a high-quality expert trace and a splitting rule; the paper assumes this in Algorithm 1 and Section A.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning." pith.science (2026). https://pith.science/paper/IYEQMZRZ

@misc{pith2026250617211,
  author       = {Pith},
  title        = {Pith review of: BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IYEQMZRZ}},
  note         = {Machine review of arXiv:2506.17211}
}
read the original abstract

Small language models (SLMs) struggle to learn complex reasoning behaviors, especially when high-quality traces are scarce or difficult to learn from. The standard training approach combines a supervised fine-tuning (SFT) stage, often to distill capabilities of a larger model, followed by a reinforcement learning (RL)stage such as Group Relative Policy Optimization (GRPO). In this paper, we investigate the fundamental limitations of this SFT + RL paradigm and propose methods to overcome them. Under a suitable theoretical model, we demonstrate that the SFT + RL strategy can fail completely when (1) the expert's traces are too difficult for the small model to express, or (2) the small model's initialization has exponentially small likelihood of success. To address these, we introduce BREAD: a GRPO variant that unifies the SFT and RL stages via partial expert guidance and branched rollouts. When self-generated traces fail, BREAD adaptively inserts short expert prefixes/hints, allowing the small model to complete the rest of the reasoning path, and ensuring that each update includes at least one successful trace. This mechanism both densifies the reward signal and induces a natural learning curriculum. BREAD requires fewer than 40% of ground-truth traces, consistently outperforming standard GRPO while speeding up the training by about 3 times. Importantly, we demonstrate that BREAD helps the model solve problems that are otherwise unsolvable by the SFT + RL strategy, highlighting how branched rollouts and expert guidance can substantially boost SLM reasoning.

Figures

Figures reproduced from arXiv: 2506.17211 by the authors.

Figure 1
Figure 1. High-level overview of approaches. In existing training methods like supervised fine-tuning (left), high-quality reasoning traces produced by LLMs are often too complex for SLMs to imitate, so they deliver little benefit and can even hurt SLM reasoning capability. Since the subsequent RL phase starts from this weak starting point, the two-stage SFT+RL procedure often fails. In reinforcement learning (center), when t… view at source ↗
Figure 2
Figure 2. Workflow of BREAD. (1) Regular rollout: Given a question Q, sample a group of rollouts. If the sampled rollouts contain correct reasoning trace, use this group of rollouts to do the policy updates. Otherwise, go to step (2) Episode anchor search: Starting with the whole expert trace (provided by the ground truth or from the correct responses generated by LLMs) as the search space for potential suitable hints, constr… view at source ↗
Figure 3
Figure 3. We compare SFT, GRPO, and BREAD according to the Navigation Task described in Section 2.1. K is number of states in the Markov chain whereas ϵ is the probability of transition to non-favorable states. Our toy model reveals settings where BREAD can succeed while SFT or GRPO completely fail. solve, each with successful completion probability of ϵ for the base student model. Then, we receive a final trajectory-level co… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Depiction of the trajectory stitching argument for T “ 3. Short subtraces (blue, red, green) are those generated at each round. The dashed yellow line are the final trace that maps x0 to xT . Our next result proves how BREAD can overcome this bottleneck and address thi…
Figure 5
Figure 5. Figure 5: Issues with vanilla GRPO on SLMs. Among 256 samples in a batch, for nearly half of them, the base model’s 8 rollouts produce no correct trace. The absence of reward inhibits further performance gains and validation accuracy stalls. 3 Empirical Insights into BREAD 3.1 L…
Figure 6
Figure 6. Figure 6: (a) Math benchmarks may show difficulty tiers, but ranking individual problems for curriculum training is still challenging. Instead, BREAD can adaptively adjust the difficulty by automatically choosing the branching point. For easier questions, it would provide no or …
Figure 7
Figure 7. Figure 7: Test accuracy over training steps (left) / FLOPs (right). BREAD, which adaptively uses hints from expert traces during GRPO, significantly improves SLM reasoning ability compared to all baselines. The gray dashed line (max accuracy of the best baseline) demonstrates th…
Figure 8
Figure 8. Figure 8: b, BREAD sharply lowers the solve-none ratio, getting more informative samples and richer feedback. This enables BREAD to learn effectively even from very hard questions and complex reasoning traces. BREAD improves sample efficiency during training. SLMs distilled via …
Figure 9
Figure 9. Figure 9: Test accuracy of SFT over training steps. The accuracy doesn’t continuously increase after [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Test accuracy over training steps with different number of rollouts. The gray dashed line [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Distribution plot of MATH and NuminaMath-CoT solution step numbers [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information

    cs.LG 2026-07 conditional novelty 6.0 of 10

    OC-GRPO reweights GRPO gradients with an importance ratio so that hints used during rollout generation still optimize the original unguided objective, delivering a 13.8% relative Pass@1 gain over vanilla GRPO.

  2. It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches

    cs.AI 2026-05 conditional novelty 6.0 of 10

    W2SPO trains a reasoning LLM by inserting short 8-token branches proposed by a weaker model into its own rollouts and updating only those branch tokens, improving Pass@1 over vanilla GRPO (64.2% vs 62.3%) with a 3.55x...

  3. Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment

    cs.CL 2026-01 conditional novelty 6.0 of 10

    Rank-Surprisal Ratio (RSR) correlates strongly (average Spearman 0.86) with post-distillation reasoning gains across five student models and trajectories from eleven teachers, outperforming existing selection metrics.

  4. Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts

    cs.LG 2025-10 conditional novelty 6.0 of 10

    HINT boosts LLM reasoning RL by injecting teacher-generated heuristic hints only on all-failed rollouts, keeping hints out of the policy-optimization prompt, and monitoring guidance quality with a new Affinity metric.

  5. Reasoning to Regulate: Chain-of-Thought for Traffic Rule Understanding

    cs.CV 2026-07 conditional novelty 5.0 of 10

    CoT data curated by two-round LLM prompting and VLM verification, then SFT+GRPO with fine-grained rewards, improves MapDR rule–lane association F1 from 0.642 to 0.723.

  6. AAPA: Adversarially Anchored Preference Alignment for Post-Training of Large Language Models

    cs.AI 2025-09 conditional novelty 4.0 of 10

    Adding a frozen POLAR-based similarity penalty to SFT, GRPO, or CHORD improves averaged instruction-following scores by up to 5.77 percent at 0.6B scale in the paper's reported runs.

Reference graph

Works this paper leans on

47 extracted references · 9 canonical work pages · cited by 6 Pith papers

  1. [1]

    Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    arXiv 2024

  2. [2]

    L1: Controlling how long a reasoning model thinks with reinforcement learning.arXiv preprint arXiv:2503.04697, 2025

    Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning.arXiv preprint arXiv:2503.04697, 2025

    arXiv 2025

  3. [3]

    Hindsight experience replay.Advances in neural information processing systems, 30, 2017

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay.Advances in neural information processing systems, 30, 2017

    2017

  4. [4]

    Thinking fast and slow with deep learning and tree search.Advances in neural information processing systems, 30, 2017

    Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search.Advances in neural information processing systems, 30, 2017

    work page 2017

  5. [5]

    Frugalgpt: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176, 2023

    Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176, 2023

    arXiv 2023

  6. [6]

    Sft memorizes, rl generalizes: A comparative study of foundation model post-training.arXiv preprint arXiv:2501.17161, 2025

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training.arXiv preprint arXiv:2501.17161, 2025. 11

    arXiv 2025

  7. [7]

    Cambridge university press, 2019

    Rick Durrett.Probability: theory and examples, volume 49. Cambridge university press, 2019

    2019

  8. [8]

    First return, then explore.Nature, 590(7847):580–586, 2021

    Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore.Nature, 590(7847):580–586, 2021

    2021

Show all 47 references
  1. [9]

    Open r1: A fully open reproduction of deepseek-r1, January 2025

    Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025

    2025

  2. [10]

    John Wiley & Sons, 1991

    William Feller.An introduction to probability theory and its applications, Volume 2, volume 2. John Wiley & Sons, 1991

    1991

  3. [11]

    Gemini 2.0 flash thinking mode (gemini-2.0f lash-thinking-exp-1219), 2024

    Google. Gemini 2.0 flash thinking mode (gemini-2.0f lash-thinking-exp-1219), 2024. https: //cloud.google.com/vertex-ai/generative-ai/docs/thinking

    2024

  4. [12]

    Last updated 14 May 2025

    Google. Gemini 2.0 flash thinking mode ( gemini-2.0-flash-thinking-exp-1219), De- cember 2024. Accessed 15 May 2025. “Last updated 14 May 2025” on the page

    2024

  5. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  6. [14]

    Language model cascades: Token-level uncertainty and beyond

    Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty and beyond. In The Twelfth International Conference on Learning Representations, 2024

    2024

  7. [15]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  8. [16]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  9. [17]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  10. [18]

    When to trust your model: Model-based policy optimization.Advances in neural information processing systems, 32, 2019

    Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization.Advances in neural information processing systems, 32, 2019

    2019

  11. [19]

    Gemini 2.5: Our most intelligent ai model

    Koray Kavukcuoglu. Gemini 2.5: Our most intelligent ai model. https://blog.google/ technology/google-deepmind/gemini-model-thinking-updates-march-2025/ , March 2025. Accessed 15 May 2025

    2025

  12. [20]

    Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    Diederik P Kingma. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  13. [21]

    Numinamath

    Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https://huggingface.co/AI-MO/NuminaMath...

    2024

  14. [22]

    Small models struggle to learn from strong reasoners

    Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ra- masubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners. arXiv preprint arXiv:2502.12143, 2025

    2025

  15. [23]

    Cppo: Accelerating the training of group relative policy optimization-based reasoning models.arXiv preprint arXiv:2503.22342, 2025

    Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. Cppo: Accelerating the training of group relative policy optimization-based reasoning models.arXiv preprint arXiv:2503.22342, 2025

    2025

  16. [24]

    s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025. 12

    2025 arXiv

  17. [25]

    Policy invariance under reward transforma- tions: Theory and application to reward shaping

    Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transforma- tions: Theory and application to reward shaping. InIcml, volume 99, pages 278–287. Citeseer, 1999

    1999

  18. [26]

    Optimizing test-time compute via meta reinforcement fine-tuning.arXiv preprint arXiv:2503.07572, 2025

    Yuxiao Qu, Matthew YR Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine-tuning.arXiv preprint arXiv:2503.07572, 2025

    2025 arXiv

  19. [27]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  20. [28]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024

    2024

  21. [29]

    A reduction of imitation learning and structured prediction to no-regret online learning

    Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. InProceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627–635. JMLR Workshop and Co...

    2011

  22. [30]

    Beyond chinchilla-optimal: Accounting for inference in language model scaling laws.arXiv preprint arXiv:2401.00448, 2023

    Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws.arXiv preprint arXiv:2401.00448, 2023

    2023 arXiv

  23. [31]

    Reasoning with latent thoughts: On the power of looped transformers.arXiv preprint arXiv:2502.17416, 2025

    Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J Reddi. Reasoning with latent thoughts: On the power of looped transformers.arXiv preprint arXiv:2502.17416, 2025

    2025 arXiv

  24. [32]

    Kickstarting deep reinforcement learning.arXiv preprint arXiv:1803.03835, 2018

    Simon Schmitt, Jonathan J Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Woj- ciech M Czarnecki, Joel Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, et al. Kickstarting deep reinforcement learning.arXiv preprint arXiv:1803.03835, 2018

    2018 arXiv

  25. [33]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  26. [34]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    2024 arXiv

  27. [35]

    Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  28. [36]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  29. [37]

    Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.arXiv preprint arXiv:1707.08817, 2017

    Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.arXiv preprint arXiv:1707...

    2017 arXiv

  30. [38]

    Dump: Automated distribution- level curriculum learning for rl-based llm post-training.arXiv preprint arXiv:2504.09710, 2025

    Zhenting Wang, Guofeng Cui, Kun Wan, and Wentian Zhao. Dump: Automated distribution- level curriculum learning for rl-based llm post-training.arXiv preprint arXiv:2504.09710, 2025

    2025

  31. [39]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  32. [40]

    Not all rollouts are useful: Down- sampling rollouts in llm reinforcement learning.arXiv preprint arXiv:2504.13818, 2025

    Yixuan Even Xu, Yash Savani, Fei Fang, and Zico Kolter. Not all rollouts are useful: Down- sampling rollouts in llm reinforcement learning.arXiv preprint arXiv:2504.13818, 2025. 13

    2025 arXiv

  33. [41]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  34. [42]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    2025 arXiv

  35. [43]

    Making small language models efficient reasoners: Intervention, supervision, reinforcement

    Xuechen Zhang, Zijian Huang, Chenchun Ni, Ziyang Xiong, Jiasi Chen, and Samet Oymak. Making small language models efficient reasoners: Intervention, supervision, reinforcement. arXiv preprint arXiv:2505.07961, 2025

    2025 arXiv

  36. [44]

    ” or“\n

    Xuechen Zhang, Zijian Huang, Ege Onur Taga, Carlee Joe-Wong, Samet Oymak, and Jiasi Chen. Efficient contextual llm cascades through budget-constrained policy learning.Conference on Neural Information Processing Systems, 2024. 14 The Supplementary Material is organized as follo...

    2024

  37. [45]

    A coefficient for the expert trace loss 1 G|oS|

  38. [46]

    A KL divergence term´βD KLpπθ||πrefq

  39. [47]

    An Advantage termmin ´ ri,tpθq ˆAS,t,clippr i,tpθq,1´ε,1`εq ˆAS,t ¯ Suppose thatri,tď1`ε , because ˆAS,tě0 , min ´ ri,tpθq ˆAS,t,clippr i,tpθq,1´ε,1`εq ˆAS,t ¯ “ ri,tpθq ˆAS,t, which is exactly the entropy loss with a coefficient ˆAS,t if we replaceπoldpoi,t|q,oiătq in 18 ri,t...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.