Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Most rollout compute in GRPO reasoning RL goes to prompts that yield no learning signal; a pre-rollout filter that exploits the temporal consistency of prompt value cuts rollout time 2.4× and total training time 2.0× at equal accuracy.

desk verdict Plausible pre-rollout filtering idea with real temporal-correlation evidence, but the headline accuracy claim rests on best-checkpoint, single-run numbers that need seeds and final-checkpoint results. read the letter →

arxiv 2506.02177 v1 pith:RND6AART submitted 2025-06-02 cs.AI cs.LG

Haizhong Zheng , Yang Zhou , Brian R. Bartoldson , Bhavya Kailkhura , Fan Lai , Jiawei Zhao , Beidi Chen This is my paper · ORCID
classification cs.AIcs.LG
keywords GRPOLLMreasoningrolloutefficiencyzero-variancepromptsselectivetemporaltrainingdynamicsdataselectionmath
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish that most of the compute spent generating responses in GRPO-style reinforcement learning for LLM reasoning (a policy-gradient method that scores each sampled response against the others for the same prompt) is wasted on prompts that contribute no learning signal, and that this waste can be recognized before generation starts. Its central observation is a strong temporal consistency in prompt value: a prompt whose sampled responses all earn identical rewards (a zero-variance prompt, whose advantage is zero) is very likely to remain zero-variance in later epochs, while about 20% of such prompts recover and become useful again. GRESO (GRPO with Efficient Selective Rollout) turns this into an online pre-rollout filter that skips each prompt with a probability derived from its recent reward history, paired with a self-adjusting exploration rate and an adaptive rollout batch size, so previously uninformative prompts are still occasionally revisited. The payoff claimed is that reasoning RL can match the accuracy of Dynamic Sampling while cutting rollout wall-clock time by up to 2.4× and total training time by up to 2.0× — for instance, on Qwen2.5-Math-7B, rollout time drops from 155.9 to 65.5 hours at essentially equal average accuracy.

What carries the argument

The carrying mechanism is the temporal consistency of prompt value, quantified by two probabilities: $P(\text{Previous}|\text{Current})$, the chance that a currently zero-variance prompt was zero-variance before (above 90% throughout training), and $P(\text{Current}|\text{Previous})$, the chance that a previously zero-variance prompt stays zero-variance (roughly 80%, so about 20% recover). GRESO applies this through a probabilistic pre-rollout filter: prompt $x_i$ is skipped with probability $p_f(x_i) = 1 - p_e^{z_i}$, where $z_i$ is the number of its most recent consecutive zero-variance epochs and $p_e$ is a base exploration probability. Two adaptive components keep the filter stable across models and training stages: the base exploration probability self-adjusts to hold the observed zero-variance percentage near a target (25% in all experiments, split 1:2 between easy and hard prompts), and the rollout batch size shrinks adaptively, $B_r = \min(B_r^{\mathrm{default}}, \beta B_\Delta/(1-\alpha))$, with safety factor $\beta = 1.25$, so the run stops oversampling once the remaining training batch can be filled.

What would settle it

Run GRESO to completion while also force-rolling-out a random sample of the prompts it skips, and check how many of those forced rollouts turn out to be effective (nonzero reward variance) and whether feeding them back into training raises final benchmark accuracy — if skipped prompts become effective at a rate well above the paper's 20% recovery baseline, the filter is suppressing useful exploration and the equal-accuracy claim would not hold up.

Watch

Extended reading notes

Core claim

The paper's central claim is that zero-variance prompts are temporally predictable, and that filtering them before rollout recovers most of the benefit of Dynamic Sampling at a fraction of its cost. A zero-variance prompt is one whose G sampled responses all receive identical rewards, so every response has zero GRPO advantage and the prompt teaches the policy nothing; the paper reports that this class can shrink the effective prompt ratio to about 20% late in training. The discovery is the temporal correlation: over 90% of zero-variance prompts in a given epoch were already zero-variance in earlier epochs, while roughly 20% of previously zero-variance prompts later become effective. GRESO operationalizes this by giving each prompt a filtering probability $p_f(x_i) = 1 - p_e^{z_i}$, where $z_i$ counts the most recent consecutive zero-variance rollouts for prompt $x_i$ and $p_e$ is a base exploration probability the algorithm adjusts online to keep the observed zero-variance ratio near its target. Empirically, on Qwen2.5-Math-7B trained on DAPO+MATH, GRESO matches Dynamic Sampling (57.5% versus 57.8% average accuracy across six math benchmarks) while using 6.3M rollouts instead of 13.1M and 65.5 rollout hours instead of 155.9.

Load-bearing premise

The load-bearing premise is that prompts which give no learning signal in one epoch keep giving no learning signal in later epochs — a pattern the paper measures on ordinary GRPO training — and that this pattern still holds after GRESO starts filtering prompts out of the training distribution.

Editorial extensions

If this is right

  • Dynamic Sampling runs can keep their accuracy with up to 3.35× fewer rollouts, cutting rollout wall-clock time by up to 2.4× and total training time by up to 2.0×, so the same results arrive in roughly half the GPU hours.
  • Late-stage GRPO training, where the effective prompt ratio can fall to about 20%, is instead held above 70% under GRESO, so rollout compute per step stays productive as training progresses.
  • Under a fixed time budget GRESO accumulates more effective rollouts — about 2 million effective rollouts in roughly 25 hours where Dynamic Sampling needs more than 40.
  • Because the filter is online and needs no auxiliary scoring model, it drops into an existing GRPO pipeline with one fixed hyperparameter set across models and datasets.
  • The probabilistic filter keeps roughly one in five previously zero-variance prompts in rotation, preserving a recovery channel that static dataset pruning would close.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The temporal-consistency signal should transfer to other verifiable-reward RL settings — code generation, tool use, agentic tasks — wherever identical rewards across sampled responses produce zero advantage; testing GRESO there is a direct next step not taken in the paper.
  • The filter acts as an implicit curriculum, concentrating compute on mid-difficulty prompts that the case study shows are most often selected; a finer-grained value ranking instead of binary filtering, which the paper names as future work, could tune that curriculum explicitly.
  • The temporal correlation is measured before filtering changes the data distribution; re-measuring it inside a GRESO run would show how long the equal-accuracy match persists as training continues.
  • Accuracy is flat across target zero-variance rates from 0% to 100% in the paper's sweep, so the target is currently a free knob; an adaptive target tied to model capability could capture further savings that the fixed 25% leaves on the table.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes GRESO, a pre-rollout filtering method for GRPO-based LLM reasoning. It observes that zero-variance prompts tend to remain zero-variance across epochs, uses each prompt's reward history to compute a probabilistic skip decision, and combines this with an adaptive rollout batch size. Experiments compare against Dynamic Sampling across six math benchmarks and three models, claiming comparable accuracy with up to 2.4x rollout speedup and 2.0x total training speedup. The central contribution is empirical: predicting uninformative prompts before rollout to avoid wasted generation.

Significance. If the empirical claims hold, the work is practically significant: it directly targets the rollout bottleneck in LLM RL and provides an online, lightweight selector with fixed hyperparameters across all experiments. Strengths include a well-motivated observation of effective-prompt decay (Figure 3), a simple probabilistic algorithm, a sweep over target zero-variance percentages (Table 3), and a public code release. The reported speedups are large and broadly consistent. However, the evidence for the headline claim of 'without accuracy degradation' is currently not strong enough: the evaluation protocol conflates best-checkpoint accuracy with end-of-training performance, there are no multiple-seed or error-bar comparisons, and the contribution of the probabilistic filter is not cleanly isolated from the adaptive batch size.

major comments (4)
  1. [Section 5.2, Tables 1 and 2, Appendix E] The no-degradation claim is measured at the checkpoint with the best average accuracy across six benchmarks, while the speedup numbers in Table 2 are for a fixed 1000-step run. This is not a matched comparison: if GRESO's best checkpoint occurs earlier than Dynamic Sampling's, the accuracy match can hold even if the final GRESO model is worse. The authors should report final-checkpoint accuracy for both methods, ideally averaged over multiple training seeds, and should make clear whether the #Rollout entries in Table 1 refer to total rollouts or rollouts up to the best checkpoint.
  2. [Section 5.3, Figure 5d] Figure 5d shows that adding only the adaptive batch size (DS + ABS) already substantially reduces rollout count, but no accuracy is reported for DS + ABS. Since GRESO combines the probabilistic pre-rollout filter with ABS, the incremental contribution of the filter to the headline efficiency is not isolated. The authors should report accuracy for DS + ABS and, if possible, for GRESO without ABS, so that the speedup can be attributed correctly.
  3. [Section 3.3 and Section 4.2] The temporal-consistency evidence in Figure 4a is measured on unmodified GRPO training, but GRESO changes which prompts are rolled out, altering the training trajectory and reward dynamics. The paper does not directly test whether the P(Current|Previous) and P(Previous|Current) correlations survive this distribution shift. The target-percentage sweep and effective-ratio plots are indirect evidence; a more direct check, such as measuring the correlation under GRESO itself or periodically rolling out a held-out set of skipped prompts, would strengthen the core assumption.
  4. [Appendix E and Table 1] All accuracy numbers come from a single training run with no error bars or multiple seeds, and several benchmarks (AIME24, AMC, Gaokao, OlympiadBench) are small. Differences of 0.3-0.5 percentage points in Table 1 are likely within evaluation noise. The paper should report variance across evaluation repetitions and ideally across seeds, or at minimum quantify the uncertainty on the small benchmarks, before claiming that accuracy is matched without degradation.
minor comments (6)
  1. [Equations (3) and (4)] Equation (4) defines z_i via a product over recent I_i,j indicators but does not state the empty-product convention for k=0; equation (3) should be typeset clearly as p_f(x_i)=1-p_e^{z_i} to avoid confusion.
  2. [Section 3.3 and Figure 4a] The sentence 'P(Current|Previous) curve shows that approximately 20% of prompts previously labeled as zero-variance become effective prompts' is easy to misread; since P(Current|Previous) is the probability of remaining zero-variance, the text should state that the remaining 80% stay zero-variance and 20% become effective.
  3. [Table 1] The column header '# Rollout' is ambiguous: it should state whether the number refers to total rollouts across the run, rollouts up to the best checkpoint, or rollouts per training step.
  4. [Section 2] There is a typo in Related Work: 'model-generated scoress' should be 'model-generated scores'.
  5. [Appendix E and Table 1] The dataset naming is inconsistent: the text uses 'Open R1 subset' and 'OR1' while Table 1 uses 'OR1'; the abbreviation should be defined consistently at first use.
  6. [Figure 1 and abstract] Figure 1 says 'reduces rollout overhead by up to 2x' while the abstract and Table 2 report up to 2.4x; the numbers should be aligned.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GRESO's pre-rollout filter is a predictive use of each prompt's own reward history, and the 'no accuracy degradation' claim is an external empirical comparison against Dynamic Sampling rather than a consequence of the filter's definition.

full rationale

The load-bearing claim—that GRESO matches Dynamic Sampling accuracy while using fewer rollouts—does not reduce to its inputs by construction. The filtering probability (Eq. 3, p_f(x_i)=1−p_e^{z_i}) is computed from each prompt's past consecutive zero-variance count, while the target concept of 'zero-variance prompt' (Eq. 5) is defined by the rewards generated during the current rollout; one is not defined in terms of the other. The temporal-consistency premise in Section 3.3 is measured directly on GRPO training in this paper (Figure 4a), not imported as an unverified self-citation; the cited prior works on temporal correlations are motivational, and the paper provides its own evidence. The self-adjusting exploration probability (Algorithm 1, lines 14–18) does control how many prompts are skipped, and the hand-set 25% target partly determines rollout count, but the headline 'without accuracy degradation' is an external comparison against Dynamic Sampling across six benchmarks and three models. Appendix F.1 further shows that varying the target from 0% to 100% changes average accuracy by only 48.1–48.5%, indicating that the accuracy outcome is not forced by the filtering knob. The speedup figures in Table 2 are measured wall-clock times from full 1000-step runs, not algebraic identities. The concerns raised by the skeptic—best-checkpoint reporting, lack of seeds, and distribution shift after filtering begins—are validity and robustness risks, not circularity: they do not exhibit any predicted quantity that is equivalent to its input by definition. I find no circular step in the derivation chain.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central method rests on reward-history-based prediction of zero-variance status, so its main free parameters are the hand-set filtering targets and exploration schedule; no new physical or mathematical entities are introduced.

free parameters (6)
  • targeted zero-variance percentage = 25%
    Chosen by hand and fixed across all experiments; it sets how aggressively GRESO filters and therefore directly influences the measured rollout speedup (Section 4.2, Appendix E).
  • easy-hard zero-variance allocation ratio = 1:2 (8.3% easy, 16.7% hard)
    Chosen from the intuition that more exploration of hard prompts is beneficial; the authors acknowledge a more optimal allocation may exist (Section 4.2 footnote).
  • base exploration probability initial value = 50%
    Hand-set initial condition for the adaptive p_e update (Appendix E).
  • base exploration probability minimum = 5%
    Floor to guarantee minimal exploration; prevents a target of 0 from being reached (Appendix E, Appendix F.1).
  • base exploration probability step size = 1%
    Fixed update increment for the adaptive p_e update (Section 4.2).
  • adaptive batch size safety factor = 1.25
    Fixed multiplier to ensure enough valid examples are collected; directly affects the number of rollouts per iteration (Section 4.2, Equation 6).
assumptions (3)
  • standard math If all rewards in a GRPO group are identical, the prompt contributes no learning signal (advantage is zero for every response).
    Follows from the advantage normalization in Equation 2, where identical rewards give zero variance and hence zero advantages; the paper treats these prompts as uninformative in Section 3.1.
  • domain assumption Zero-variance status is temporally consistent enough that history predicts future status, even after the policy changes.
    Motivated by Figure 4a measured under standard GRPO; GRESO relies on this for pre-rollout skipping while also assuming occasional exploration catches reverted prompts (Section 4.2). The filtered distribution is not used to re-measure the correlation.
  • domain assumption Skipping predicted zero-variance prompts does not degrade final accuracy because exploration probability keeps them in the training loop occasionally.
    The method's safety rests on this; Table 1 provides one supporting comparison per setup, but no repeated trials or error bars, and Appendix F.1 shows accuracy is insensitive to the target ratio.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts." pith.science (2026). https://pith.science/paper/RND6AART

@misc{pith2026250602177,
  author       = {Pith},
  title        = {Pith review of: Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RND6AART}},
  note         = {Machine review of arXiv:2506.02177}
}
read the original abstract

Reinforcement learning, such as PPO and GRPO, has powered recent breakthroughs in LLM reasoning. Scaling rollout to sample more prompts enables models to selectively use higher-quality data for training, which can stabilize RL training and improve model performance. However, this comes at the cost of significant computational overhead. In this paper, we show that a substantial portion of this overhead can be avoided by skipping uninformative prompts before rollout. Our analysis of reward dynamics reveals a strong temporal consistency in prompt value: prompts that are uninformative in one epoch of training are likely to remain uninformative in future epochs. Based on these insights, we propose GRESO (GRPO with Efficient Selective Rollout), an online, lightweight pre-rollout filtering algorithm that predicts and skips uninformative prompts using reward training dynamics. By evaluating GRESO on a broad range of math reasoning benchmarks and models, such as Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, and Qwen2.5-Math-7B, we show that GRESO achieves up to 2.4x wall-clock time speedup in rollout and up to 2.0x speedup in total training time without accuracy degradation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Kalman Meets Curriculum: Efficient Dynamic Prompt Selection for Adaptive RL Finetuning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A Kalman filter with policy-update-coupled process noise tracks non-stationary prompt difficulty and selects intermediate-difficulty batches for RL finetuning without extra rollouts.

  2. Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR

    cs.LG 2026-07 conditional novelty 6.0 of 10

    SARA uses a Bayesian stopping rule on a prompt's first few responses to abandon saturated response groups early and reallocate rollouts, matching accuracy with up to 67% fewer rollouts than dynamic sampling.

  3. Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance

    cs.CL 2026-08 conditional novelty 5.0 of 10

    RSTG selectively distills a teacher on negative zero-variance prompts, with confidence weighting, token-level selection, and auxiliary SFT, improving math and code RL post-training over naive GRPO+OPD.

Reference graph

Works this paper leans on

38 extracted references · 9 canonical work pages · cited by 3 Pith papers

  1. [1]

    Aime problems and solutions

    Art of Problem Solving. Aime problems and solutions. https://artofproblemsolving.com/wiki/index.php/AIME Problems and Solutions, 2024a. Accessed: 2025-04-20. Art of Problem Solving. Amc problems and solutions. https://artofproblemsolving.com/wiki/index.php?title=AMC Problems and Solutions, 2024b. Accessed: 2025-04-20. Yuntao Bai, Andy Jones, Kamal Ndousse...

    arXiv 2025

  2. [2]

    Training.Our method is implemented based on verl (Sheng et al.,

    OPEN-R1 30k subset (R1): A 30,000-example subset of the OPEN-R1 math dataset (Face, 2025). Training.Our method is implemented based on verl (Sheng et al.,

    work page 2025

  3. [6]

    On designing effective rl reward at training time for llm reasoning.arXiv preprint arXiv:2410.15115,

    Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for llm reasoning.arXiv preprint arXiv:2410.15115,

    arXiv

  4. [7]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.14008,

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.14008,

    arXiv

  5. [9]

    Data-efficient finetuning using cross-task nearest neighbors

    Hamish Ivison, Noah A Smith, Hannaneh Hajishirzi, and Pradeep Dasigi. Data-efficient finetuning using cross-task nearest neighbors. InFindings of the Association for Computational Linguistics: ACL 2023, pages 9036–9061,

    work page 2023

  6. [10]

    Large-scale data selection for instruction tuning.arXiv preprint arXiv:2503.01807,

    Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi. Large-scale data selection for instruction tuning.arXiv preprint arXiv:2503.01807,

    arXiv

  7. [11]

    Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment.arXiv preprint arXiv:2410.01679,

    Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment.arXiv preprint arXiv:2410.01679,

    arXiv

  8. [13]

    Let’s verify step by step.arXiv preprint arXiv:2305.20050,

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050,

    arXiv

Show all 38 references
  1. [14]

    Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783,

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783,

    arXiv

  2. [15]

    Enabling weak llms to judge response reliability via meta ranking.arXiv preprint arXiv:2402.12146,

    Zijun Liu, Boqun Kou, Peng Li, Ming Yan, Ji Zhang, Fei Huang, and Yang Liu. Enabling weak llms to judge response reliability via meta ranking.arXiv preprint arXiv:2402.12146,

    arXiv

  3. [16]

    s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025a

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand` es, and Tatsunori Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025a. Niklas Muennighoff, Zitong Yang, Weijia ...

    arXiv

  4. [17]

    Asynchronous rlhf: Faster and more efficient off-policy rl for language models.arXiv preprint arXiv:2410.18252,

    Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous rlhf: Faster and more efficient off-policy rl for language models.arXiv preprint arXiv:2410.18252,

    arXiv

  5. [18]

    OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennet...

    2024 arXiv

  6. [19]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    arXiv

  7. [20]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    arXiv

  8. [21]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599,

    arXiv

  9. [23]

    A minimalist approach to llm reasoning: from rejection sampling to reinforce.arXiv preprint arXiv:2504.11343,

    Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. A minimalist approach to llm reasoning: from rejection sampling to reinforce.arXiv preprint arXiv:2504.11343,

    arXiv

  10. [24]

    Not all rollouts are useful: Down-sampling rollouts in llm reinforcement learning.arXiv preprint arXiv:2504.13818,

    Yixuan Even Xu, Yash Savani, Fei Fang, and Zico Kolter. Not all rollouts are useful: Down-sampling rollouts in llm reinforcement learning.arXiv preprint arXiv:2504.13818,

    arXiv

  11. [25]

    13 An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement.arXiv preprint arXiv:2409.12122,

    arXiv

  12. [26]

    Limo: Less is more for reasoning.arXiv preprint arXiv:2502.03387,

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning.arXiv preprint arXiv:2502.03387,

    arXiv

  13. [27]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    arXiv

  14. [28]

    Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118, 2025a

    Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118, 2025a. Yufeng Yuan, Yu Yue...

    arXiv

  15. [29]

    Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu

    https://arxiv.org/abs/2504.14286. Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on gaokao benchmark.arXiv preprint arXiv:2305.12474,

    arXiv

  16. [30]

    Pytorch fsdp: experiences on scaling fully sharded data parallel.arXiv preprint arXiv:2304.11277,

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel.arXiv preprint arXiv:2304.11277,

    arXiv

  17. [31]

    Coverage-centric coreset selection for high pruning rates

    Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates. In11th International Conference on Learning Representations, ICLR 2023,

    2023

  18. [32]

    LLNL-affiliated authors were supported under Contract DE-AC52-07NA27344 and supported by the LLNL-LDRD Program under Project Nos

    14 Appendix Acknowledgment This work was partially supported by Google Research Award, Amazon Research Award, Intel, Li Auto, Moffett AI, and CMU CyLab Seed funding. LLNL-affiliated authors were supported under Contract DE-AC52-07NA27344 and supported by the LLNL-LDRD Program ...

    2024

  19. [33]

    We find that training on DAPO alone can degrade performance on LaTeX-based benchmarks, so we augment it with MATH to preserve formatting diversity and improve generalization

    DAPO+MATH (DM): We combine the DAPO dataset (Yu et al., 2025), which contains only integer solutions, with the MATH dataset (Hendrycks et al., 2021), which also contains 15 LaTeX-formatted solutions. We find that training on DAPO alone can degrade performance on LaTeX-based be...

    2025

  20. [35]

    We use 4xH100 for Qwen2.5-Math-1.5B training and 8xH100 for Qwen2.5-Math-7B and DeepSeek-R1-Distill-Qwen-1.5B

    for rollout. We use 4xH100 for Qwen2.5-Math-1.5B training and 8xH100 for Qwen2.5-Math-7B and DeepSeek-R1-Distill-Qwen-1.5B. We set the rollout temperature to 1 for vLLM (Kwon et al., 2023). The training batch size is set to 256, and the mini-batch size to

    2023

  21. [37]

    for efficient distributed training. We use 4 H100 for Qwen2.5-Math-1.5B training and 8 H100 for Qwen2.5-Math-7B and DeepSeek-R1-Distill-Qwen-1.5B (as it has a longer context length.) We set the targeted zero-variance percentage to 25% for all experiments and allocate it betwee...

    2021

  22. [2017]

    Safe rlhf: Safe reinforcement learning from human feedback.arXiv preprint arXiv:2310.12773,

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback.arXiv preprint arXiv:2310.12773,

    arXiv

  23. [2019]

    We use β1 = 0.9, β2 = 0.999, and apply a weight decay of 0.01

    optimizer with a constant learning rate of 1e-6. We use β1 = 0.9, β2 = 0.999, and apply a weight decay of 0.01. We use the following question template to prompt the LLM. For reward assignment, we give a score of 0.1 for successfully extracting an answer and a score of 1.0 if t...

    2025

  24. [2020]

    Reinforcement learning for reasoning in large language models with one training example.arXiv preprint arXiv:2504.20571,

    Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example.arXiv preprint arXiv:2504.20571,

    arXiv

  25. [2021]

    Reinforce++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262,

    11 Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262,

    arXiv

  26. [2022]

    Limr: Less is more for rl scaling.arXiv preprint arXiv:2502.11886,

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling.arXiv preprint arXiv:2502.11886,

    arXiv

  27. [2023]

    Active preference optimization for sample efficient rlhf.arXiv preprint arXiv:2402.10500,

    Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference optimization for sample efficient rlhf.arXiv preprint arXiv:2402.10500,

    arXiv

  28. [2024]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    arXiv

  29. [2025]

    Open r1: A fully open reproduction of deepseek-r1, January 2025.https://github.com/huggingface/open-r1

    Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025.https://github.com/huggingface/open-r1. Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning. arXiv preprint arXiv:2504.05185,

    2025

  30. [8196]

    We evaluate all models with temperature = 1 and repeat the test set 4 times for evaluation stability, i.e.,pass@1(avg@4), for all benchmarks

    Similar to (Wang et al., 2025), we evaluate models on those benchmarks every 50 steps and report the performance of the checkpoint that obtains the best average performance on six benchmarks. We evaluate all models with temperature = 1 and repeat the test set 4 times for evalu...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.