Pith. sign in

REVIEW 4 major objections 5 minor 14 cited by

Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ReMix reuses historical RL rollouts to cut reasoning-model training data volume by more than 30x while matching top accuracy.

desk verdict ReMix shows real efficiency gains by reusing off-policy data in RFT, but the 30x-450x headline is on rollout counts rather than tokens and deserves variance-aware verification before the exact magnitude is trusted. read the letter →

arxiv 2507.06892 v3 pith:GZC6EAZP submitted 2025-07-09 cs.LG cs.AIcs.CL

Jing Liang , Hongyao Tang , Yi Ma , Jinyi Liu , Yan Zheng , Shuyue Hu , Lei Bai , Jianye Hao This is my paper
classification cs.LGcs.AIcs.CL
keywords off-policyreinforcementlearningfinetuninglargelanguagemodelsmathreasoningproximalpolicyoptimizationsampleefficiencyreincarnationGRPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the sample-inefficiency bottleneck in reinforcement finetuning (RFT) of large reasoning models can be broken by resurrecting off-policy RL: instead of discarding rollouts after each PPO or GRPO update, a method called ReMix reuses them in mixed training batches, applies more gradient updates per rollout, and then resets to on-policy training partway through. On five math benchmarks, ReMix-PPO reports 52.10% average Pass@1 accuracy at 1.5B scale with 0.079M rollouts versus DeepScaleR's 52.14% with 2.519M rollouts, and 64.39% at 7B scale with 0.011M rollouts versus AceReason-Nemotron's 63.24% with an estimated 3.584M rollouts. The paper further argues that off-policyness has a measurable behavioral signature: heavier reuse of historical data shortens responses and suppresses self-reflection tokens, which speeds early learning but can collapse later performance, so the method balances the mix and then reincarnates. If right, RFT becomes far cheaper to scale, and the behavioral analysis offers a concrete explanation for why off-policy training of LLMs often degrades.

What carries the argument

The load-bearing object is the Mix-PPG objective: an importance-sampled clipped surrogate over a mixture of current-policy and historical-policy trajectories, combined with an increased update-to-data ratio that repeats gradient updates, a KL-convex constraint anchoring to both the base model and the previous policy, and a policy-reincarnation switch at a scheduled step that replaces the KL reference with the current policy and reverts to on-policy PPO or GRPO. The objective is grounded in a generalized off-policy policy improvement bound and uses a V-trace-corrected advantage estimator with truncated importance-sampling ratios. These components work together: the mixed objective and high update-to-data ratio give the early efficiency gain, the two-anchor KL keeps updates stable, and the reincarnation stage supplies the asymptotic convergence that pure off-policy training lacks.

What would settle it

An audit that records total generated tokens, GPU-hours, and hardware configuration for ReMix and for DeepScaleR and AceReason-Nemotron would settle it: if the token-normalized or hardware-normalized cost gap disappears or reverses, the central efficiency claim fails; if it survives, the claim's magnitude should be re-expressed in those units.

Watch

Extended reading notes

Core claim

ReMix establishes that proximal policy-gradient RFT can learn from its own history without losing final accuracy. It mixes a proportion p of rollouts sampled from recent historical policies with on-policy rollouts, applies an increased update-to-data ratio, regularizes the policy with a convex combination of KL constraints to the base model and to the previous policy, and at a scheduled step resets the KL reference to the current policy and switches back to ordinary PPO or GRPO. The central empirical discovery is that this staged scheme converts the early-stage efficiency of off-policy reuse into state-of-the-art-level math reasoning: with far fewer rollouts and training steps, ReMix-PPO outperforms or matches both its on-policy counterparts and recent RFT baselines at 1.5B and 7B scales. A secondary discovery is the "Whipping Effect": because negative advantages dominate, an off-policy objective pushes the policy toward lower importance ratios, which is systematically easier for shorter responses, so higher off-policy proportions shorten generations; the paper links this to a collapse in self-reflection behavior and shows that ReMix degrades less than baselines when the maximum response length is halved.

Load-bearing premise

The load-bearing premise is that counting rollouts (generated responses) faithfully measures training cost across models that differ in response length, context length, and hardware; if it does not, the reported 30x to 450x reductions are not actual compute reductions.

Editorial extensions

If this is right

  • Standard PPO and GRPO can be upgraded to reuse historical rollouts through ReMix, yielding higher accuracy than their on-policy counterparts at the same or lower rollout volume.
  • State-of-the-art-level math reasoning at 1.5B and 7B scales is achievable with well under 0.1M rollouts, implying roughly 30x to 450x less rollout data than leading baselines.
  • Off-policy RFT has a characteristic failure mode: too much off-policyness shortens responses and suppresses self-reflection, so maintaining a sufficient on-policy fraction is critical for stable training.
  • ReMix-trained models are comparatively robust to halved maximum response lengths because they have learned more concise reasoning.
  • Removing explicit guide tokens from the prompt template during ReMix training preserves accuracy on MATH500 while improving format robustness under prompt variation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If rollout-volume accounting were replaced by token-level or FLOP-level accounting, the headline 30x to 450x factors would likely shrink, since ReMix's own responses are shorter; the qualitative efficiency advantage may survive, but the stated magnitude depends on the chosen cost metric.
  • The Whipping Effect suggests a tunable control knob: an adaptive off-policy proportion that shrinks as the importance ratio drifts could extend the efficient early stage and postpone the need for reincarnation.
  • The self-reflection collapse result implies that reflection-token rate could serve as a cheap early-warning monitor for off-policy instability in other reasoning behaviors, such as tool use or verification loops.
  • The reincarnation schedule is a discrete switch; continuously interpolating the off-policy fraction during the transition would test whether the staging itself, rather than the stage identities, preserves asymptotic performance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ReMix, a method that adapts on-policy proximal policy gradient methods (PPO, GRPO) for LLM reinforcement finetuning to reuse off-policy data from historical policies. ReMix combines a mixed-policy clipped objective with an increased Update-To-Data ratio, a KL-convex constraint anchored to both the base model and the previous policy, and a policy-reincarnation transition to on-policy training after a fixed number of steps. Experiments on 1.5B and 7B DeepSeek-R1-Distill-Qwen models over five math benchmarks report ReMix-PPO reaching 52.10% average Pass@1 at 1.5B with 0.079M rollouts and 63.27%/64.39% at 7B with 0.007M/0.011M rollouts, which the authors compare to DeepScaleR (52.14%, 2.519M) and AceReason-Nemotron (63.24%, estimated 3.584M). The paper also presents ablations and analyses of off-policy training dynamics, including a claimed 'Whipping Effect' that explains shorter responses under off-policy training.

Significance. If the efficiency claims hold, ReMix is a practically valuable contribution: it shows that historical rollout data can be reused effectively in LLM reasoning RL, with controlled comparisons against PPO and GRPO that support the qualitative efficiency improvement. The method is presented with enough implementation detail to reproduce, and the ablations in Table 3 give a clear picture of which components matter. The strongest headline result, however, is an efficiency claim measured in rollout count, and the current manuscript does not fully verify that this metric is faithful for cross-model cost comparisons. The analysis of reasoning dynamics is also more suggestive than conclusive. These issues are fixable with additional measurements and framing, so the work is likely to be a solid contribution after revision.

major comments (4)
  1. [§4.1, §4.2, Figure 1, Tables 1–2] The central 30x–450x training-cost reduction is expressed purely in rollout data volume. Section 4.1 acknowledges that a stricter metric should account for rollout response length and justifies rollout counts by asserting that average response lengths are roughly at the same scale, but Section 4.2 immediately notes that ReMix's responses are shorter (Figure 5), and the compared baselines use very different maximum response lengths (8k to 32k in Tables 7–8). Since LLM inference cost scales with generated tokens, rollout count is not a faithful cross-model cost measure, and the exact magnitude of the claimed reduction is unverified. I note the likely direction of the bias favors ReMix, so the qualitative claim may survive, but the paper should report token-level or FLOP-level normalization, or at minimum the average response lengths for each model, before advertising the specific 30x–450x factors.
  2. [§4.2, Tables 1–2] All reported results are single-run point estimates without error bars or seed variance. This matters for the headline comparison: ReMix-PPO (1.5B, 350 steps) scores 52.10 versus DeepScaleR's 52.14, a difference well within typical run-to-run variation for RL training, yet the paper describes this as 'comparable' and uses it to support the 30x reduction claim. Similarly, the 7B results compare point estimates from a single checkpoint. At least a small number of independent seeds, or bootstrap intervals over checkpoints, should be reported for the main efficiency–performance comparisons.
  3. [§4.4.2, Eq. (7), Figure 9] The formal analysis of the 'Whipping Effect' is circular as written. The argument assumes importance ratios are close to 1, infers negative advantages from the positive policy loss in Figure 9, and then claims that longer responses produce larger distribution shift on later states, which is exactly the quantity that needs direct measurement. No per-token importance-ratio or length-conditioned evidence is provided to connect the sign of the loss to response shortening. This part should be reframed as a hypothesis, or supplemented with a direct measurement of how importance ratios and gradients vary with response length.
  4. [§4.2, footnote 7, Table 2] The 450x reduction against AceReason-Nemotron depends on a rollout volume that is estimated from the original paper rather than measured from training logs (Section 4.2 footnote 7). Given that this is one of the two anchor numbers for the headline claim, the sensitivity of the reduction factor to this estimate should be reported, for example by giving a plausible range over the estimated rollout count.
minor comments (5)
  1. [§1, §3.3] Section 3.3 is titled 'Policy Reincatenation' and the text contains several typos and grammatical errors; the paper would benefit from a careful copyedit.
  2. [§4.4.3, Table 4] The halved maximum response length is inconsistently reported as 4,096, 4,192, and 4,196 tokens in different places; unify these values.
  3. [§4.2, Figure 3 and Appendix D] Figure 3's caption says ReMix is 'around 3x to 6x faster' on Olympiad, while the AIME curve in the appendix reports 'around 1.2x to 1.6x faster'; check that the reported speedup factors match the actual curves.
  4. [§4.4.4, Table 5] The Pass@1 metric in Table 5 counts correct answers even when the format is wrong, while Format Cor is reported separately; state this explicitly in the main text so readers do not conflate the two columns.
  5. [§4.1, Tables 7–8] The notation 'Underlined values denote the fresh on-policy rollout in addition to off-policy data reuse in ReMix' is helpful, but the rollout counts in Table 1 do not appear to make the off-policy reuse portion explicit; clarify how the stated rollout volumes are derived from the batch sizes and steps in Tables 7–8.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the efficiency/perf claims are measured against external baselines, not derived from the objective; the Whipping Effect is an interpretation, and the one self-citation is not load-bearing.

full rationale

No significant circularity found. The paper's central efficiency and accuracy claims are empirical measurements: ReMix is trained on DeepScaleR-Preview-Dataset and evaluated on five held-out math benchmarks against fifteen external baseline models (Section 4.2, Tables 1–2). The pass@1 numbers and rollout-volume counts are recorded from the training runs, not predicted by fitting a parameter to the benchmark outcome. The ReMix objective (Eq. 6) is a combination of a mix-policy surrogate (Eq. 4), a KL-convex constraint (Eq. 5), and a two-stage reincarnation schedule; none of these equations is defined in terms of the benchmark aggregate that they are claimed to explain. The Whipping Effect analysis (Section 4.4.2) is a mechanistic interpretation of observed training dynamics: it uses the empirically positive policy-loss sign (Figure 9) to argue that negative advantages push the policy toward lower importance ratios, and then argues that long responses induce larger distribution shift, favoring shorter responses. This is a causal explanation, not a quantity fitted to the observed response-length decrease; there is no equation in which the predicted response length is equal by construction to the input data. The only self-citation of note is the KL-convex anchor idea referencing Ma et al. (2024), which shares authors with this paper; however, it is presented as design inspiration ('Recent studies ... demonstrate ... Inspired by this'), its effectiveness is independently ablated in Table 3, and no uniqueness theorem or forced-choice argument from that prior work is invoked. The paper even self-limits its efficiency metric in Section 4.1, admitting that a stricter metric would account for response length and that average lengths are only 'roughly at the same scale,' and later states ReMix's responses are shorter, which would only increase the efficiency advantage. That is a measurement-choice limitation, not circular derivation. All load-bearing evidence is empirical and self-contained, so the appropriate circularity score is 0.

Assumptions & free parameters 6 free parameters · 6 assumptions · 1 invented entities

The central efficiency claim rests on several hand-chosen hyperparameters (p, m, N, T, lambda schedule) and on standard off-policy RL assumptions imported from cited work. The Whipping Effect is a proposed mechanism rather than an independently evidenced entity. The most load-bearing assumption is that the rollout-count efficiency metric fairly represents compute cost across different training setups.

free parameters (6)
  • off-policy data proportion p = 0.4
    Default mixing ratio between historical and fresh data; sensitivity analysis in Fig. 4 covers p in 0.1-0.5, but 0.4 is hand-selected.
  • Update-To-Data ratio m = 2
    Number of gradient updates per data batch; chosen as default; higher UTD caused degradation per Fig. 5.
  • historical policy window size N = 2
    Number of past policies retained in the replay buffer; default setting.
  • policy reincarnation step T = 50 or 100 (PPO), 50 (GRPO)
    Step at which training switches from Mix-PPG to on-policy PPG; set by default, not tuned per benchmark.
  • KL-convex coefficient schedule lambda(t) = max(1 - 0.1*ceil(max(t-50,0)/10), 0.5)
    Hand-designed decay of the base-model KL weight; no derivation or sensitivity study beyond ablation w/o KL-Convex.
  • advantage clipping threshold c_bar = 1
    V-trace truncation threshold in Appendix A; fixed value.
assumptions (6)
  • domain assumption Generalized off-policy policy improvement lower bound (Queeney et al. 2021) guarantees stable off-policy proximal updates.
    Invoked to justify Mix-PPG but not proved or restated in the paper.
  • domain assumption V-trace with c-bar=1 provides a consistent advantage estimate under off-policy drift.
    Adopted from Espeholt et al. without proof; the truncation at 1 may introduce bias.
  • ad hoc to paper The importance sampling ratio in Mix-PPG is close to 1, so the clipping mechanism can be ignored in the formal analysis.
    Stated in Section 4.4.2 based on Figure 4; this simplification underpins Eq. 7 and the Whipping Effect explanation.
  • domain assumption Policy loss positivity implies advantages are negative most of the time.
    Inferred from Figure 9 in Appendix D; used to conclude the model minimizes loss by reducing the importance ratio, but the sign of advantages is an empirical observation, not a proven property.
  • domain assumption The base models DeepSeek-R1-Distill-Qwen-1.5B/7B are appropriate starting points and the DeepScaleR dataset is a suitable training distribution.
    Training and evaluation depend on these choices; results may not transfer to other bases or datasets.
  • domain assumption Rollout data volume (number of responses) is a faithful proxy for training cost across methods.
    Used as the foundational efficiency metric; the paper acknowledges response lengths differ and that a stricter metric would include length.
invented entities (1)
  • Whipping Effect
    purpose: Explains why off-policy training in ReMix implicitly prefers shorter responses, linking negative advantages to a growing importance-ratio mismatch over long trajectories.
    The effect is inferred from the authors' own training curves and a simplified loss analysis; it is not independently measured or predicted outside this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model." pith.science (2026). https://pith.science/paper/GZC6EAZP

@misc{pith2026250706892,
  author       = {Pith},
  title        = {Pith review of: Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GZC6EAZP}},
  note         = {Machine review of arXiv:2507.06892}
}
read the original abstract

Reinforcement Learning (RL) has demonstrated its potential to improve the reasoning ability of Large Language Models (LLMs). One major limitation of most existing Reinforcement Finetuning (RFT) methods is that they are on-policy RL in nature, i.e., data generated during the past learning process is not fully utilized. This inevitably comes at a significant cost of compute and time, posing a stringent bottleneck on continuing economic and efficient scaling. To this end, we launch the renaissance of off-policy RL and propose Reincarnating Mix-policy Proximal Policy Gradient (ReMix), a general approach to enable on-policy RFT methods like PPO and GRPO to leverage off-policy data. ReMix consists of three major components: (1) Mix-policy proximal policy gradient with an increased Update-To-Data (UTD) ratio for efficient training; (2) KL-Convex policy constraint to balance the trade-off between stability and flexibility; (3) Policy reincarnation to achieve a seamless transition from efficient early-stage learning to steady asymptotic improvement. In our experiments, we train a series of ReMix models upon PPO, GRPO and 1.5B, 7B base models. ReMix shows an average Pass@1 accuracy of 52.10% (for 1.5B model) with 0.079M response rollouts, 350 training steps and achieves 63.27%/64.39% (for 7B model) with 0.007M/0.011M response rollouts, 50/75 training steps, on five math reasoning benchmarks (i.e., AIME'24, AMC'23, Minerva, OlympiadBench, and MATH500). Compared with 15 recent advanced models, ReMix shows SOTA-level performance with an over 30x to 450x reduction in training cost in terms of rollout data volume. In addition, we reveal insightful findings via multifaceted analysis, including the implicit preference for shorter responses due to the Whipping Effect of off-policy discrepancy, the collapse mode of self-reflection behavior under the presence of severe off-policyness, etc.

Figures

Figures reproduced from arXiv: 2507.06892 by the authors.

Figure 1
Figure 1. Efficiency-Performance Comparison for 1.5B Models ( [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The conceptual illustration of RFT for LLMs with different proximal policy gradient (PPG) methods [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Training Efficiency Comparison for ReMix-PPO and PPO (1.5B) on MATH and Olympiad. [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Training Dynamics regarding Importance Sampling Ratio, Accuracy, and Response Length under [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Training Dynamics regarding Accuracy, Response Length, Self-reflection Rate for On-policy v.s. Off [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Training Efficiency Comparison for ReMix-PPO and PPO (1.5B) on Minerva. [PITH_FULL_IMAGE:figures/full_fig_p031_6.png]
Figure 7
Figure 7. Figure 7: Training Efficiency Comparison for ReMix-PPO and PPO (1.5B) on AMC’23. [PITH_FULL_IMAGE:figures/full_fig_p031_7.png]
Figure 8
Figure 8. Figure 8: Training Efficiency Comparison for ReMix-PPO and PPO (1.5B) on AIME’24. [PITH_FULL_IMAGE:figures/full_fig_p032_8.png]
Figure 9
Figure 9. Figure 9: Policy Loss under Varying Proportions of Off-policy Data [PITH_FULL_IMAGE:figures/full_fig_p032_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Models

    cs.CV 2026-04 unverdicted novelty 8.0 of 10

    OP-GRPO is the first off-policy GRPO method for flow-matching models that reuses trajectories via replay buffer and importance sampling corrections, matching on-policy performance with 34.2% of the training steps.

  2. Learning Agentic Policy from Action Guidance

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    ActGuide-RL uses human action data as plan-style guidance in mixed-policy RL to overcome exploration barriers in LLM agents, matching SFT+RL performance on search benchmarks without cold-start training.

  3. Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    This survey introduces the Generate-Filter-Control-Replay (GFCR) taxonomy to structure rollout pipelines for RL-based post-training of reasoning LLMs.

  4. Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information

    cs.LG 2026-07 conditional novelty 6.0 of 10

    OC-GRPO reweights GRPO gradients with an importance ratio so that hints used during rollout generation still optimize the original unguided objective, delivering a 13.8% relative Pass@1 gain over vanilla GRPO.

  5. From $P(y|x)$ to $P(y)$: Investigating Reinforcement Learning in Pre-train Space

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    PreRL applies reward-driven updates to P(y) in pre-train space, uses Negative Sample Reinforcement to prune bad reasoning paths and boost reflection, and combines with standard RL in Dual Space RL to outperform baseli...

  6. ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

    cs.LG 2026-07 conditional novelty 5.5 of 10

    Injecting correct reference-policy samples and optimizing a mixed importance-sampling ratio prevents validation collapse and raises asymptotic math-reasoning scores beyond reverse-KL baselines.

  7. RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents

    cs.LG 2026-07 conditional novelty 5.5 of 10

    Reward-Swap Policy Optimization uses dense-reward exploration to generate diverse trajectories that then train the final policy strictly under true outcome rewards, lifting multi-turn agent performance.

  8. ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

    cs.LG 2026-07 conditional novelty 5.0 of 10

    ARMOR adds correct reference-policy anchor samples to each RL batch and optimizes a reference-mixture importance ratio, preventing validation collapse and extending performance gains.

  9. Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    Survey mapping RL techniques onto LLM training and highlighting gaps in value-based, off-policy, and bootstrapping methods.

  10. RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    POPO uses recency-based prioritized group replay and decoupled off-policy optimization to avoid zero-variance ineffective samples in RLVR, accelerating LLM reasoning finetuning with fewer rollouts.

  11. OGER: A Robust Offline-Guided Exploration Reward for Hybrid Reinforcement Learning

    cs.AI 2026-04 unverdicted novelty 5.0 of 10

    OGER adds an auxiliary exploration reward built from offline trajectories and model entropy to hybrid RL training, yielding gains on math reasoning benchmarks and out-of-domain generalization.

  12. OGER: A Robust Offline-Guided Exploration Reward for Hybrid Reinforcement Learning

    cs.AI 2026-04 conditional novelty 4.0 of 10

    Novice programmers completed more tasks with lower workload using GitHub Copilot versus a human partner, but reported significantly more positive and arousing emotions with the human teammate.

  13. Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies

    cs.LG 2025-08 conditional novelty 4.0 of 10

    A survey that categorizes deep reinforcement learning scaling strategies into data, network, and training budget dimensions and outlines challenges for scaling DRL systems.

  14. A Survey of Reinforcement Learning for Large Reasoning Models

    cs.CL 2025-09 accept novelty 3.0 of 10

    A survey compiling RL methods, challenges, data resources, and applications for enhancing reasoning in large language models and large reasoning models since DeepSeek-R1.

Reference graph

Works this paper leans on

60 extracted references · 28 canonical work pages · cited by 12 Pith papers

  1. [1]

    Courville, and Marc G

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Bellemare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. In NeurIPS, 2022

    work page 2022

  2. [2]

    L1: Controlling how long a reasoning model thinks with reinforcement learning

    Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04697, 2025

    arXiv 2025

  3. [3]

    Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms

    Arash Ahmadian, Chris Cremer, Matthias Gall \' e , Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet \" U st \" u n, and Sara Hooker. Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. In ACL, pages 12248--12267, 2024

    work page 2024

  4. [4]

    Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025

    Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025. URL https://hkunlp.github.io/blog/2025/Polaris

    2025

  5. [5]

    Asymmetric reinforce for off-policy reinforcement learning: Balancing positive and negative rewards

    Charles Arnal, Ga A T tan Narozniak, Vivien Cabannes, Yunhao Tang, Julia Kempe, and Remi Munos. Asymmetric reinforce for off-policy reinforcement learning: Balancing positive and negative rewards. arXiv preprint arXiv:2506.20520, 2025

    arXiv 2025

  6. [6]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022 a

    arXiv 2022

  7. [7]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022 b

    arXiv 2022

  8. [8]

    Randomized ensembled double q-learning: Learning fast without a model

    Xinyue Chen, Che Wang, Zijian Zhou, and Keith W Ross. Randomized ensembled double q-learning: Learning fast without a model. In ICLR, 2021

    work page 2021

Show all 60 references
  1. [9]

    Acereason-nemotron: Advancing math and code reasoning through reinforcement learning

    Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acereason-nemotron: Advancing math and code reasoning through reinforcement learning. arXiv preprint arXiv:2505.16400, 2025

    2025 arXiv

  2. [10]

    Soft policy optimization: Online off-policy rl for sequence models

    Taco Cohen, David W Zhang, Kunhao Zheng, Yunhao Tang, Remi Munos, and Gabriel Synnaeve. Soft policy optimization: Online off-policy rl for sequence models. arXiv preprint arXiv:2503.05453, 2025

    2025 arXiv

  3. [11]

    Reinforcement learning for reasoning in small llms: What works and what doesn't

    Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small llms: What works and what doesn't. arXiv preprint arXiv:2503.16219, 2025

    2025

  4. [12]

    IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures

    Lasse Espeholt, Hubert Soyer, R \' e mi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. In ICML, v...

    2018

  5. [13]

    Concise reasoning via reinforcement learning

    Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning. arXiv preprint arXiv:2504.05185, 2025

    2025

  6. [14]

    Fujimoto, H

    S. Fujimoto, H. v. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In ICML, volume 80, pages 1582--1591, 2018

    2018

  7. [15]

    Omni-math: A universal olympiad level mathematic benchmark for large language models

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni-math: A universal ...

    2025

  8. [16]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  9. [17]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML, pages 1861--1870. Pmlr, 2018

    2018

  10. [18]

    O lympiad B ench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. O lympiad B ench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientifi...

    2024

  11. [19]

    Skywork open reasoner 1 technical report

    Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner 1 technical report. arXiv preprint arXiv:2505.2...

    2025 arXiv

  12. [20]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In NeurIPS, 2021

    2021

  13. [21]

    Rainbow: Combining improvements in deep reinforcement learning

    Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Sheila A. McIlraith and Kilian Q. Weinberger, edito...

    2018

  14. [22]

    Dropout q-functions for doubly efficient reinforcement learning

    Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout q-functions for doubly efficient reinforcement learning. arXiv preprint arXiv:2110.02034, 2021

    2021 arXiv

  15. [23]

    Ii-thought

    Intelligent-Internet. Ii-thought. https://ii.inc/web/blog/post/ii-thought, 2025

    2025

  16. [24]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  17. [25]

    Towards mitigating hallucination in large language models via self-reflection

    Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating hallucination in large language models via self-reflection. arXiv preprint arXiv:2310.06271, 2023

    2023 arXiv

  18. [26]

    Kakade and John Langford

    Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In ICML, pages 267--274, 2002

    2002

  19. [27]

    Kimi, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  20. [28]

    Solving quantitative reasoning problems with language models

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. NeurIPS, 35: 0 3843--3857, 2022

    2022

  21. [29]

    Repo: Replay-enhanced policy optimization

    Siheng Li, Zhanhui Zhou, Wai Lam, Chao Yang, and Chaochao Lu. Repo: Replay-enhanced policy optimization. arXiv preprint arXiv:2506.09340, 2025 a

    2025 arXiv

  22. [30]

    From system 1 to system 2: A survey of reasoning large language models

    Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. From system 1 to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502.17419, 2025 b

    2025 arXiv

  23. [31]

    Enhancing robotic manipulation with ai feedback from multimodal large language models

    Jinyi Liu, Yifu Yuan, Jianye Hao, Fei Ni, Lingzhi Fu, Yibin Chen, and Yan Zheng. Enhancing robotic manipulation with ai feedback from multimodal large language models. arXiv preprint arXiv:2402.14245, 2024

    2024 arXiv

  24. [32]

    From chaos to order: The atomic reasoner framework for fine-grained reasoning in large language models

    Jinyi Liu, Yan Zheng, Rong Cheng, Qiyu Wu, Wei Guo, Fei Ni, Hebin Liang, Yifu Yuan, Hangyu Mao, Fuzheng Zhang, et al. From chaos to order: The atomic reasoner framework for fine-grained reasoning in large language models. arXiv preprint arXiv:2503.15944, 2025

    2025 arXiv

  25. [33]

    Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, 2025

    2025

  26. [34]

    Reining generalization in offline reinforcement learning via representation distinction

    Yi Ma, Hongyao Tang, Dong Li, and Zhaopeng Meng. Reining generalization in offline reinforcement learning via representation distinction. NeurIPS, 36: 0 40773--40785, 2023

    2023

  27. [35]

    Iteratively refined behavior regularization for offline reinforcement learning

    Yi Ma, Jianye Hao, Xiaohan Hu, Yan Zheng, and Chenjun Xiao. Iteratively refined behavior regularization for offline reinforcement learning. NeurIPS, 37: 0 56215--56243, 2024

    2024

  28. [36]

    Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems, 2024

    Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems, 2024

    2024

  29. [37]

    OpenAI . Gpt-3.5. Technical report, OpenAI, 2022. URL https://platform.openai.com/docs/models/gpt-3-5

    2022

  30. [38]

    Cassandras

    James Queeney, Yannis Paschalidis, and Christos G. Cassandras. Generalized proximal policy optimization with sample reuse. In NeurIPS, pages 11909--11919, 2021

    2021

  31. [39]

    Speq: Offline stabilization phases for efficient q-learning in high update-to-data ratio reinforcement learning

    Carlo Romeo, Girolamo Macaluso, Alessandro Sestini, and Andrew D Bagdanov. Speq: Offline stabilization phases for efficient q-learning in high update-to-data ratio reinforcement learning. In RLC, 2021

    2021

  32. [40]

    Tapered off-policy reinforce: Stable and efficient reinforcement learning for llms

    Nicolas Le Roux, Marc G Bellemare, Jonathan Lebensold, Arnaud Bergeron, Joshua Greaves, Alex Fr \'e chette, Carolyne Pelletier, Eric Thibodeau-Laufer, S \'a ndor Toth, and Sam Work. Tapered off-policy reinforce: Stable and efficient reinforcement learning for llms. arXiv prepr...

    2025 arXiv

  33. [41]

    Jordan, and Philipp Moritz

    John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, volume 37, pages 1889--1897, 2015

    2015

  34. [42]

    Jordan, and Pieter Abbeel

    John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In ICLR, 2016

    2016

  35. [43]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  36. [44]

    ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, et al. Seed1. 5-thinking: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13914, 2025

    2025

  37. [45]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  38. [46]

    David Silver and Richard S. Sutton. Welcome to the era of experience, 2025

    2025

  39. [47]

    Fastcurl: Curriculum reinforcement learning with progressive context extension for efficient training r1-like reasoning models

    Mingyang Song, Mao Zheng, Zheng Li, Wenjie Yang, Xuan Luo, Yue Pan, and Feng Zhang. Fastcurl: Curriculum reinforcement learning with progressive context extension for efficient training r1-like reasoning models. arXiv preprint arXiv:2503.17287, 2025

    2025

  40. [48]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement learning - an introduction. Adaptive computation and machine learning. MIT Press, 1998. ISBN 978-0-262-19398-6

    1998

  41. [49]

    Rl-finetuning llms from on-and off-policy data with a single algorithm

    Yunhao Tang, Taco Cohen, David W Zhang, Michal Valko, and R \'e mi Munos. Rl-finetuning llms from on-and off-policy data with a single algorithm. arXiv preprint arXiv:2503.19612, 2025

    2025 arXiv

  42. [50]

    Reft: Reasoning with reinforced fine-tuning

    Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. In ACL, pages 7601--7614, 2024

    2024

  43. [51]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  44. [52]

    Truly proximal policy optimization

    Yuhui Wang, Hao He, and Xiaoyang Tan. Truly proximal policy optimization. In UAI, volume 115, pages 113--122, 2019

    2019

  45. [53]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 35: 0 24824--24837, 2022

    2022

  46. [54]

    Light-r1: Curriculum sft, DPO and RL for long COT from scratch and beyond

    Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang. Light-r1: Curriculum sft, DPO and RL for long COT from scratch and beyond. arXiv preprint arXiv::2503....

    2025 arXiv

  47. [55]

    Learning to reason under off-policy guidance

    Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945, 2025

    2025 arXiv

  48. [56]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  49. [57]

    Reasonflux: Hierarchical llm reasoning via scaling thought templates

    Ling Yang, Zhaochen Yu, Bin Cui, and Mengdi Wang. Reasonflux: Hierarchical llm reasoning via scaling thought templates. arXiv preprint arXiv:2502.06772, 2025 b

    2025 arXiv

  50. [58]

    Star: Self-taught reasoner bootstrapping reasoning with reasoning

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D Goodman. Star: Self-taught reasoner bootstrapping reasoning with reasoning. In NeurIPS, volume 1126, 2024

    2024

  51. [59]

    Rest-mcts*: Llm self-training via process reward guided tree search

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search. NeurIPS, 37: 0 64735--64772, 2024

    2024

  52. [60]

    Adaptthink: Reasoning models can learn when to think

    Jiajie Zhang, Nianyi Lin, Lei Hou, Ling Feng, and Juanzi Li. Adaptthink: Reasoning models can learn when to think. arXiv preprint arXiv:2505.13417, 2025

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.