Pith. sign in

REVIEW 4 major objections 5 minor 56 references

A generalist language model can be trained to match the accuracy of online multi-teacher distillation by recycling the replay memory left over from domain experts' RL runs, at roughly 20x higher training throughput.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:26 UTC pith:W2X5MHSV

load-bearing objection A well-ablated, plausible method that reuses RL replay to train a generalist offline; the efficiency win is real, but parity with MOPD is not yet pinned down because of single runs and an unreported threshold. the 4 major comments →

arxiv 2607.19450 v2 pith:W2X5MHSV submitted 2026-07-21 cs.LG cs.AI

REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learning

Yunjie Chen , Xiaoxin Chen , Fang Wang This is my paper
classification cs.LG cs.AI
keywords replay memoryoffline reinforcement learningpolicy distillationmulti-teacher distillationlarge language modelsclass-balanced samplingtruncated importance samplingpost-training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the rollout trajectories an RL run logs as a by-product—its replay memory—are not waste but a reusable dataset. It proposes REGEN, which trains a single generalist student on the pooled replay memories of several domain-specialized teachers using an offline RL objective with class-balanced sampling and a clipped importance-weight correction. The paper reports that REGEN matches the accuracy of online multi-teacher distillation across math reasoning, code generation, and instruction following, while running about 20 times faster because the student no longer needs teachers or a rollout engine during training. If correct, this turns online RL into a data-production stage: the same logs could train students of different sizes and future objectives, and the heavy compute moves to a single offline pass.

Core claim

The paper claims that a generalist can be trained entirely offline by: (1) recording, for every rollout in the teachers' replay buffers, the negative log-likelihood z of the exact policy snapshot that generated it, (2) down-sampling each query's positive and negative responses to the smaller group so the advantage signal is class-balanced, and (3) optimizing the student with a loss where positive examples receive a group-normalized advantage and negative examples receive that advantage scaled by a truncated importance ratio clipped to [0,1]. This objective, REGEN, is shown to match or slightly trail the online multi-teacher distillation baseline on five benchmarks (within -2.1 to +1.1 accura

What carries the argument

The load-bearing object is the replay buffer augmented with the recorded negative log-likelihood z, so that each trajectory carries its generating policy mu via mu(y|x)=exp(-z). On this buffer, the REGEN loss applies a per-domain reward threshold r_H^d to separate positive from negative responses, a per-query class-balanced downsampling step, and an asymmetric truncated importance weight—clipped to [0,1] only for negatives—so the student can safely learn from trajectories produced by many different policy snapshots and domains without online teacher inference.

Load-bearing premise

The method hinges on the per-domain reward threshold r_H^d correctly splitting trajectories into positive and negative groups, and on the stored NLL z exactly recovering the behavior policy mu for every rollout; the paper never reports these thresholds and its own appendices give conflicting reward scales for Math (0/1 vs -1/1), so the premise is under-specified.

What would settle it

Inspect the released replay buffers: if the Math-domain rewards are stored as 0/1 as stated in Appendix B, then the class-balanced grouping in Eq. 8-9 and the advantage normalization in Eq. 5 cannot yield the centered-at-zero, ±1 bimodal reward distribution shown in Figure 4, implying the threshold r_H^d is set outside the actual reward range and the sign of the clipped importance weight for most Math examples is ambiguous. A simpler test: re-run REGEN while shifting r_H^d by a small amount; if accuracy collapses sharply, the method is not robust to the threshold choice.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single RL run can become a reusable data asset: the same replay memory can train multiple students of different sizes (the paper shows 1.5B logs training a 0.5B student) without fresh teacher rollouts.
  • Student training cost scales with the offline dataset size and the student's own forward/backward pass, not with the number of teachers or the rollout distribution, since teachers and rollout engines are fully removed from the loop.
  • Multi-domain aggregation introduces negligible interference: the multi-domain REGEN student stays within 2 accuracy points of per-domain specialists on all five benchmarks.
  • Because replay memory contains successful trajectories from early and late training stages, REGEN outperforms online distillation under pass@10 on code, suggesting the method transfers a broader behavior distribution than the converged teacher's on-policy sampling.
  • The objective applies to multi-turn tasks and to students whose backbone or vocabulary differs from the teachers', since the importance correction only needs the stored NLL, not a shared tokenizer.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the result holds, the marginal cost of adding a new domain to a generalist is just the cost of one RL run to produce a new replay buffer; future post-training could be decomposed into many small independent RL data runs feeding one shared offline learner.
  • The per-domain reward threshold r_H^d is not reported, and the paper's Appendix B and Figure 4 disagree about the Math reward scale (text says 0/1, figure shows -1/1 with mean 0). A reader cannot currently reproduce the class-balanced split; setting this threshold wrongly would flip the sign of the contrastive signal.
  • The method may also simplify curriculum and multi-stage training: instead of distilling a converged teacher, one can concatenate replay memories from several generations of base models and retrain a generalist, turning model upgrades into a data-aggregation problem.
  • A natural test beyond the paper: hold out one domain's replay memory and ask whether a generalist trained on the other domains still gains from a small amount of the held-out domain's negative examples—this would isolate the value of the contrastive signal versus mere imitation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes REGEN, an offline RL method that trains a multi-domain generalist LLM by recycling the replay memories logged during per-domain teacher RL runs, instead of distilling online from the teachers as in MOPD. The objective combines a truncated-importance-sampling policy gradient (following TOPR), a group-relative advantage normalization, class-balanced sampling, and a per-domain reward threshold to classify positive and negative trajectories. Experiments on Qwen2.5-1.5B across math, code, and instruction following report accuracy roughly on par with MOPD while claiming ~20x higher training throughput, plus ablations on sampling/objective, capacity mismatch, and multi- vs single-domain training.

Significance. If the central claim holds, REGEN is a conceptually useful step: it decouples data generation from student training, turns RL runs into reusable data assets, and allows a single offline pass to serve multiple student sizes. The paper has genuine strengths: it gives a concrete objective with a plausible motivation, includes ablations isolating sampling and objective choices, tests capacity mismatch, and releases code. However, the empirical parity claim rests on several under-specified and internally inconsistent components—most importantly the per-domain reward threshold and the reward scale recorded in the replay buffer—and the efficiency comparison conflates an algorithmic advantage with an uneven GPU allocation. These issues are fixable but require additional detail and analysis.

major comments (4)
  1. [§3.2, Eq. (6)-(9)] The per-domain threshold r_H^d is a free parameter that determines whether a trajectory is treated as positive or negative. This classification directly drives class-balanced sampling (Eq. 8-9) and the sign of the truncated importance weight in Eq. 6. For the continuous-reward Code and Alignment domains, the paper never reports the chosen threshold values, how they were selected, or any sensitivity analysis. Without this information, the Table 1 claim that REGEN 'matches MOPD' is not reproducible and could reflect threshold tuning rather than a robust property of the method. Please report the exact thresholds for all three domains and provide a sensitivity study (e.g., varying r_H^d across plausible reward quantiles).
  2. [Appendix B vs. Appendix C, Fig. 4] There is a direct factual contradiction about the reward scale. Appendix B states that the Math teacher uses reward 1 for correct and 0 for incorrect, with -1 used only when redefining the reward for the TOPR baseline. But Fig. 4, described as the 'accumulated replay memory' used by REGEN, shows the Math reward distribution concentrated at -1.0 and 1.0 with mean 0.000. If REGEN's replay comes from GRPO teacher training, the logged Math rewards should be 0/1, not -1/1. If Fig. 4 was produced after applying the TOPR relabeling, that must be stated explicitly. This ambiguity matters because Eq. 6's threshold and the clipped importance weight depend on the exact reward values stored in Ξ_d.
  3. [§3.1-3.2, Eq. (6)-(7)] The method assumes that the stored NLL z exactly recovers the behavior policy: μ(y|x) = exp(-z). This is load-bearing for the importance weight, since the clipped ratio π_θ(y|x)/μ(y|x) is used to down-weight negative samples under potentially large distribution shift. The paper does not state whether z records the exact log-probability under the sampling distribution used at rollout time, including decoding temperature, top-p, and the specific policy snapshot. If z is only an approximation or is averaged over responses, the importance weight is biased and the sign/magnitude of the off-policy correction is unreliable. Please specify the exact recording procedure and, if possible, report diagnostics that the recovered μ matches the empirical rollout distribution.
  4. [§4.2, Table 1 and Fig. 2] Two issues undermine the quantitative comparison. First, Fig. 2 compares MOPD with three GPUs serving teachers and one GPU training the student against REGEN using all four GPUs for the student. The reported 20x throughput gap is therefore in part a hardware-allocation artifact, not purely an algorithmic property of REGEN; an iso-resource comparison (e.g., all methods on four GPUs, or per-GPU throughput) is needed to support the efficiency claim. Second, all tables report a single run with no error bars or significance tests. Differences such as -2.1 on MBPP, -1.7 on IFEval, and +1.1 on MATH are small relative to typical seed variance in LLM post-training; without multiple seeds or confidence intervals, the claim that REGEN 'matches' MOPD is not statistically grounded.
minor comments (5)
  1. [Figure 1 caption] Typography errors: 'Muti-Experts' and 'Recyling' should be corrected to 'Multi-Experts' and 'Recycling'.
  2. [§3.2, Eq. (5)-(6)] The notation for the advantage is inconsistent: Eq. (5) writes the loss with an overall A^{REGEN}, while Eq. (6) defines a per-token or per-trajectory Ā. Clarify whether A^{REGEN} is constant over tokens or is indexed by t, and define the stop-gradient placement unambiguously.
  3. [§4.1] The sentence 'yielding 140K trajectories with an equal number of positive and negative examples' should specify whether this count is after valid-query filtering and whether the distribution is uniform across domains; the text says 'roughly uniformly' but does not give per-domain counts.
  4. [Appendix A] The discussion of pass@10 says 'REGEN consistently outperforms MOPD under pass@10' but the MBPP gap is only +0.6 and HumanEval +4.2; 'consistently' is too strong given the single-run nature. Also, the phrase 'Other trend is observed' should read 'Another trend is observed.'
  5. [References] References to unpublished 2026 technical reports (Xiaomi MIMO-V2, Ma et al. MOPD, Ritter et al., Arnal et al.) should be updated with any published versions, and the MOPD citation is currently a preprint with no DOI.

Circularity Check

0 steps flagged

No significant circularity: REGEN's claims are empirical comparisons against external baselines; the loss function is assembled from cited prior methods and no fitted parameter is relabeled as a prediction.

full rationale

REGEN's central claim is an empirical accuracy/efficiency comparison against external baselines (BC, MOPD) on held-out benchmarks, not a derivation that feeds its own conclusion. The REGEN objective in Eq. 5–6 is explicitly assembled from prior work: truncated importance sampling from TOPR, group-relative advantage normalization from GRPO, and class-balanced sampling from CBS. The behavior-policy recovery via stored NLL z (Eq. 7) is a recording mechanism, not a fitted parameter; it defines the importance weight by construction but does not smuggle in the target result. The per-domain threshold r_H^d is under-specified and the reward-scale inconsistency between Appendix B and Appendix C is a reproducibility/correctness concern, not a circularity: no reported number is fit to the benchmarks it is then used to predict. There are no load-bearing self-citations from the present authors, no imported uniqueness theorem, and no renamed known result presented as a new derivation. The paper is self-contained against external benchmarks, so the honest finding is no significant circularity.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The central claim rests on four domain assumptions: replay data sufficiency, exact recovery of behavior-policy NLL, validity of TIS under large distribution shift, and cross-domain advantage normalization. The only fitted/chosen scalar is the per-domain reward threshold r_H^d, whose values are not reported. No new entities are postulated.

free parameters (1)
  • Per-domain reward threshold r_H^d
    Eq 6 classifies samples as positive/negative using r_H^d; no values are reported (Math, Code, Alignment). The threshold determines the contrastive set and the sign of the clipped-IS weight, so it is a hand-set parameter of the central objective.
axioms (4)
  • domain assumption Replay memories accumulated during specialist RL contain enough correct/incorrect signal to train a competent generalist.
    Sec 3.2: 'accumulating experiences across those domains should at least be capable of driving the model to become competent in those domains.' This is the core premise; if false, offline recycling cannot work.
  • domain assumption Stored NLL z(i) exactly recovers the generating policy mu(y(i)|x(i)) = exp(-z(i)).
    Eq 7; the importance weight in Eq 6 relies on an exact record of the behavior policy at generation time, across many policy snapshots.
  • domain assumption TOPR truncated importance sampling remains a valid correction under the large distribution shift between heterogeneous teacher policies and the student.
    Sec 3.2 bullet 1; the authors adopt TIS as the off-policy correction without a new theoretical guarantee.
  • domain assumption Per-query group normalization produces a scale-invariant advantage across domains with different reward scales.
    Eq 6 and Appendix C; reward ranges are [-1,1] for Math and [0,1] for Code/Alignment, so the normalized advantage must transfer across reward scales.

pith-pipeline@v1.3.0-alltime-deepseek · 15383 in / 15686 out tokens · 140937 ms · 2026-08-01T13:26:51.523819+00:00 · methodology

0 comments
read the original abstract

Large-scale online reinforcement learning (RL) is the predominant means of eliciting advanced abilities including long-term reasoning and agentic tool use in large language models (LLMs). However, continuing to scale it across vast task domains of interest remains challenging in both computational infrastructure and cost, especially when considering RL as merely a one-off learning stage. Recently, a widely used technique for distilling knowledge across various domains and training stages, multi-teacher on-policy distillation (MOPD), helps to decouple the RL stage, saving costs, while maintaining generality across vast domains. Nonetheless, similar to online RL, MOPD requires coupled inference and backward passes, which continues to limit its scalability and computational efficiency. To address these challenges, we propose REGEN: Replay-recycling for Expert-to-Generalist Distillation with Offline RL. Instead of distilling from multiple teacher models, REGEN trains a generalist by simply recycling the replay memory -- the free by-product of the teachers' specialized RL training -- and employing offline RL algorithms. REGEN completely decouples the rollout sampling from the backward training process and thus greatly reduces the training cost. Across mathematical reasoning, code generation, and instruction following, REGEN matches the accuracy of MOPD at substantially lower cost. It potentially turns online RL into a data synthesis process instead of a one-off learning stage, and can be extended to large-scale post-training without requiring heavy computational load. Code is available at https://github.com/yunjie-sysu/REGEN.

Figures

Figures reproduced from arXiv: 2607.19450 by Fang Wang, Xiaoxin Chen, Yunjie Chen.

Figure 1
Figure 1. Figure 1: Sketch of the Replay-recycling for Expert-to-Generalist Distillation with Offline RL (REGEN) method in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Training efficiency of REGEN versus MOPD on four [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Multi-domain versus single-domain training with REGEN on Qwen2.5-1.5B-Instruct. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Reward (Top) vs Advantage (Bottom) Distribution by Domain. The dashed lines indicate the mean values. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 19 linked inside Pith

  1. [1]

    Scaling Learning Algorithms Towards

    Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards

  2. [2]

    and Osindero, Simon and Teh, Yee Whye , journal =

    Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =

  3. [3]

    2016 , publisher=

    Deep learning , author=. 2016 , publisher=

    2016

  4. [4]

    Findings of the Association for Computational Linguistics: EACL 2026 , pages=

    Imbalanced gradients in rl post-training of multi-task llms , author=. Findings of the Association for Computational Linguistics: EACL 2026 , pages=

    2026

  5. [5]

    Advances in neural information processing systems , volume=

    Gradient surgery for multi-task learning , author=. Advances in neural information processing systems , volume=

  6. [6]

    The Thirteenth International Conference on Learning Representations , year=

    Prevalence of negative transfer in continual reinforcement learning: Analyses and a simple baseline , author=. The Thirteenth International Conference on Learning Representations , year=

  7. [7]

    Andrew Bagnell and Pieter Abbeel and Jan Peters , title =

    Takayuki Osa and Joni Pajarinen and Gerhard Neumann and J. Andrew Bagnell and Pieter Abbeel and Jan Peters , title =. CoRR , volume =. 2018 , url =. 1811.06711 , timestamp =

    Pith/arXiv arXiv 2018

  8. [8]

    Advances in Neural Information Processing Systems 1,

    Dean Pomerleau , editor =. Advances in Neural Information Processing Systems 1,. 1988 , url =

    1988

  9. [9]

    Efficient Reductions for Imitation Learning , booktitle =

    St. Efficient Reductions for Imitation Learning , booktitle =. 2010 , url =

    2010

  10. [10]

    A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning , booktitle =

    St. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning , booktitle =. 2011 , url =

    2011

  11. [11]

    The Twelfth International Conference on Learning Representations,

    Rishabh Agarwal and Nino Vieillard and Yongchao Zhou and Piotr Stanczyk and Sabela Ramos Garea and Matthieu Geist and Olivier Bachem , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    2024

  12. [12]

    The Twelfth International Conference on Learning Representations,

    Yuxian Gu and Li Dong and Furu Wei and Minlie Huang , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    2024

  13. [13]

    Rush , editor =

    Yoon Kim and Alexander M. Rush , editor =. Sequence-Level Knowledge Distillation , booktitle =. 2016 , url =. doi:10.18653/V1/D16-1139 , timestamp =

    doi:10.18653/v1/d16-1139 2016

  14. [14]

    CoRR , volume =

    Sergey Levine and Aviral Kumar and George Tucker and Justin Fu , title =. CoRR , volume =. 2020 , url =. 2005.01643 , timestamp =

    Pith/arXiv arXiv 2020

  15. [15]

    Off-Policy Deep Reinforcement Learning without Exploration , booktitle =

    Scott Fujimoto and David Meger and Doina Precup , editor =. Off-Policy Deep Reinforcement Learning without Exploration , booktitle =. 2019 , url =

    2019

  16. [16]

    Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction , booktitle =

    Aviral Kumar and Justin Fu and Matthew Soh and George Tucker and Sergey Levine , editor =. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction , booktitle =. 2019 , url =

    2019

  17. [17]

    Conservative Q-Learning for Offline Reinforcement Learning , booktitle =

    Aviral Kumar and Aurick Zhou and George Tucker and Sergey Levine , editor =. Conservative Q-Learning for Offline Reinforcement Learning , booktitle =. 2020 , url =

    2020

  18. [18]

    Sutton and Satinder Singh , editor =

    Doina Precup and Richard S. Sutton and Satinder Singh , editor =. Eligibility Traces for Off-Policy Policy Evaluation , booktitle =. 2000 , timestamp =

    2000

  19. [19]

    Safe and Efficient Off-Policy Reinforcement Learning , booktitle =

    R. Safe and Efficient Off-Policy Reinforcement Learning , booktitle =. 2016 , url =

    2016

  20. [20]

    Bellemare and Jonathan Lebensold and Arnaud Bergeron and Joshua Greaves and Alexandre Fr

    Nicolas Le Roux and Marc G. Bellemare and Jonathan Lebensold and Arnaud Bergeron and Joshua Greaves and Alexandre Fr. Tapered Off-Policy. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2503.14286 , eprinttype =. 2503.14286 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.14286 2025

  21. [21]

    Christiano and Jan Leike and Tom B

    Paul F. Christiano and Jan Leike and Tom B. Brown and Miljan Martic and Shane Legg and Dario Amodei , editor =. Deep Reinforcement Learning from Human Preferences , booktitle =. 2017 , url =

    2017

  22. [22]

    Long Ouyang and Jeffrey Wu and Xu Jiang and Diogo Almeida and Carroll L. Wainwright and Pamela Mishkin and Chong Zhang and Sandhini Agarwal and Katarina Slama and Alex Ray and John Schulman and Jacob Hilton and Fraser Kelton and Luke Miller and Maddie Simens and Amanda Askell and Peter Welinder and Paul F. Christiano and Jan Leike and Ryan Lowe , editor =...

    2022

  23. [23]

    CoRR , volume =

    John Schulman and Filip Wolski and Prafulla Dhariwal and Alec Radford and Oleg Klimov , title =. CoRR , volume =. 2017 , url =. 1707.06347 , timestamp =

    Pith/arXiv arXiv 2017

  24. [24]

    Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2402.03300 , eprinttype =. 2402.03300 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.03300 2024

  25. [25]

    Manning and Stefano Ermon and Chelsea Finn , editor =

    Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn , editor =. Direct Preference Optimization: Your Language Model is Secretly a Reward Model , booktitle =. 2023 , url =

    2023

  26. [26]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , journal =

    DeepSeek. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , journal =. 2025 , url =. doi:10.48550/ARXIV.2501.12948 , eprinttype =. 2501.12948 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.12948 2025

  27. [27]

    Hinton and Oriol Vinyals and Jeffrey Dean , title =

    Geoffrey E. Hinton and Oriol Vinyals and Jeffrey Dean , title =. CoRR , volume =. 2015 , url =. 1503.02531 , timestamp =

    Pith/arXiv arXiv 2015

  28. [28]

    CoRR , volume =

    OpenAI , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2412.16720 , eprinttype =. 2412.16720 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.16720 2024

  29. [29]

    MiMo-V2-Flash Technical Report , journal =

    LLM. MiMo-V2-Flash Technical Report , journal =. 2026 , url =. doi:10.48550/ARXIV.2601.02780 , eprinttype =. 2601.02780 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2601.02780 2026

  30. [30]

    Proceedings of the 23rd

    Shan You and Chang Xu and Chao Xu and Dacheng Tao , title =. Proceedings of the 23rd. 2017 , url =. doi:10.1145/3097983.3098135 , timestamp =

    arXiv 2017

  31. [31]

    CoRR , volume =

    Congming Zheng and Jiachen Zhu and Zhuoying Ou and Yuxiang Chen and Kangning Zhang and Rong Shan and Zeyu Zheng and Mengyue Yang and Jianghao Lin and Yong Yu and Weinan Zhang , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2510.08049 , eprinttype =. 2510.08049 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2510.08049 2025

  32. [32]

    CoRR , volume =

    Karl Cobbe and Vineet Kosaraju and Mohammad Bavarian and Mark Chen and Heewoo Jun and Lukasz Kaiser and Matthias Plappert and Jerry Tworek and Jacob Hilton and Reiichiro Nakano and Christopher Hesse and John Schulman , title =. CoRR , volume =. 2021 , url =. 2110.14168 , timestamp =

    Pith/arXiv arXiv 2021

  33. [33]

    Measuring Mathematical Problem Solving With the

    Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt , editor =. Measuring Mathematical Problem Solving With the. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual , year =

    2021

  34. [34]

    KodCode:

    Zhangchen Xu and Yang Liu and Yueqin Yin and Mingyuan Zhou and Radha Poovendran , editor =. KodCode:. Findings of the Association for Computational Linguistics,. 2025 , url =. doi:10.18653/V1/2025.FINDINGS-ACL.365 , timestamp =

    doi:10.18653/v1/2025.findings-acl.365 2025

  35. [35]

    2024 , howpublished =

    IFEval-Like Data , author =. 2024 , howpublished =

    2024

  36. [36]

    Evaluating Large Language Models Trained on Code , journal =

    Mark Chen and Jerry Tworek and Heewoo Jun and Qiming Yuan and Henrique Pond. Evaluating Large Language Models Trained on Code , journal =. 2021 , url =. 2107.03374 , timestamp =

    Pith/arXiv arXiv 2021

  37. [37]

    Nye and Maarten Bosma and Henryk Michalewski and David Dohan and Ellen Jiang and Carrie J

    Jacob Austin and Augustus Odena and Maxwell I. Nye and Maarten Bosma and Henryk Michalewski and David Dohan and Ellen Jiang and Carrie J. Cai and Michael Terry and Quoc V. Le and Charles Sutton , title =. CoRR , volume =. 2021 , url =. 2108.07732 , timestamp =

    Pith/arXiv arXiv 2021

  38. [38]

    CoRR , volume =

    Jeffrey Zhou and Tianjian Lu and Swaroop Mishra and Siddhartha Brahma and Sujoy Basu and Yi Luan and Denny Zhou and Le Hou , title =. CoRR , volume =. 2023 , url =. doi:10.48550/ARXIV.2311.07911 , eprinttype =. 2311.07911 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2311.07911 2023

  39. [39]

    CoRR , volume =

    An Yang and Baosong Yang and Beichen Zhang and Binyuan Hui and Bo Zheng and Bowen Yu and Chengyuan Li and Dayiheng Liu and Fei Huang and Haoran Wei and Huan Lin and Jian Yang and Jianhong Tu and Jianwei Zhang and Jianxin Yang and Jiaxi Yang and Jingren Zhou and Junyang Lin and Kai Dang and Keming Lu and Keqin Bao and Kexin Yang and Le Yu and Mei Li and Mi...

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.15115 2024

  40. [40]

    Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets , booktitle =

    Zhang. Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets , booktitle =. 2023 , url =

    2023

  41. [41]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Class-balanced loss based on effective number of samples , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  42. [42]

    Nature , volume=

    DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning , author=. Nature , volume=. 2025 , publisher=

    2025

  43. [43]

    The Landscape of Agentic Reinforcement Learning for

    Guibin Zhang and Hejia Geng and Xiaohang Yu and Zhenfei Yin and Zaibin Zhang and Zelin Tan and Heng Zhou and Zhong-Zhi Li and Xiangyuan Xue and Yijiang Li and Yifan Zhou and Yang Chen and Chen Zhang and Yutao Fan and Zihu Wang and Songtao Huang and Francisco Piedrahita Velez and Yue Liao and Hongru WANG and Mengyue Yang and Heng Ji and Jun Wang and Shuich...

    2026

  44. [44]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  45. [45]

    Proceedings of the Twentieth European Conference on Computer Systems , pages=

    Hybridflow: A flexible and efficient rlhf framework , author=. Proceedings of the Twentieth European Conference on Computer Systems , pages=

  46. [46]

    arXiv preprint arXiv:2405.11143 , volume=

    Openrlhf: An easy-to-use, scalable and high-performance rlhf framework , author=. arXiv preprint arXiv:2405.11143 , volume=

    Pith/arXiv arXiv

  47. [47]

    Proceedings of the Seventeenth International Conference on Machine Learning , pages=

    Algorithms for Inverse Reinforcement Learning , author=. Proceedings of the Seventeenth International Conference on Machine Learning , pages=

  48. [48]

    arXiv preprint arXiv:2606.30406 , year=

    MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training , author=. arXiv preprint arXiv:2606.30406 , year=

    Pith/arXiv arXiv

  49. [49]

    Machine learning proceedings 1995 , pages=

    Residual algorithms: Reinforcement learning with function approximation , author=. Machine learning proceedings 1995 , pages=. 1995 , publisher=

    1995

  50. [50]

    The 2010 international joint conference on neural networks (IJCNN) , pages=

    Deep auto-encoder neural networks in reinforcement learning , author=. The 2010 international joint conference on neural networks (IJCNN) , pages=. 2010 , organization=

    2010

  51. [51]

    nature , volume=

    Human-level control through deep reinforcement learning , author=. nature , volume=. 2015 , publisher=

    2015

  52. [52]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    On the efficacy of knowledge distillation , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

  53. [53]

    arXiv preprint arXiv:2505.22257 , year=

    Revisiting group relative policy optimization: Insights into on-policy and off-policy training , author=. arXiv preprint arXiv:2505.22257 , year=

    Pith/arXiv arXiv

  54. [54]

    Advances in Neural Information Processing Systems , volume=

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? , author=. Advances in Neural Information Processing Systems , volume=

  55. [55]

    arXiv preprint arXiv:2602.19362 , year=

    LLMs can learn to reason via off-policy RL , author=. arXiv preprint arXiv:2602.19362 , year=

    arXiv

  56. [56]

    arXiv preprint arXiv:2604.08706 , year=

    Efficient RL Training for LLMs with Experience Replay , author=. arXiv preprint arXiv:2604.08706 , year=

    Pith/arXiv arXiv