Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Supervised fine-tuning on curated data is a lower bound on reinforcement learning, and an importance-weighted variant tightens the bound and improves reasoning and control.

desk verdict A clean but unsurprising theoretical framing, an empirically promising but weakly validated iw-SFT variant, and a real mismatch between the theory's assumption and the experiments' reference model. read the letter →

arxiv 2507.12856 v2 pith:J4ZCC52W submitted 2025-07-17 cs.LG cs.AI

Chongli Qin , Jost Tobias Springenberg This is my paper
classification cs.LGcs.AI
keywords supervisedfine-tuningimportanceweightingreinforcementlearningasinferencesparserewardsqualityscoreddatareasoningLLMsofflinebehaviorcloning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Behavior cloning or supervised fine-tuning on a filtered, high-quality dataset is usually treated as a poor man's substitute for reinforcement learning. This paper argues the opposite: SFT on curated data is a maximum-likelihood estimator of a lower bound on an RL objective, in the special case where the reward is a binary success signal. That viewpoint reveals one free knob: reweight each training trajectory by the likelihood ratio between an auxiliary distribution and the reference policy that produced the data. The paper shows this importance-weighted variant (iw-SFT) optimizes a tighter bound that becomes exact as the auxiliary distribution approaches the trained policy, and that it can recover information lost when filtering discards failures. In experiments, iw-SFT on a 1,000-example curated math-reasoning set reaches 66.7% on AIME 2024 and 64.1% on GPQA Diamond, above SFT on the same data, and the weighted variants are competitive with offline RL algorithms on continuous control. If the argument holds, it gives a practical and theoretically grounded way to improve LLM reasoning and control policies without running an explicit RL loop.

What carries the argument

The load-bearing object is the lower-bound inequality applied to importance-sampling ratios: for $x=p(\tau;\theta)/q(\tau)$, the bound $x \geq 1+\log x$ converts the RL return into a weighted log-likelihood surrogate. The paper introduces an auxiliary distribution $q(\tau)$, typically a time-lagged copy of the policy being trained, so the importance weight $q(\tau)/\pi_{\mathrm{ref}}(\tau)$ adapts as training proceeds. Variance is controlled by clipping per-token log-ratios or smoothing trajectory-level weights with a temperature, and the same bound extends to ordinal quality scores by treating each quality threshold as a binary reward. This machinery is what turns a maximum-likelihood loss into a trainable approximation of the RL objective whose gap closes as $q\to p(\tau;\theta)$.

What would settle it

Compute the importance-weighted surrogate and the true RL objective exactly on a small finite MDP, sweeping $q$ from $\pi_{\mathrm{ref}}$ toward the trained policy; the claimed bound requires the gap $J(\theta)-\tilde{J}(\theta)$ to decrease as $q\to p(\tau;\theta)$, and if it does not, the theoretical mechanism fails. Separately, if the AIME and GPQA gains persist when the reference model is replaced by a deliberately misspecified one, the practical improvement is not explained by the bound itself.

Watch

Extended reading notes

Core claim

The paper's central claim is that supervised fine-tuning on filtered data is not merely a heuristic approximation to RL; in a sparse-reward setting with binary success indicators it maximizes a lower bound on the RL objective. Writing the RL return with importance sampling from a reference policy $\pi_{\mathrm{ref}}$ and applying $x \geq 1 + \log x$ to the likelihood ratio yields $J(\theta) \geq \mathbb{E}_{\pi_{\mathrm{ref}}}\![\mathbf{1}(S(\tau)>0)\log p(\tau;\theta)]$, which is exactly maximum likelihood on the filtered dataset. Introducing an auxiliary distribution $q(\tau)$ and weighting each filtered example by $q(\tau)/\pi_{\mathrm{ref}}(\tau)$ gives a bound that tightens as $q$ approaches the trained policy, and the paper constructs this as the iw-SFT objective. On the same curated data, iw-SFT outperforms SFT on math-reasoning benchmarks, needs no test-time budget forcing, and the quality-scored variants SFT$(Q)$ and iw-SFT$(Q)$ match or beat behavior-cloning baselines in offline control.

Load-bearing premise

The derivation requires the curated examples to have been generated by the reference policy used to form the importance weights; in the LLM experiments that policy is an unavailable model, so a different open model stands in for it, and if that stand-in is not close to the true generator the weights are miscalibrated and the tightened-bound conclusion does not follow.

Editorial extensions

If this is right

  • On the 1,000-example curated math dataset used in the paper, iw-SFT scores 66.7% on AIME 2024 and 64.1% on GPQA Diamond, beating SFT on the same data and matching the comparison baseline that relies on test-time budget forcing, without needing that forcing at all.
  • Because iw-SFT reweights rather than resamples, it can recover information from discarded failures: in the paper's two-arm bandit example, SFT saturates at 5/6 average reward while iw-SFT reaches the optimal policy.
  • With quality scores available, SFT$(Q)$ and iw-SFT$(Q)$ provide a tighter bound than plain threshold filtering and are competitive with established offline RL algorithms on D4RL locomotion and Franka Kitchen tasks.
  • The derivation implies that any curated dataset carries an implicit reward function, so the choice of curation filter determines the RL objective being bounded.
  • Since the bound tightens as $q$ approaches the trained policy, periodic or exponential-average updates of $q$ give a principled schedule for adaptive filtering during fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reference-policy approximation used in the LLM experiments is adequate, the same importance-weighting trick should transfer to other domains where curated traces are generated by a known or learnable policy, such as code generation or tool use, without needing reward-model RL.
  • The theory suggests a direct diagnostic for when iw-SFT will help: it should help most when the curated set contains heterogeneous success probabilities, because the weighting corrects for over- or under-sampling by the reference, and help least when the data is already drawn from near-optimal trajectories.
  • One testable extension is to estimate $\pi_{\mathrm{ref}}$ explicitly from the unfiltered pool before curation, rather than approximating it with the initial model; if the bound argument is right, using the true sampling distribution should improve or at least stabilize the weighted objective.
  • The clipping and temperature parameters act as a trust region around the reference policy; varying them should interpolate between plain SFT and a high-variance policy-gradient-like update, which could be used to control how far the final policy departs from the data distribution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper argues that supervised fine-tuning (SFT) on curated/filtered data can be understood as maximizing a lower bound on a sparse-reward reinforcement learning (RL) objective. Building on existing RL-as-inference bounds, the authors introduce importance-weighted SFT (iw-SFT), which reweights curated examples by q(τ)/π_ref(τ) with an auxiliary distribution q, and claim it optimizes a tighter bound that asymptotically approaches the RL objective. They support this with a toy bandit example, LLM reasoning experiments on the s1.1K dataset (AIME 2024 66.7%, GPQA 64.1%), and D4RL continuous-control experiments, reporting competitive or improved results over SFT and several offline RL baselines.

Significance. If the theoretical claim and experimental results hold, the paper offers a conceptually useful unification of SFT and RL and a simple, practical modification that could improve reasoning-model training without explicit RL. Strengths include a clear derivation of the bound in Section 3 and Appendix A, a toy example that illustrates a real failure mode of filtered-data SFT, open code and checkpoints, and experiments across two very different domains. However, the central theoretical guarantee depends on the curated data being generated by the reference policy π_ref, and in the main LLM experiments this assumption is not met and is not adequately validated. The significance is therefore conditional: the empirical gains in Table 1 may reflect a mismatched-reference artifact rather than the claimed bound-tightening mechanism, and the control experiments use per-task hyperparameter tuning that weakens the simplicity claim.

major comments (5)
  1. [Section 4.1 and Appendix C] Equation (8) is derived under the assumption that the curated dataset D+ is sampled from π_ref. However, the s1.1K traces were generated by Gemini Flash, while π_ref is approximated by Qwen2.5-32B-Instruct. The implemented objective therefore averages (q/π_ref) log p(τ;θ) over a different distribution μ (Gemini), not over π_ref. Consequently, the expression is not a lower bound on J(θ), and even as q→p(τ;θ) the weights approach p/π_ref rather than p/μ, leaving a bias proportional to μ/π_ref. The validation in Appendix C — comparing summed log-likelihoods of several 32B models — does not establish the per-token conditional calibration that product importance weights require. The paper should either use a reference model that actually generated the data, provide direct evidence that μ/π_ref is close to 1 on the relevant tokens, or re-derive the bound with the mismatch made explicit.
  2. [Section 4.2 and Appendix E/F] The same reference-distribution issue affects the control experiments. The D4RL trajectories were generated by behavior policies that are not known in closed form; the paper instead uses a BC policy trained on all data as π_ref. If this BC approximation is imperfect, the importance weights q/π_ref are miscalibrated and the lower-bound argument in Eq. (8) does not strictly apply. The paper should quantify the discrepancy between the fitted π_ref and the true behavior policy (e.g., by reporting held-out log-likelihood or effective sample size of the importance weights), or argue why the bias is negligible for these datasets.
  3. [Section 3 and Section 3.3] The statement that iw-SFT 'asymptotically maximizes the RL objective' is not supported by a proof. Equation (8) only shows that the pointwise bound becomes equality when q(τ)=p(τ;θ); it does not establish that the iterative optimization converges to the maximizer of J(θ). Moreover, the practical implementation clips or smooths the importance weights (Section 3.4), so q never exactly equals p(τ;θ). The asymptotic claim should be softened or replaced with a convergence statement under precise conditions.
  4. [Table 1] The main LLM results are reported from single training runs with no error bars or multiple seeds. The AIME 2024 gap between iw-SFT (66.7) and s1.1K (56.7) is large, but without run-to-run variance or at least a confidence interval, the improvement cannot be fully assessed, especially given the known sensitivity of reasoning benchmarks to training noise. The paper should report multiple seeds or provide some other measure of variability.
  5. [Table 4 in Appendix F] The D4RL hyperparameters are tuned per environment: learning rate, total steps, EMA coefficient α, the smoothing constant k/|τ|, percentile cutoffs, and whether weights are normalized all differ across halfcheetah, hopper, and walker2d. This is a substantial amount of per-task tuning and weakens the claim that iw-SFT is a simple, generally applicable modification. The authors should either use a shared hyperparameter setting, or report sensitivity analyses showing that performance degrades gracefully when individual hyperparameters are fixed.
minor comments (6)
  1. [Throughout] There are several typos and grammatical errors, including 'exeplified' (Section 1), 'obective' (Section 2), 'wok' (Section 5), and 'additoinal' (Appendix G). A careful proofreading pass is needed.
  2. [Section 3.4] The sentence 'choosing q(τ) such that it strikes this trade-off ... will not bias our algorithm' is confusing. Clipping or smoothing the importance weights changes the objective being optimized, so it does bias the estimate of J(θ); the intended meaning may be that it does not break the lower-bound property. Please clarify.
  3. [Algorithm 1 in Appendix B] The gradient update line contains a typo: '∆θ ← ∂/∂θ Pib j=i1 wj log f (τ j; θ)' should use a sum over the batch with a proper index (e.g., Σ_{j=1}^b), and 'idenotes' should be 'i denotes'.
  4. [Appendix G] The note that Figure 2 was created with an earlier training run with slightly different settings, and not re-run for the latest model, should be stated more prominently in the main text, as it limits the comparability of the budget-forcing ablation.
  5. [Table 4] There are formatting issues in Table 4: 'k/|τ|' is not defined in the table caption, and some rows contain trailing commas (e.g., '[90, 95, 98],'). Please clean up the table and define all column headers.
  6. [Section 4.2] The text says 'we omit standard deviations for space but mark statistically equivalent results in bold (see supplementary).' In Table 2, no bold marking is actually shown, and the supplementary Table 5 lists standard deviations only for the authors' methods. Please either implement the bold marking or revise the statement to match what is presented.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the SFT/RL bound derivation is algebraic and benchmark evaluations are external.

full rationale

The derivation chain is self-contained. Equations 3-8 apply the standard inequality x >= 1 + log x to a known importance-sampling identity, with no parameter fitted to the target benchmark. The SFT lower bound (Eq. 5) and iw-SFT bound (Eq. 8) follow algebraically from the definitions, and the claim that the bound tightens as q approaches p is a mathematical property of the auxiliary distribution, not an input that is renamed as an output. The empirical results are independent benchmark scores, and the paper credits the bound to prior work (Roux, Peters, Kober, Dayan, etc.) rather than to its own authors. The notable gap is the approximation of pi_ref by Qwen2.5-32B-Instruct for Gemini-generated traces (Section 4.1, Appendix C), which is an assumption violation that could invalidate the bound's applicability, but it is not a circular reduction: the implemented objective is not defined in terms of the claimed outcome. Similarly, the smoothed/clipped importance weights (Section 3.4) may break the lower-bound property in practice, but that is a correctness or robustness issue, not a definitional equivalence. No step of the paper's derivation reduces to its own input by construction, and no load-bearing self-citation chain is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard inequalities, the assumption that the curated data is drawn from the reference policy, and several hand-chosen constants (clipping thresholds, temperatures, percentile cutoffs). The latter are tuned per task, so the method as evaluated includes free parameters.

free parameters (4)
  • importance weight clipping thresholds = alpha_min=0.2, alpha_max=1.8 (LLM); per-step clip before trajectory-level clip
    Chosen without a reported sweep; affect the effective bound and could inflate or deflate performance.
  • temperature k in g(x) = 0.1 for LLM; various per D4RL task (0.8, 1, 10)
    Controls sharpness of importance weights; tuned per environment in Table 4.
  • EMA coefficient alpha for q update = 0.9 to 0.995 per task
    Tuned per D4RL environment; affects how quickly q tracks theta and hence the tightness of the bound.
  • D4RL quality percentile cutoffs = [90, 95, 98]
    Data selection thresholds for quality sampling; chosen by the authors, not from theory.
assumptions (4)
  • standard math x >= 1 + log x for x > 0
    Used in Eq. (4) and Eq. (18) to derive lower bounds.
  • domain assumption Curated data is sampled from the reference policy pi_ref
    Importance sampling identity in Eq. (3) requires trajectories drawn from pi_ref; the LLM experiments violate this because data was generated by Gemini Flash, not Qwen.
  • domain assumption Support of p(pi) is a subset of support of pi_ref
    Stated in Section 3.1; required for importance weights to be finite.
  • ad hoc to paper q is chosen to track p(theta) without blowing up variance
    The method's effectiveness depends on this heuristic; no convergence proof is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)." pith.science (2026). https://pith.science/paper/J4ZCC52W

@misc{pith2026250712856,
  author       = {Pith},
  title        = {Pith review of: Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J4ZCC52W}},
  note         = {Machine review of arXiv:2507.12856}
}
read the original abstract

Behavior Cloning (BC) on curated (or filtered) data is the predominant paradigm for supervised fine-tuning (SFT) of large language models; as well as for imitation learning of control policies. Here, we draw on a connection between this successful strategy and the theory and practice of finding optimal policies via Reinforcement Learning (RL). Building on existing literature, we clarify that SFT can be understood as maximizing a lower bound on the RL objective in a sparse reward setting. Giving support to its often observed good performance. From this viewpoint, we realize that a small modification to SFT leads to an importance weighted variant that behaves closer to training with RL as it: i) optimizes a tighter bound to the RL objective and, ii) can improve performance compared to SFT on curated data. We refer to this variant as importance weighted supervised fine-tuning (iw-SFT). We show that it is easy to implement and can be further generalized to training with quality scored data. The resulting SFT variants are competitive with more advanced RL algorithms for large language models and for training policies in continuous control tasks. For example achieving 66.7% on the AIME 2024 dataset.

Figures

Figures reproduced from arXiv: 2507.12856 by the authors.

Figure 1
Figure 1. Toy example showing a fail￾ure mode of SFT on curated data. Thus the optimal strategy is to always pull the right arm. We assume that we can observe the behavior of the system only by watching (collecting data from) a reference policy πref(a) = U{pull-left, pull-right} that picks between the two arms uniformly at random. We collect a dataset D using this policy which will contain 50% pull-left and 50% pull-right act… view at source ↗
Figure 2
Figure 2. iw-SFT shown in blue compared to budged forcing results from Muennighoff et al. [2025] [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Few Teacher Steps Go a Long Way: Cost-Efficient On-Policy Data Augmentation for Agent Post-Training

    cs.LG 2026-07 conditional novelty 5.5 of 10

    A few unfiltered teacher turns at learner-induced contexts are a more cost-efficient SFT allocation than longer or success-filtered teacher completions for LLM agents.

Reference graph

Works this paper leans on

63 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [1]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  2. [2]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    arXiv 1909

  3. [3]

    Rlaif vs

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback, 2024. URL https://arxiv.org/abs/2309.00267

    arXiv 2024

  4. [4]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    arXiv 2022

  5. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  6. [6]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    arXiv 2024

  7. [7]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms

    Arash Ahmadian, Chris Cremer, Matthias Gall \'e , Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet \"U st \"u n, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024

    arXiv 2024

  8. [8]

    Remax: A simple, effective, and efficient method for aligning large language models

    Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient method for aligning large language models. arXiv preprint arXiv:2310.10505, 2023

    arXiv 2023

Show all 63 references
  1. [9]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  2. [10]

    Preference optimization as probabilistic inference

    Abbas Abdolmaleki, Bilal Piot, Bobak Shahriari, Jost Tobias Springenberg, Tim Hertweck, Rishabh Joshi, Junhyuk Oh, Michael Bloesch, Thomas Lampe, Nicolas Heess, et al. Preference optimization as probabilistic inference. arXiv preprint arXiv:2410.04166, 2024

    arXiv

  3. [11]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  4. [12]

    Reinforcement learning for reasoning in large language models with one training example, 2025

    Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and Yelong Shen. Reinforcement learning for reasoning in large language models with one training example, 202...

    2025 arXiv

  5. [13]

    Peters and S

    J. Peters and S. Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th Annual International Conference on Machine Learning, pages 745--750, 2007. URL http://www-clmc.usc.edu/publications//P/peters_ICML2007.pdf. clmc

    2007

  6. [14]

    Policy search for motor primitives in robotics

    Jens Kober and Jan Peters. Policy search for motor primitives in robotics. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2008. URL https://proceedings.neurips.cc/paper_file...

    2008

  7. [15]

    Reinforcement learning and control as probabilistic inference: Tutorial and review

    Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. CoRR, abs/1805.00909, 2018. URL http://arxiv.org/abs/1805.00909

    2018 arXiv

  8. [16]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  9. [17]

    D4rl: Datasets for deep data-driven reinforcement learning, 2020

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020

    2020

  10. [18]

    Peter Dayan and Geoffrey E. Hinton. Using expectation-maximization for reinforcement learning. Neural Computation, 9 0 (2): 0 271--278, 1997. doi:10.1162/neco.1997.9.2.271

    1997 doi

  11. [19]

    Learning math reasoning from self-sampled correct and partially-correct solutions

    Ansong Ni, Jeevana Priya Inala, Chenglong Wang, Alex Polozov, Christopher Meek, Dragomir Radev, and Jianfeng Gao. Learning math reasoning from self-sampled correct and partially-correct solutions. In ICLR, 2023. URL https://openreview.net/forum?id=4D4TSJE6-K

    2023

  12. [20]

    Learning to generalize from sparse and underspecified rewards

    Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. Learning to generalize from sparse and underspecified rewards. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Procee...

    2019

  13. [21]

    Reward augmented maximum likelihood for neural structured prediction

    Mohammad Norouzi, Samy Bengio, zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans. Reward augmented maximum likelihood for neural structured prediction. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Informat...

    2016

  14. [22]

    Scaling relationship on learning mathematical reasoning with large language models, 2024

    Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models, 2024. URL https://openreview.net/forum?id=cijO0f8u35

    2024

  15. [23]

    Simplify rlhf as reward-weighted sft: A variational method, 2025

    Yuhao Du, Zhuo Li, Pengyu Cheng, Zhihong Chen, Yuejiao Xie, Xiang Wan, and Anningzhe Gao. Simplify rlhf as reward-weighted sft: A variational method, 2025. URL https://arxiv.org/abs/2502.11026

    2025

  16. [24]

    Reinforced self-training (rest) for language modeling, 2023

    Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. Reinforced self-training (rest) for language mode...

    2023 arXiv

  17. [25]

    Beyond human data: Scaling self-training for problem-solving with language models

    Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T Parisi, Abhishek Kumar, Alexander A Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Hanie ...

    2024

  18. [26]

    Peters, K

    J. Peters, K. Muelling, and Y. Altun. Relative entropy policy search. In Proceedings of 24th AAAI Conference on Artificial Intelligence (AAAI '10), pages 1607 -- 1612, July 2010

    2010

  19. [27]

    Efficient iterative policy optimization

    Nicolas Le Roux. Efficient iterative policy optimization. CoRR, abs/1612.08967, 2016 a . URL http://arxiv.org/abs/1612.08967

    2016 arXiv

  20. [28]

    Tighter bounds lead to improved classifiers

    Nicolas Le Roux. Tighter bounds lead to improved classifiers. arXiv preprint arXiv:1606.09202, 2016 b

    2016 arXiv

  21. [29]

    Process for adapting language models to society (palms) with values-targeted datasets

    Irene Solaiman and Christy Dennison. Process for adapting language models to society (palms) with values-targeted datasets. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pag...

    2021

  22. [30]

    Self-consuming generative models with curated data provably optimize human preferences

    Damien Ferbach, Quentin Bertrand, Joey Bose, and Gauthier Gidel. Self-consuming generative models with curated data provably optimize human preferences. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=c...

    2024

  23. [31]

    Maximum a posteriori policy optimisation

    Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=S1ANxQW0b

    2018

  24. [32]

    Abbas Abdolmaleki, Sandy H. Huang, Giulia Vezzani, Bobak Shahriari, Jost Tobias Springenberg, Shruti Mishra, Dhruva TB, Arunkumar Byravan, Konstantinos Bousmalis, Andr \' a s Gy \" o rgy, Csaba Szepesv \' a ri, Raia Hadsell, Nicolas Heess, and Martin A. Riedmiller. On multi-ob...

    2021 arXiv

  25. [33]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. CoRR, abs/1910.00177, 2019. URL https://arxiv.org/abs/1910.00177

    1910 arXiv

  26. [35]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/1707.06347

    2017 arXiv

  27. [36]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021

    2021 arXiv

  28. [37]

    When does return-conditioned supervised learning work for offline reinforcement learning? In S

    David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-conditioned supervised learning work for offline reinforcement learning? In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Info...

    2022

  29. [38]

    Decision transformer: Reinforcement learning via sequence modeling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaugha...

    2021

  30. [39]

    Kahn and Andrew W

    H. Kahn and Andrew W. Marshall. Methods of reducing sample size in monte carlo computations. Oper. Res., 1: 0 263--278, 1953. URL https://api.semanticscholar.org/CorpusID:21103559

    1953

  31. [40]

    Rubinstein and Dirk P

    Reuven Y. Rubinstein and Dirk P. Kroese. Simulation and the Monte Carlo Method. Wiley Publishing, 3rd edition, 2016. ISBN 1118632168

    2016

  32. [41]

    Stochastic simulation

    Brian D Ripley. Stochastic simulation. John Wiley & Sons, 2009

    2009

  33. [42]

    Doubly robust off-policy value evaluation for reinforcement learning

    Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research...

    2016

  34. [43]

    Policy optimization via importance sampling

    Alberto Maria Metelli, Matteo Papini, Francesco Faccio, and Marcello Restelli. Policy optimization via importance sampling. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume ...

    2018

  35. [44]

    Andrad\'ottir, D

    S. Andrad\'ottir, D. P. Heyman, and T. J. Ott. On the choice of alternative measures in importance sampling with M arkov chains. Operations Research, 43 0 (3): 0 509--519, 1995

    1995

  36. [45]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  37. [46]

    Numinamath, 2024

    Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath, 2024. URL https://github.com/project-num...

    2024

  38. [47]

    Aime, February 2024

    Mathematical Association of America. Aime, February 2024. URL https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions/

    2024

  39. [48]

    Omni-math: A universal olympiad level mathematic benchmark for large language models, 2024

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni-math: A universal ...

    2024 arXiv

  40. [49]

    Agieval: A human-centric benchmark for evaluating foundation models, 2023

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023. URL https://arxiv.org/abs/2304.06364

    2023 arXiv

  41. [50]

    Gemini 2.0 flash thinking mode (gemini-2.0-flash-thinking-exp-1219), December 2024

    Google. Gemini 2.0 flash thinking mode (gemini-2.0-flash-thinking-exp-1219), December 2024. URL https://cloud.google.com/vertex-ai/generative-ai/docs/thinking-mode

    2024

  42. [51]

    Qwq: Reflect deeply on the boundaries of the unknown, November 2024

    Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URL https://qwenlm.github.io/blog/qwq-32b-preview/

    2024

  43. [52]

    Learning to reason with llms, September 2024

    OpenAI. Learning to reason with llms, September 2024. URL https://openai.com/index/learning-to-reason-with-llms/

    2024

  44. [53]

    Bespoke-stratos: The unreasonable effectiveness of reasoning distillation, 2025

    Bespoke Labs. Bespoke-stratos: The unreasonable effectiveness of reasoning distillation, 2025. URL https://hf.co/bespokelabs/Bespoke-Stratos-32B. Accessed: 2025-01-22

    2025

  45. [54]

    Sky-t1: Fully open-source reasoning model with o1-preview performance in \ 450 budget, 2025

    NovaSky Team. Sky-t1: Fully open-source reasoning model with o1-preview performance in \ 450 budget, 2025. URL https://novasky-ai.github.io/posts/sky-t1. Accessed: 2025-01-09

    2025

  46. [55]

    Gemini 2.5: Our most intelligent ai model, March 2025

    Google. Gemini 2.5: Our most intelligent ai model, March 2025. URL https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025

    2025

  47. [56]

    Fixing weight decay regularization in adam

    Ilya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 5: 0 5, 2017

    2017 arXiv

  48. [57]

    Continuous control with deep reinforcement learning

    Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  49. [58]

    A minimalist approach to offline reinforcement learning

    Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34: 0 20132--20145, 2021

    2021

  50. [59]

    Awac: Accelerating online reinforcement learning with offline datasets

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020 b

    2006 arXiv

  51. [60]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33: 0 1179--1191, 2020

    2020

  52. [61]

    Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K \" o pf, Edward Z. Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner...

    1912 arXiv

  53. [62]

    Accelerate: Training and inference at scale made simple, efficient and adaptable

    Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable. https://github.com/huggingface/accelerate, 2022

    2022

  54. [63]

    SGDR : Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. SGDR : Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=Skq89Scxx

    2017

  55. [64]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv....

    2015 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.