Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

Bridging Offline and Online Reinforcement Learning for LLMs

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that the gap between offline and online reinforcement learning for LLMs is mostly set by how often the response generator is synchronized with the trainer, and that semi-online DPO—syncing every 5 to 100 steps—recovers…

desk verdict Useful, well-run empirical paper: the offline-vs-online gap is real, and the semi-online finding is worth taking seriously, but the non-verifiable checkpoint selection deserves a robustness check before the headline is fully trusted. read the letter →

arxiv 2506.21495 v1 pith:SO4VX2BG submitted 2025-06-26 cs.CL

Jack Lanchantin , Angelica Chen , Janice Lan , Xian Li , Swarnadeep Saha , Tianlu Wang , Jing Xu , Ping Yu
show 4 more authors
Weizhe Yuan Jason E Weston Sainbayar Sukhbaatar Ilia Kulikov
This is my paper
classification cs.CL
keywords offline-to-onlinereinforcementlearningdirectpreferenceoptimizationgrouprelativepolicysemi-onlinetrainingLLMpost-trainingverifiablerewardsrewardmodelsinstructionfollowing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to determine how much of the benefit of online reinforcement learning for large language models survives when training is moved offline or semi-online, and whether the choice of objective (DPO versus GRPO) matters. Its answer, across both verifiable math and non-verifiable instruction following, is that the online/offline distinction matters much more than the objective: fully online DPO and GRPO perform comparably and both strongly beat offline DPO, while semi-online DPO—in which the generator model is synchronized with the trainer only every 5 to 100 steps—nearly closes the gap. A practitioner who wants most of the online gain without paying for synchronous rollouts at every step can therefore use semi-online DPO. The paper also claims that training on verifiable and non-verifiable rewards jointly improves both task types compared with training on either alone.

What carries the argument

The paper's central object is the synchronization step $s$: the number of parameter updates the trainer performs before copying its weights to the generator that produces rollouts. $s=\infty$ is offline DPO (pre-generated responses), $s=1$ is fully online training (generator always current), and intermediate values are semi-online. Varying $s$ isolates how much of the gain of online RL comes from training on the current policy's own responses; the paper's result is that periodic synchronization, not per-step synchronization, carries almost all of the benefit. A second mechanism is the reference model in DPO: syncing the reference model along with the generator prevents the response-length collapse seen in verifiable-task runs, and the paper treats this as a stability requirement rather than a performance booster.

What would settle it

Re-run the offline, semi-online, and online DPO recipes but select checkpoints by held-out benchmark scores (math accuracy and instruction-following winrates) instead of the length-normalized reward score; if offline DPO then matches or beats the online variants, the reported gap is a checkpoint-selection artifact rather than a training-regime effect.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the synchronization interval between the model generating responses and the model being trained—not the choice between DPO and GRPO—is the main driver of post-training quality. With an 8-billion-parameter instruct-tuned seed, offline DPO improves over the seed but is left far behind by online DPO, online GRPO, and semi-online DPO; semi-online DPO with sync intervals of 10 or 100 steps on math and 5 or 10 steps on instruction following lands close to fully online DPO. The paper reports similar convergence and performance for the online and semi-online variants, finds no benefit from larger GRPO groups beyond eight responses, and shows that combining verifiable rule-based rewards with reward-model scores in a single online DPO run improves non-verifiable benchmarks relative to training on non-verifiable rewards alone.

Load-bearing premise

The load-bearing premise is that the automated reward score used to choose checkpoints faithfully measures response quality; the paper itself notes that this reward model has a length bias, so if the score rewards length rather than quality, the reported gaps between training regimes could be an artifact of checkpoint selection.

Editorial extensions

If this is right

  • Semi-online DPO with $s=10$ or $s=100$ on math and $s=5$ or $s=10$ on instruction following can substitute for fully online DPO, retaining most of the benchmark gain while allowing rollouts to be generated in parallel between syncs.
  • Online DPO is a practical alternative to GRPO for verifiable math: it needs only one preference pair per prompt per step, matches GRPO's accuracy, and avoids the need for a group of responses.
  • Offline DPO should not be treated as equivalent to online preference optimization for post-training LLMs; the gap on math and instruction-following benchmarks is large and consistent.
  • A single model can be trained on verifiable and non-verifiable rewards together, and the combination improves non-verifiable evaluations over non-verifiable-only training, even though one-task-only checkpoints show little cross-task transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reward score used for checkpoint selection is biased toward length, the absolute sizes of the reported gaps between online/semi-online and offline DPO may shrink under a length-agnostic judge; the qualitative ranking could survive, but the semi-online bridge might be narrower than it appears.
  • A natural extension is to treat $s$ as a tunable compute-quality knob: for a fixed inference budget, the optimal synchronization interval may depend on rollout cost and dataset epoch size, and the paper's setup makes that measurement straightforward.
  • The entropy-collapse observation suggests that stability, not objective choice, will be the binding constraint as online methods scale; testing adaptive synchronization or entropy regularization as a function of rollout entropy is a direct follow-up.
  • The multi-task result hints that mixing a verifiable signal into general instruction-following data may substitute for part of the reward model's role; a testable consequence is that the same benefit should appear with other verifiers such as code execution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper empirically compares offline, semi-online, and fully online post-training algorithms (DPO and GRPO) for LLMs on verifiable math and non-verifiable instruction-following tasks. Using Llama-3.1-8B-Instruct as the seed model, it reports that semi-online DPO with periodic generator synchronization (s=10 or 100 for math, s=5 or 10 for instruction following) performs comparably to fully online DPO (s=1) and GRPO, and that all of these substantially outperform offline DPO. It also studies reference-model synchronization, entropy collapse, and joint training with verifiable and non-verifiable rewards, reporting that multi-task training yields gains on non-verifiable benchmarks while roughly preserving verifiable performance.

Significance. If the central result holds, the paper provides a practically valuable finding: much of the benefit of fully online RL for LLM alignment can be captured by semi-online DPO at lower inference cost. The verifiable-task results are independently corroborated by checkpoint selection based on heldout math accuracy rather than reward-model scores, and the use of external benchmarks (Math500, NuminaMath, AMC23, AlpacaEval, ArenaHard) with reported standard errors is a strength. The paper also contains useful analysis of length collapse, reference-model sync, and entropy degeneration. However, the non-verifiable headline rests on a checkpoint-selection rule that shares its reward model with the training signal, and the verifiable comparison confounds online rollouts with reference-model synchronization, so the central claim is not yet fully established. The absence of code or data release and the use of a single base model further limit immediate reproducibility.

major comments (3)
  1. [Section 3.1, Section 4.2, Table 2] Checkpoint selection for non-verifiable tasks uses the highest length-normalized Athene-RM-8B reward on a heldout set, and the same Athene-RM-8B model supplies the training rewards. Section 4.2 acknowledges that Athene-RM-8B has a length bias and that response lengths increase in online/semi-online training. Although AlpacaEval LC and ArenaHard control for length, they do not control for overfitting to Athene-RM-8B's specific notion of quality. The claim that online and semi-online DPO substantially outperform offline DPO on non-verifiable tasks is load-bearing for the paper, so the selection rule could be inflating the gap. Please report results under a neutral selection rule (e.g., fixed training budget, or checkpoint selection by an independent judge such as GPT-4 on a heldout set), or at minimum report the best checkpoint per external benchmark. Without such a check, the non-verifiable headline remains conditional on the reward model used for both training and selection.
  2. [Section 4.2, Table 4, Figure 2] In the verifiable-task experiments, all online and semi-online DPO runs use reference-model synchronization, while offline DPO does not; Table 4 shows this difference, and Figure 2 shows that online DPO without reference sync suffers length collapse and lower validation reward. This confounds the comparison: the large verifiable-task advantage of online/semi-online DPO over offline DPO could be driven by reference-model sync rather than by fresh on-policy rollouts. The manuscript does not report benchmark scores for online DPO without reference sync on Math500/NuminaMath/AMC23. Please add those benchmark numbers, and ideally for semi-online DPO without sync as well, so the independent contributions of on-policy data and reference-model stabilization can be separated.
  3. [Abstract, Section 4.1, Table 3] The abstract states that joint training on verifiable and non-verifiable rewards yields 'improved performance across both task types,' but Table 3 shows that the NM+WC model has Math500 accuracy 57.3 versus 58.7 for NM-only training and AMC23 accuracy 31.7 versus 32.9 for NM-only training, i.e., numerically lower verifiable performance. The body text in Section 4.1 more carefully says the result is 'comparable' to individual-task training with 'slight improvements in the non-verifiable evaluations.' Please align the abstract's claim with the data, or clarify that the improvement is on non-verifiable benchmarks only.
minor comments (6)
  1. [Throughout] No code, training data, or model checkpoints are released; for an empirical methods paper with comparisons of this sort, a public repository would substantially improve reproducibility and should be provided.
  2. [Tables 1-2] The reported standard errors and confidence intervals appear to reflect evaluation-sample noise (e.g., N=50 generated solutions per problem) rather than variance across independent training runs; please state this explicitly so readers do not interpret the intervals as training-seed variance.
  3. [Section 4.1] The statement that 'we find s = 100 to be too unstable with our non-verifiable hyperparameters' is not accompanied by any plot, table, or numerical evidence; please provide the supporting results or label this as an informal observation.
  4. [Introduction] There is a typo in the second paragraph: 'it it was only relatively recently' should read 'it was only relatively recently.'
  5. [Section 4.2 and Figure 3] The text says 'GPRO' where 'GRPO' is intended; please correct this typo in the sentence 'in both DPO and GPRO experiments' and in the Figure 3 caption.
  6. [Conclusion] The final paragraph says 'transitioning from offline to offline learning'; this should read 'from offline to online learning.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical benchmark comparisons, and the shared Athene-RM-8B checkpoint-selection signal is a flagged limitation rather than a by-construction equivalence.

full rationale

This paper makes no first-principles derivation whose output is equivalent to its input; its central claims are comparative measurements of offline, semi-online, and online DPO/GRPO on external benchmarks. The only place where the same Athene-RM-8B model appears on both sides is checkpoint selection (Section 3.1: 'We select the best model checkpoint based on the highest length-normalized Athene-RM-8B rewards on a heldout set of 470 examples') and as the source of training rewards. The paper itself flags the associated risk in Section 4.2: 'the model tends to hack its length bias to maximize rewards... we choose the last option [length-normalized checkpoint selection] and find that this selection method generalizes well.' This is a genuine selection-bias concern, but it is not circularity: the headline offline-versus-online gap is corroborated on independent, length-controlled benchmarks (AlpacaEval LC and Arena-Hard) and on verifiable math tasks (Math500, NuminaMath, AMC23) where checkpoint selection is based on heldout accuracy rather than the Athene-RM-8B reward. No fitted parameter is renamed as a prediction, no load-bearing claim reduces to a self-citation, and no uniqueness or ansatz is imported from the authors' prior work. The paper is therefore self-contained as an empirical study, with the acknowledged reward-model selection risk belonging to the correctness/robustness discussion rather than to circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No fitted constants enter the training objectives; all free parameters are hand-chosen hyperparameters. The paper's contribution is empirical, so the main burden sits on the reward model used for training and checkpoint selection, and on the restriction of GRPO to one on-policy update.

free parameters (3)
  • Sync interval s = s in {5, 10, 100} for semi-online, s=1 for online, s=infinity for offline
    Chosen by hand as a sweep. The central comparison between semi-online and online performance depends on these values.
  • DPO KL beta coefficient = 0.1 verifiable, 0.01 non-verifiable, 0.001 for GRPO
    Hyperparameter chosen per task; not derived from theory. GRPO uses a smaller value.
  • Adam epsilon = 1e-4 verifiable, 1e-8 non-verifiable
    The paper recommends higher Adam epsilon for training stability. Values were tuned per task and affect the reported results.
assumptions (4)
  • domain assumption Direct Preference Optimization assumes the Bradley-Terry model of preferences and an implicit KL-regularized reward objective (Appendix A.1).
    Borrowed from Rafailov et al. 2023 and used as the training objective.
  • domain assumption GRPO's sequence-level advantage is only valid in a purely on-policy, single-update regime. The paper states the sequence-level advantage lacks an off-policy proof (Section 2.1).
    The authors restrict GRPO to one update per generation, so the GRPO comparison is conditional on this restriction.
  • domain assumption Athene-RM-8B scalar scores are valid proxies for human preference quality on WildChat prompts, and length-normalized validation reward is a valid checkpoint selection criterion (Sections 3.1, 4.2).
    Training and checkpoint selection both use this reward model. If it is biased, cross-method comparisons could be distorted.
  • ad hoc to paper Reference model synchronization with the generator is a necessary stabilization mechanism for online DPO (Table 4, Figure 2).
    This is an implementation choice treated as important for stability, not a proven general requirement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bridging Offline and Online Reinforcement Learning for LLMs." pith.science (2026). https://pith.science/paper/SO4VX2BG

@misc{pith2026250621495,
  author       = {Pith},
  title        = {Pith review of: Bridging Offline and Online Reinforcement Learning for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SO4VX2BG}},
  note         = {Machine review of arXiv:2506.21495}
}
read the original abstract

We investigate the effectiveness of reinforcement learning methods for finetuning large language models when transitioning from offline to semi-online to fully online regimes for both verifiable and non-verifiable tasks. Our experiments cover training on verifiable math as well as non-verifiable instruction following with a set of benchmark evaluations for both. Across these settings, we extensively compare online and semi-online Direct Preference Optimization and Group Reward Policy Optimization objectives, and surprisingly find similar performance and convergence between these variants, which all strongly outperform offline methods. We provide a detailed analysis of the training dynamics and hyperparameter selection strategies to achieve optimal results. Finally, we show that multi-tasking with verifiable and non-verifiable rewards jointly yields improved performance across both task types.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Safety Alignment of LMs via Non-cooperative Games

    cs.AI 2025-12 conditional novelty 7.0 of 10

    Jointly training an Attacker and Defender LLM in a non-zero-sum game with pairwise preference judges produces a defender with much lower jailbreak success while preserving general utility.

  2. Multi$^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Multi² decomposes LLM agent decision-making into high-level sub-goal generation with SFT and low-level action execution with RL, claims to outperform baselines across environments, and releases three hierarchical benc...

  3. Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation

    cs.CL 2025-10 reject novelty 6.0 of 10

    A turn-level faithfulness reward improves a Search-R1-style agent's Information-Think and Think-Answer faithfulness as judged by the same reward model used for training, while task accuracy is roughly unchanged.

Reference graph

Works this paper leans on

55 extracted references · 12 canonical work pages · cited by 3 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    fairseq2, 2023

    Can Balioglu, Martin Gleize, Artyom Kozhevnikov, Ilia Kulikov, Tuan Tran, and Julien Yao. fairseq2, 2023. URL http://github.com/facebookresearch/fairseq2

    work page 2023

  3. [3]

    Preference learning algorithms do not learn preference rankings

    Angelica Chen, Sadhika Malladi, Lily H Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, and Kyunghyun Cho. Preference learning algorithms do not learn preference rankings. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 a . URL https://openreview.net/forum?id=YkJ5BuEXdD

    work page 2024

  4. [4]

    Self-play fine-tuning converts weak language models to strong language models

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024 b

    arXiv 2024

  5. [5]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  6. [6]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Llama 3.1 Community License Agreement

    arXiv 2024

  7. [7]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators

    Yann Dubois, Bal \'a zs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024

    arXiv 2024

  8. [8]

    Athene-70b: Redefining the boundaries of post-training for open models, July 2024 a

    Evan Frick, Peter Jin, Tianle Li, Karthik Ganesan, Jian Zhang, Jiantao Jiao, and Banghua Zhu. Athene-70b: Redefining the boundaries of post-training for open models, July 2024 a . URL https://nexusflow.ai/blogs/athene

    work page 2024

Show all 55 references
  1. [9]

    Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E

    Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N. Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. How to evaluate reward models for rlhf, 2024 b . URL https://arxiv.org/abs/2410.14872

    2024 arXiv

  2. [10]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835--10866. PMLR, 2023

    2023

  3. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  4. [12]

    Direct language model alignment from online ai feedback

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024

    2024 arXiv

  5. [13]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. MIT license

    2021 arXiv

  6. [14]

    ORPO : Monolithic preference optimization without reference model

    Jiwoo Hong, Noah Lee, and James Thorne. ORPO : Monolithic preference optimization without reference model. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170--11189,...

    2024 doi

  7. [15]

    Lo RA : Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  8. [16]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems...

    2023

  9. [17]

    T " ulu 3: Pushing frontiers in open language model post-training

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T " ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  10. [18]

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face reposito...

    2024

  11. [19]

    From live data to high-quality benchmarks: The arena-hard pipeline, April 2024 b

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024 b . URL https://lmsys.org/blog/2024-04-19-arena-hard/. Apache-2.0 license

    2024

  12. [20]

    From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939, 2024 c

    2024 arXiv

  13. [21]

    Self-alignment with instruction backtranslation

    Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprint arXiv:2308.06259, 2023 a

    2023 arXiv

  14. [22]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 5 2023 b . Apache-2.0 license

    2023

  15. [23]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023

    2023 arXiv

  16. [24]

    Statistical rejection sampling improves preference optimization

    Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657, 2023

    2023 arXiv

  17. [25]

    Understanding r1-zero-like training: A critical perspective, 2025

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective, 2025

    2025

  18. [26]

    Mixtral of experts: A high quality sparse mixture-of-experts

    Mistral AI team, 2023. Mixtral of experts: A high quality sparse mixture-of-experts. https://mistral.ai/news/mixtral-of-experts/, 2023. Accessed: Dec 12th 2023

    2023

  19. [27]

    Ray: A distributed framework for emerging \ AI \ applications

    Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. Ray: A distributed framework for emerging \ AI \ applications. In 13th USENIX symposium on operating systems design a...

    2018

  20. [28]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  21. [29]

    West-of-n: Synthetic preference generation for improved reward modeling

    Aliz \'e e Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. West-of-n: Synthetic preference generation for improved reward modeling. arXiv preprint arXiv:2401.12086, 2024

    2024 arXiv

  22. [30]

    Iterative reasoning preference optimization

    Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems, 37: 0 116617--116637, 2024

    2024

  23. [31]

    Disentangling length from quality in direct preference optimization

    Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 4998--5017, B...

    2024 doi

  24. [32]

    Disentangling length from quality in direct preference optimization

    Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159, 2024 b

    2024 arXiv

  25. [33]

    Online dpo: Online direct preference optimization with fast-slow chasing, 2024

    Biqing Qi, Pengfei Li, Fangyuan Li, Junqi Gao, Kaiyan Zhang, and Bowen Zhou. Online dpo: Online direct preference optimization with fast-slow chasing, 2024

    2024

  26. [34]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36: 0 53728--53741, 2023

    2023

  27. [35]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  28. [36]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889--1897. PMLR, 2015

    2015

  29. [38]

    Proximal policy optimization algorithms, 2017 b

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017 b . URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  30. [39]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  31. [40]

    Welcome to the era of experience

    David Silver and Richard S Sutton. Welcome to the era of experience. Google AI, 1, 2025

    2025

  32. [41]

    A long way to go: Investigating length correlations in RLHF

    Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in RLHF . In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=G8LaO1P0xv

    2024

  33. [42]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 a

    2023 arXiv

  34. [43]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    2023 arXiv

  35. [44]

    Zephyr: Direct distillation of lm alignment

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Cl \'e mentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944, 2023

    arXiv

  36. [45]

    Thinking llms: General instruction following with thought generation

    Tianhao Wu, Janice Lan, Weizhe Yuan, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Thinking llms: General instruction following with thought generation. arXiv preprint arXiv:2410.10630, 2024 a

    2024 arXiv

  37. [46]

    Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge

    Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv preprint arXiv:2407.19594, 2024 b

    2024 arXiv

  38. [47]

    Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. arXiv preprint arXiv:2312.11456, 2023 a

    2023 arXiv

  39. [48]

    Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf

    Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf. CoRR, 2023 b

    2023

  40. [49]

    Wizardlm: Empowering large language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023 a

    2023 arXiv

  41. [50]

    Some things are more cringe than others: Preference optimization with the pairwise cringe loss

    Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682, 2023 b

    2023 arXiv

  42. [51]

    Is dpo superior to ppo for llm alignment? a comprehensive study

    Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study. arXiv preprint arXiv:2404.10719, 2024 a

    2024 arXiv

  43. [52]

    BPO : Staying close to the behavior LLM creates better online LLM alignment

    Wenda Xu, Jiachen Li, William Yang Wang, and Lei Li. BPO : Staying close to the behavior LLM creates better online LLM alignment. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processi...

    2024 doi

  44. [53]

    Self-rewarding language models

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024

    2024 arXiv

  45. [54]

    Wildchat: 1m chatgpt interaction logs in the wild

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470, 2024. Open Data Commons License Attribution family License

    2024 arXiv

  46. [55]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36: 0 55006--55021, 2023

    2023

  47. [56]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.