Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that aligning large language models is an inverse reinforcement learning problem: the reward function is missing and must be inferred from human data, and this reframing unifies RLHF, DPO, and reward-guided decoding.

desk verdict A genuinely useful IRL-oriented survey of LLM post-training, but the abstract overclaims reward-model necessity and one reference looks fabricated. read the letter →

arxiv 2507.13158 v1 pith:ZHO3NIOG submitted 2025-07-17 cs.LG cs.AIcs.CL

Hao Sun , Mihaela van der Schaar This is my paper
classification cs.LGcs.AIcs.CL
keywords LLMalignmentinversereinforcementlearningrewardmodelingfromhumanfeedbackdirectpreferenceoptimizationMDPwithoutdatapost-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that LLM post-training—alignment—is best understood as inverse reinforcement learning, not as imitation or as standard RL with a known objective. In the token-generation MDP, the transition dynamics are deterministic (token concatenation) and the reward function is missing, so it has to be learned from human data: pairwise preferences, demonstrations, or verifiable outcomes. The authors treat this as a paradigm shift: explicit neural reward models are necessary, not optional, because they are what enable generalization beyond demonstrations and test-time optimization such as best-of-N sampling and reward-guided decoding. If the framing is right, research effort should concentrate on reward learning from behavior—how to collect data, what preference model to assume, and how to handle diversity and overoptimization—rather than on hand-designed objectives or pure behavior cloning.

What carries the argument

The machinery is the MDP\R formulation: an MDP with deterministic token-concatenation dynamics but no reward function, paired with a learned neural reward model that supplies the missing signal. The workhorse preference model is the Bradley-Terry (logistic) model applied in the embedding space of a pretrained LLM, converting pairwise comparisons into scalar reward differences. A second mechanism is divergence minimization over trajectory occupancy measures: forward KL reduces to the supervised fine-tuning objective, reverse KL to adversarial imitation with a discriminator-derived reward, which unifies SFT and reward modeling as two ends of the same IRL spectrum. The final piece is the reward-model infrastructure itself—active learning by Fisher information, decomposed orthogonal reward heads for diverse preferences, and best-of-N as the canonical way to consume a reward model at inference time.

What would settle it

An experiment that would falsify the necessity claim: on a fixed annotation budget covering a diverse set of alignment tasks and model families, use the same preference data to (a) train a reward model and optimize a policy with PPO, and (b) fine-tune directly on the preferred responses; if (b) matches or beats (a) on held-out evaluations across the board, the paper's core premise that explicit reward models are necessary collapses.

Watch

Extended reading notes

Core claim

The central claim is that every step of LLM alignment, from RLHF to DPO to reward-guided decoding, is a practical instance of inverse reinforcement learning. The argument proceeds by recasting token generation as an MDP\R: states are contexts, actions are tokens, transitions are deterministic concatenation, the initial state is the query distribution, and the reward is unspecified. Because the reward is absent, it must be reconstructed from real-world evidence—preferences, demonstrations, correctness labels—and the paper insists this reconstruction is the load-bearing component of alignment. Accordingly, the paper defends three roles for explicit reward models: they turn scalable preference data into training signals, they let reinforcement learning discover generalizable reasoning behaviors that imitation cannot induce, and they enable inference-time optimization. It also refines the theory of the reward model itself: modern reward models act on the embedding space of a pretrained LLM, making Bradley-Terry regression (rather than classical Bradley-Terry estimation) the appropriate foundation, and order consistency rather than exact score fidelity is the property that matters for ranking tasks.

Load-bearing premise

The whole argument rests on the assumption that what we want from an aligned model can be captured by a single number per response, recoverable from human comparisons, rather than being multidimensional, context-dependent, or non-stationary.

Editorial extensions

If this is right

  • Alignment research should be organized around reward learning from human data, not around hand-crafted objectives; this makes data collection and preference modeling first-class problems.
  • Explicit reward models unlock inference-time optimization—best-of-N filtering, reward-guided decoding, and search-based planning—so the same model can improve at deployment without retraining.
  • Because reward functions in IRL are not unique, the choice of preference model matters; order-consistent objectives such as binary classification can outperform Bradley-Terry when annotations are noisy.
  • Diverse human preferences can be decomposed into orthogonal reward heads, giving a route to personalized and pluralistic alignment rather than a single global reward.
  • The performance gains of RLVR-style reasoning training may come largely from internalizing structured response formats, which connects RL to prompt optimization and makes proxy verifiers like Prompt-OIRL a promising direction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I conjecture that, if the IRL framing is correct, the practical ceiling on alignment quality is set by the informativeness of human data and the expressiveness of the reward model, not by the policy optimizer.
  • A testable extension: best-of-N performance of a reward model should predict downstream policy alignment quality, so improving best-of-N scores on held-out tasks should transfer to deployed behavior.
  • The scalar-reward premise remains the weak link; evidence of systematic non-transitivity or irreducibly multidimensional preferences in realistic alignment tasks would push the field toward multi-objective or distributional reward models.
  • A further consequence I draw: the embedding-space view implies models could generate their own preference labels through self-play loops, reducing reliance on human annotation for reward learning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This survey proposes to organize LLM alignment and post-training as inverse reinforcement learning. It formalizes LLM generation as an MDP without a reward function (Section 2.2), reviews IL/IRL foundations and algorithms (Sections 2.3-2.4), motivates explicit neural reward models (Section 3), and surveys reward modeling from preferences, verifiable rewards, demonstrations, and the use of reward models in best-of-N, PPO/GRPO, and reward-guided decoding (Sections 4.1-4.4). It closes with a discussion of reward overoptimization, data quality, and off-policy issues (Section 4.5). The paper's central thesis, stated in the abstract, is that LLM alignment is best understood as IRL and that 'the necessity of constructing neural reward models from human data' is a key implication of this paradigm.

Significance. If the thesis were fully established, the paper would provide a useful unifying lens for the LLM alignment literature and a coherent research agenda centered on reward learning. The survey has genuine strengths: clear taxonomies (Tables 1, 3, and 5), explicit take-aways throughout, a clean token-level MDP formulation, and honest coverage of RLVR, reward overoptimization, preference diversity, and test-time optimization. It also reports evidence that qualifies its own thesis, including the heuristic status of the Bradley-Terry model, the existence of classification-based alternatives, and the success of RLVR without neural reward models. The contribution is best read as a structured review with a strong editorial viewpoint rather than a new derivation; its significance depends on whether the authors reconcile the abstract's necessity claim with the more qualified evidence presented in the body.

major comments (3)
  1. [Abstract; Sections 3.3 and 4.2] The abstract's central claim that neural reward models from human data are 'necessary' is directly contradicted by the paper's own content. Section 4.2 states that RLVR methods such as DeepSeek-R1 operate 'without the need for neural reward models,' and the same section reports that observed RLVR gains are largely attributable to format and structure learning rather than reward optimization, citing Shao et al. (2025) and Wang et al. (2025b). Section 4.1 likewise states that the Bradley-Terry choice is heuristic, that alternative preference models exist, and that simpler binary classification objectives can achieve order consistency. These passages show that preference-based scalar reward modeling is one successful route, not a demonstrated requirement. The authors should either weaken the abstract and the Section 3 framing to a context-dependent claim or explicitly delimit the scope (for example, to tasks without verifiable rewards) before the necessity claim is load-bearing.
  2. [Abstract; Section 4.5 (structure)] The abstract promises that the paper will 'draw insights from the literature on sparse-reward RL to identify open questions and potential research directions,' but the manuscript contains no such section. Section 4.5 addresses reward overoptimization, data quality, and off-policy data, and the paper then ends without the promised sparse-reward analysis. Either add the missing section or revise the abstract; as it stands, the advertised scope of the survey is not delivered.
  3. [Sections 4.1 and 4.5] The survey does not delimit the conditions under which the IRL reduction is valid. It relies throughout on a reward function recoverable from pairwise preferences (Section 4.1), while simultaneously documenting preference diversity, personalized and decomposed reward models (DRMs), and the fragility of optimizing learned rewards (Section 4.5). The reader is never told under what assumptions a single scalar reward suffices, nor how the IRL framing should be modified when preferences are non-scalar, non-stationary, or only partially observed. An explicit statement of scope, or a falsifiable criterion for when reward modeling is necessary, would make the survey's central claim defensible.
minor comments (6)
  1. [Section 4.1, Fisher-information box] The displayed Fisher information formulas contain several typos: the sum is written 'I = \sum_{i=1}^I' with I used for both the matrix and the summation index; the Bernoulli probability p_i is written with (\phi(x_i,y_{i,1}) - \phi(x_i,y_{i,1})) where the second argument is repeated; the text says the covariance matrix is 'I−∞' instead of I^{-1}; and 'theory fro' should be 'theory from'. These should be corrected before readers can use the formulas.
  2. [References; Table 1] The bibliography entry 'DeepSeek Team. Grpo: Generalized reinforcement preference optimization. arXiv preprint arXiv:2405.00000' uses a placeholder identifier, and Table 1's 'GRPO Team (2024)' does not match the reference entry. The paper should cite the actual DeepSeekMath/GRPO reference or a real arXiv number and make the in-text citation consistent.
  3. [Section 2.2] The symbol T is overloaded: it denotes the transition function early in the section and later denotes the maximal number of new tokens in a generation. This overloading is confusing in a paper aimed at newcomers and should be resolved by using separate symbols.
  4. [Section 4.1, active learning] The notation h_i for the Bernoulli observation is introduced but not defined, and the displayed distribution for h_i is missing a closing bracket. Please define h_i and clean up the expression.
  5. [Section 4.3, equation (5)] The product in the occupancy-measure derivation is written as \Pi_{t=k}^{t=0}, which has the index limits in reverse order; it should likely be \Pi_{t=0}^{k}.
  6. [Section 1.3] The paper states that 'alignment' and 'post-training' are used interchangeably. This conflation should be flagged explicitly, since not all post-training is alignment and not all alignment is post-training, and the distinction matters for the IRL framing.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the survey's key derivations are self-contained, and the authors' self-citations are pointers to prior work rather than load-bearing inputs to the central claim.

full rationale

This paper is a survey and tutorial, not a derivation paper. Its central claim that LLM alignment should be understood as inverse reinforcement learning and that neural reward models are needed is an interpretive thesis supported by cited empirical results such as RLHF, PPO/DPO, and DeepSeek-R1, not by an equation that assumes the conclusion as an input. The derivations actually shown in the text are self-contained: the behavior-clone error bounds cite classical work by Ross et al.; the f-divergence adversarial imitation framework is quoted from Ghasemipour et al.; the DPO objective is reproduced from Rafailov et al.; and the AfD forward-KL/reverse-KL identities are derived explicitly in Equations (5) and (6) before being attributed to Sun and van der Schaar. The self-citations to Sun et al. (2024b), Shen et al. (2025), and related papers appear in literature-review passages that summarize the authors' prior methods or point to a formal justification for Bradley-Terry regression, but the paper itself concedes that BT is a heuristic choice, that alternatives such as KTO and discrete choice models exist, and that simpler classification objectives can achieve order consistency. The self-citations are therefore illustrative rather than load-bearing for the abstract's 'necessity' claim. The tension between the abstract's emphasis on neural reward models and Section 4.2's documentation of RLVR succeeding without them is a genuine overclaim, but overclaiming is a correctness concern, not circularity. Under the hard rule that circularity requires a specific reduction or a load-bearing self-citation chain, no such step is present; the score reflects only a mild, non-load-bearing self-citation pattern common in survey papers.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The tutorial relies on standard RL/IRL concepts and on the domain assumptions listed. It introduces no new free parameters or invented entities, because it presents no new fitted model or derivation.

assumptions (3)
  • domain assumption LLM token generation is well modeled as an MDP with deterministic token-concatenation transitions and known dynamics.
    Section 2.2 defines the MDP for LLM generation; this assumption justifies applying MDP/IRL machinery to LLM alignment.
  • domain assumption Human preferences are generated by a latent scalar reward function through a Bradley-Terry (or similar) comparison model.
    Section 4.1 builds reward modeling on the Bradley-Terry model; if preferences are not representable by a scalar reward, the central IRL framing loses force.
  • domain assumption Behavior datasets (demonstrations or preferences) contain enough signal to recover a useful reward function.
    Section 2.3 and Section 4 assume behavior data is sufficient; the paper itself notes the off-policy problem but relies on this sufficiency.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities." pith.science (2026). https://pith.science/paper/ZHO3NIOG

@misc{pith2026250713158,
  author       = {Pith},
  title        = {Pith review of: Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZHO3NIOG}},
  note         = {Machine review of arXiv:2507.13158}
}
read the original abstract

In the era of Large Language Models (LLMs), alignment has emerged as a fundamental yet challenging problem in the pursuit of more reliable, controllable, and capable machine intelligence. The recent success of reasoning models and conversational AI systems has underscored the critical role of reinforcement learning (RL) in enhancing these systems, driving increased research interest at the intersection of RL and LLM alignment. This paper provides a comprehensive review of recent advances in LLM alignment through the lens of inverse reinforcement learning (IRL), emphasizing the distinctions between RL techniques employed in LLM alignment and those in conventional RL tasks. In particular, we highlight the necessity of constructing neural reward models from human data and discuss the formal and practical implications of this paradigm shift. We begin by introducing fundamental concepts in RL to provide a foundation for readers unfamiliar with the field. We then examine recent advances in this research agenda, discussing key challenges and opportunities in conducting IRL for LLM alignment. Beyond methodological considerations, we explore practical aspects, including datasets, benchmarks, evaluation metrics, infrastructure, and computationally efficient training and inference techniques. Finally, we draw insights from the literature on sparse-reward RL to identify open questions and potential research directions. By synthesizing findings from diverse studies, we aim to provide a structured and critical overview of the field, highlight unresolved challenges, and outline promising future directions for improving LLM alignment through RL and IRL techniques.

Figures

Figures reproduced from arXiv: 2507.13158 by the authors.

Figure 1
Figure 1. A comparison of different LLM generation optimization approaches. The first row represents (1) direct generation, (2) prompt optimization, (3) Supervised Fine-Tuning (SFT) on a high-quality dataset. The second row represents methods that leverage reward models (i.e., the IRL approach): (4) reward models can be used to filter out low-quality generations, (5-6) reward models can be combined with prompt optimization or… view at source ↗
Figure 2
Figure 2. Reward model overoptimization ( [PITH_FULL_IMAGE:figures/full_fig_p019_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectives

    cs.LG 2025-10 conditional novelty 4.0 of 10

    Bayesian IRL with sequential posterior updates can recover a usable toxicity-reduction reward from LLM demonstrations and reproduce ground-truth RLHF detoxification.

Reference graph

Works this paper leans on

117 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  2. [7]

    20 (AAAI 2025 and ACL

    2025

  3. [8]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

    1901

  4. [11]

    Pad: Personalized alignment of llms at decoding-time.arXiv preprint arXiv:2410.04070, 2024a

    Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, and Zuozhu Liu. Pad: Personalized alignment of llms at decoding-time.arXiv preprint arXiv:2410.04070, 2024a. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models.arXiv preprint arXiv:2401.01335, 2024b. Paul F Chr...

    arXiv

  5. [13]

    Reward model ensembles help mitigate overoptimization

    Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743,

    arXiv

  6. [14]

    Phaseevo: Towards unified in-context prompt optimization for large language models.arXiv preprint arXiv:2402.11347,

    Wendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley Malin, and Sricharan Kumar. Phaseevo: Towards unified in-context prompt optimization for large language models.arXiv preprint arXiv:2402.11347,

    arXiv

  7. [15]

    Automatic prompt optimization via heuristic search: A survey.arXiv preprint arXiv:2502.18746,

    Wendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley A Malin, and Sricha- ran Kumar. Automatic prompt optimization via heuristic search: A survey.arXiv preprint arXiv:2502.18746,

    arXiv

  8. [17]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186,

    2019

Show all 117 references
  1. [18]

    Raft: Reward ranked finetuning for generative foundation model alignment

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767,

    arXiv

  2. [19]

    21 (AAAI 2025 and ACL

    2025

  3. [20]

    Tutorial: Inverse Reinforcement Learning Meets LLM Alignment Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Se...

    arXiv

  4. [21]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475,

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475,

    arXiv

  5. [22]

    Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306,

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306,

    arXiv

  6. [24]

    Self-attentional credit assignment for transfer in reinforcement learning.arXiv preprint arXiv:1907.08027,

    Johan Ferret, Raphaël Marinier, Matthieu Geist, and Olivier Pietquin. Self-attentional credit assignment for transfer in reinforcement learning.arXiv preprint arXiv:1907.08027,

    1907 arXiv

  7. [25]

    Learning robust rewards with adversarial inverse reinforcement learning

    Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248,

    arXiv

  8. [27]

    Bonbon alignment for large language models and the sweetness of best-of-n sampling.arXiv preprint arXiv:2406.00832,

    Lin Gui, Cristina Gârbacea, and Victor Veitch. Bonbon alignment for large language models and the sweetness of best-of-n sampling.arXiv preprint arXiv:2406.00832,

    arXiv

  9. [28]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv

  10. [29]

    Connecting large language models with evolutionary algorithms yields powerful prompt optimizers

    Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08532,

    arXiv

  11. [30]

    Direct language model alignment from online ai feedback

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792,

    arXiv

  12. [31]

    22 (AAAI 2025 and ACL

    2025

  13. [32]

    Soft actor-critic algorithms and applications.arXiv preprint arXiv:1812.05905,

    Tutorial: Inverse Reinforcement Learning Meets LLM Alignment Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications.arXiv preprint arXiv...

    arXiv

  14. [34]

    Prompt-to- prompt image editing with cross attention control.arXiv preprint arXiv:2208.01626,

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control.arXiv preprint arXiv:2208.01626,

    arXiv

  15. [36]

    Explaining length bias in llm-based preference evaluations

    Zhengyu Hu, Linxin Song, Jieyu Zhang, Zheyuan Xiao, Tianfu Wang, Zhengyu Chen, Nicholas Jing Yuan, Jianxun Lian, Kaize Ding, and Hui Xiong. Explaining length bias in llm-based preference evaluations. arXiv preprint arXiv:2407.01085,

    arXiv

  16. [37]

    Large language models cannot self-correct reasoning yet.arXiv preprint arXiv:2310.01798,

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet.arXiv preprint arXiv:2310.01798,

    arXiv

  17. [38]

    Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback.arXiv preprint arXiv:2406.09279,

    Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback.arXiv preprint arXiv:2406.09279,

    arXiv

  18. [39]

    Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    arXiv

  19. [40]

    Towards efficient exact optimization of language model alignment.arXiv preprint arXiv:2402.00856,

    Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, and Minlie Huang. Towards efficient exact optimization of language model alignment.arXiv preprint arXiv:2402.00856,

    arXiv

  20. [41]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    2001 arXiv

  21. [43]

    Args: Alignment as reward-guided search.arXiv preprint arXiv:2402.01694,

    Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guided search.arXiv preprint arXiv:2402.01694,

    arXiv

  22. [44]

    Decomposed prompting: A modular approach for solving complex tasks.arXiv preprint arXiv:2210.02406,

    Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks.arXiv preprint arXiv:2210.02406,

    arXiv

  23. [45]

    Few-shot steerable alignment: Adapting rewards and llm policies with neural processes.arXiv preprint arXiv:2412.13998,

    Katarzyna Kobalczyk, Claudio Fanconi, Hao Sun, and Mihaela van der Schaar. Few-shot steerable alignment: Adapting rewards and llm policies with neural processes.arXiv preprint arXiv:2412.13998,

    arXiv

  24. [46]

    Diffwave: A versatile diffusion model for audio synthesis.arXiv preprint arXiv:2009.09761,

    Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis.arXiv preprint arXiv:2009.09761,

    2009 arXiv

  25. [47]

    Privacy in large language models: Attacks, defenses and future directions.arXiv preprint arXiv:2310.10383, 2023a

    Haoran Li, Yulin Chen, Jinglong Luo, Yan Kang, Xiaojin Zhang, Qi Hu, Chunkit Chan, and Yangqiu Song. Privacy in large language models: Attacks, defenses and future directions.arXiv preprint arXiv:2310.10383, 2023a. Jiaxiang Li, Siliang Zeng, Hoi-To Wai, Chenliang Li, Alfredo G...

    arXiv

  26. [48]

    Personalized language modeling from personalized human feedback.arXiv preprint arXiv:2402.05133, 2024b

    Xinyu Li, Ruiyang Zhou, Zachary C Lipton, and Liu Leqi. Personalized language modeling from personalized human feedback.arXiv preprint arXiv:2402.05133, 2024b. Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient meth...

    arXiv

  27. [49]

    Reward-guided speculative decoding for efficient llm reasoning.arXiv preprint arXiv:2501.19324,

    Baohao Liao, Yuhui Xu, Hanze Dong, Junnan Li, Christof Monz, Silvio Savarese, Doyen Sahoo, and Caiming Xiong. Reward-guided speculative decoding for efficient llm reasoning.arXiv preprint arXiv:2501.19324,

    arXiv

  28. [50]

    Ai transparency in the age of llms: A human-centered research roadmap

    Q Vera Liao and Jennifer Wortman Vaughan. Ai transparency in the age of llms: A human-centered research roadmap. arXiv preprint arXiv:2306.01941, 10,

    arXiv

  29. [51]

    24 (AAAI 2025 and ACL

    URL https://transformer-circuits.pub/2025/attribution-graphs/biology.html. 24 (AAAI 2025 and ACL

    2025

  30. [52]

    Statistical rejection sampling improves preference optimization.arXiv preprint arXiv:2309.06657,

    Tutorial: Inverse Reinforcement Learning Meets LLM Alignment Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization.arXiv preprint arXiv:2309.06657,

    arXiv

  31. [53]

    Rrm: Robust reward model training mitigates reward hacking.arXiv preprint arXiv:2409.13156,

    Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, et al. Rrm: Robust reward model training mitigates reward hacking.arXiv preprint arXiv:2409.13156,

    arXiv

  32. [54]

    Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692,

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692,

    1907 arXiv

  33. [55]

    Inference-time scaling for generalist reward modeling.arXiv preprint arXiv:2504.02495,

    Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling.arXiv preprint arXiv:2504.02495,

  34. [56]

    Rethinking diverse human preference learning through principal component analysis.arXiv preprint arXiv:2502.13131,

    Feng Luo, Rui Yang, Hao Sun, Chunyuan Deng, Jiarui Yao, Jingyan Shen, Huan Zhang, and Hanjie Chen. Rethinking diverse human preference learning through principal component analysis.arXiv preprint arXiv:2502.13131,

    arXiv

  35. [57]

    Generative reward models.arXiv preprint arXiv:2410.12832,

    Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models.arXiv preprint arXiv:2410.12832,

    arXiv

  36. [58]

    A survey of explainable reinforcement learning

    Stephanie Milani, Nicholay Topin, Manuela Veloso, and Fei Fang. A survey of explainable reinforcement learning. arXiv preprint arXiv:2202.08434,

    arXiv

  37. [59]

    Playing atari with deep reinforcement learning.arXiv preprint arXiv:1312.5602,

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning.arXiv preprint arXiv:1312.5602,

    arXiv

  38. [60]

    William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber

    https://www.amazon.science/publications/optimal-design-for-human-preference-elicitation. William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models. arXiv preprint arXiv:2402.08114,

    arXiv

  39. [61]

    25 (AAAI 2025 and ACL

    2025

  40. [62]

    Deep Research is a new agentic AI capability integrated within ChatGPT that autonomously conducts multi-step web research and synthesizes comprehensive reports

    Accessed: 2025-04-16. Deep Research is a new agentic AI capability integrated within ChatGPT that autonomously conducts multi-step web research and synthesizes comprehensive reports. Manu Orsini, Anton Raichuk, Léonard Hussenot, Damien Vincent, Robert Dadashi, Sertan Girgin, M...

    2025

  41. [63]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177,

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177,

    1910 arXiv

  42. [65]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952,

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952,

    arXiv

  43. [67]

    gradient descent

    Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and beam search.arXiv preprint arXiv:2305.03495,

    arXiv

  44. [68]

    A survey on explainable reinforcement learning: Concepts, algorithms, challenges.arXiv preprint arXiv:2211.06665,

    Yunpeng Qing, Shunyu Liu, Jie Song, Huiqiong Wang, and Mingli Song. A survey on explainable reinforcement learning: Concepts, algorithms, challenges.arXiv preprint arXiv:2211.06665,

    arXiv

  45. [69]

    Direct preference optimization: Your language model is secretly a reward model.arXiv preprint arXiv:2305.18290,

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.arXiv preprint arXiv:2305.18290,

    arXiv

  46. [70]

    26 (AAAI 2025 and ACL

    2025

  47. [71]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3,

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3,

    arXiv

  48. [72]

    A critical look at tokenwise reward-guided text generation.arXiv preprint arXiv:2406.07780,

    Ahmad Rashid, Ruotian Wu, Julia Grosse, Agustinus Kristiadi, and Pascal Poupart. A critical look at tokenwise reward-guided text generation.arXiv preprint arXiv:2406.07780,

  49. [73]

    A generalist agent.arXiv preprint arXiv:2205.06175,

    Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth- Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent.arXiv preprint arXiv:2205.06175,

    arXiv

  50. [74]

    Learning long-term reward redistribution via randomized return decomposition.arXiv preprint arXiv:2111.13485,

    Zhizhou Ren, Ruihan Guo, Yuan Zhou, and Jian Peng. Learning long-term reward redistribution via randomized return decomposition.arXiv preprint arXiv:2111.13485,

    arXiv

  51. [76]

    High-dimensional continuous control using generalized advantage estimation.arXiv preprint arXiv:1506.02438,

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation.arXiv preprint arXiv:1506.02438,

    arXiv

  52. [78]

    Spurious rewards: Rethinking training signals in rlvr.arXiv preprint arXiv:2506.10947,

    Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, et al. Spurious rewards: Rethinking training signals in rlvr.arXiv preprint arXiv:2506.10947,

    arXiv

  53. [79]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,

    arXiv

  54. [80]

    Reviving the classics: Active reward modeling in large language model alignment

    Yunyi Shen, Hao Sun, and Jean-François Ton. Reviving the classics: Active reward modeling in large language model alignment. arXiv preprint arXiv:2502.04354,

    arXiv

  55. [81]

    Understanding the performance gap in preference learning: A dichotomy of rlhf and dpo.arXiv preprint arXiv:2505.19770,

    Ruizhe Shi, Minhak Song, Runlong Zhou, Zihan Zhang, Maryam Fazel, and Simon S Du. Understanding the performance gap in preference learning: A dichotomy of rlhf and dpo.arXiv preprint arXiv:2505.19770,

    arXiv

  56. [82]

    When life gives you lemons, make cherryade: Converting feedback from bad responses into good labels.arXiv preprint arXiv:2210.15893,

    Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Weston, and Jing Xu. When life gives you lemons, make cherryade: Converting feedback from bad responses into good labels.arXiv preprint arXiv:2210.15893,

    arXiv

  57. [83]

    27 (AAAI 2025 and ACL

    2025

  58. [84]

    Mastering chess and shogi by self-play with a general reinforcement learning algorithm.arXiv preprint arXiv:1712.01815,

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm.arXiv preprint arXiv:1712.01815,

    arXiv

  59. [85]

    Distributional preference learning: Understanding and accounting for hidden context in rlhf.arXiv preprint arXiv:2312.08358,

    Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell. Distributional preference learning: Understanding and accounting for hidden context in rlhf.arXiv preprint arXiv:2312.08358,

    arXiv

  60. [86]

    Should we be going mad? a look at multi-agent debate strategies for llms.arXiv preprint arXiv:2311.17371,

    Andries Smit, Paul Duckworth, Nathan Grinsztajn, Thomas D Barrett, and Arnu Pretorius. Should we be going mad? a look at multi-agent debate strategies for llms.arXiv preprint arXiv:2311.17371,

    arXiv

  61. [87]

    A roadmap to pluralistic alignment.arXiv preprint arXiv:2402.05070,

    Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. A roadmap to pluralistic alignment.arXiv preprint arXiv:2402.05070,

    arXiv

  62. [89]

    Inverse-rlignment: Inverse reinforcement learning from demonstrations for llm alignment.arXiv preprint arXiv:2405.15624,

    Hao Sun and Mihaela van der Schaar. Inverse-rlignment: Inverse reinforcement learning from demonstrations for llm alignment.arXiv preprint arXiv:2405.15624,

    arXiv

  63. [90]

    Query-dependent prompt evaluation and optimization with offline inverse rl

    Hao Sun, Alihan Hüyük, and Mihaela van der Schaar. Query-dependent prompt evaluation and optimization with offline inverse rl. InThe Twelfth International Conference on Learning Representations, 2023a. Hao Sun, Alihan Hüyük, Daniel Jarrett, and Mihaela van der Schaar. Accounta...

    2024 arXiv

  64. [92]

    Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback.arXiv preprint arXiv:2112.09737,

    Tutorial: Inverse Reinforcement Learning Meets LLM Alignment Niket Tandon, Aman Madaan, Peter Clark, and Yiming Yang. Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback.arXiv preprint arXiv:2112.09737,

    arXiv

  65. [93]

    Deepmind control suite.arXiv preprint arXiv:1801.00690,

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite.arXiv preprint arXiv:1801.00690,

    arXiv

  66. [94]

    Grpo: Generalized reinforcement preference optimization.arXiv preprint arXiv:2405.00000,

    DeepSeek Team. Grpo: Generalized reinforcement preference optimization.arXiv preprint arXiv:2405.00000,

  67. [95]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530,

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530,

    arXiv

  68. [96]

    Gram: A generative foundation reward model for reward generalization.arXiv preprint arXiv:2506.14175, 2025a

    Chenglong Wang, Yang Gan, Yifu Huo, Yongyu Mu, Qiaozhi He, Murun Yang, Bei Li, Tong Xiao, Chunliang Zhang, Tongran Liu, et al. Gram: A generative foundation reward model for reward generalization.arXiv preprint arXiv:2506.14175, 2025a. Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqian...

  69. [97]

    Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171,

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171,

    arXiv

  70. [98]

    Reinforcement learning for reasoning in large language models with one training example

    Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.20571, 2025b. Jason Wei, Yi Tay,...

    2025 arXiv

  71. [100]

    How to leverage demonstration data in alignment for large language model? a self-imitation learning perspective.arXiv preprint arXiv:2410.10093,

    Teng Xiao, Mingxiao Li, Yige Yuan, Huaisheng Zhu, Chao Cui, and Vasant G Honavar. How to leverage demonstration data in alignment for large language model? a self-imitation learning perspective.arXiv preprint arXiv:2410.10093,

    arXiv

  72. [101]

    Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf.arXiv preprint arXiv:2312.11456,

    Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf.arXiv preprint arXiv:2312.11456,

    arXiv

  73. [102]

    A minimalist approach to llm reasoning: from rejection sampling to reinforce.arXiv preprint arXiv:2504.11343,

    Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. A minimalist approach to llm reasoning: from rejection sampling to reinforce.arXiv preprint arXiv:2504.11343,

    arXiv

  74. [103]

    Towards large reasoning models: A survey of reinforced reasoning with large language models.arXiv preprint arXiv:2501.09686,

    Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, et al. Towards large reasoning models: A survey of reinforced reasoning with large language models.arXiv preprint arXiv:2501.09686,

    arXiv

  75. [104]

    Is dpo superior to ppo for llm alignment? a comprehensive study.arXiv preprint arXiv:2404.10719,

    Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study.arXiv preprint arXiv:2404.10719,

    arXiv

  76. [105]

    Large language models as optimizers.arXiv preprint arXiv:2309.03409,

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers.arXiv preprint arXiv:2309.03409,

    arXiv

  77. [106]

    Rethinking goal-conditioned supervised learning and its connection to offline rl.arXiv preprint arXiv:2202.04478,

    Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl.arXiv preprint arXiv:2202.04478,

    arXiv

  78. [107]

    Regularizing hidden states enables learning generalizable reward model for llms.arXiv preprint arXiv:2406.10216,

    Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for llms.arXiv preprint arXiv:2406.10216,

    arXiv

  79. [108]

    Relative preference optimization: Enhancing llm alignment through contrasting responses across identical and diverse prompts

    Yueqin Yin, Zhendong Wang, Yi Gu, Hai Huang, Weizhu Chen, and Mingyuan Zhou. Relative preference optimization: Enhancing llm alignment through contrasting responses across identical and diverse prompts. arXiv preprint arXiv:2402.10958,

    arXiv

  80. [109]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    arXiv

  81. [110]

    Rrhf: Rank responses to align language models with human feedback without tears.arXiv preprint arXiv:2304.05302,

    Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears.arXiv preprint arXiv:2304.05302,

    arXiv

  82. [111]

    Diversifying ai: Towards creative chess with alphazero

    Tom Zahavy, Vivek Veeriah, Shaobo Hou, Kevin Waugh, Matthew Lai, Edouard Leurent, Nenad Tomasev, Lisa Schut, Demis Hassabis, and Satinder Singh. Diversifying ai: Towards creative chess with alphazero. arXiv preprint arXiv:2308.09175,

    arXiv

  83. [112]

    When scaling meets LLM finetuning: The effect of data, model and finetuning method

    Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets LLM finetuning: The effect of data, model and finetuning method. InThe Twelfth International Conference on Learning Representations, 2024a. URL https://openreview.net/forum?id=5HCnKDeTws. 30 (AAAI 2025 and ACL

    2025

  84. [113]

    Rest-mcts*: Llm self-training via process reward guided tree search.Advances in Neural Information Processing Systems, 37:64735–64772, 2024b

    Tutorial: Inverse Reinforcement Learning Meets LLM Alignment Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search.Advances in Neural Information Processing Systems, 37:64735–64772, 2024...

    arXiv

  85. [114]

    Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, and Yang Liu

    URL https://openreview.net/forum?id=u1cQYxRI1H. Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, and Yang Liu. Overcoming reward overoptimization via adversarial policy optimization with lightweight uncertainty estimation.arXiv preprint arXiv:2403.05171, 2024d. Yao ...

    arXiv

  86. [115]

    Secrets of rlhf in large language models part i: Ppo.arXiv preprint arXiv:2307.04964,

    Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of rlhf in large language models part i: Ppo.arXiv preprint arXiv:2307.04964,

    arXiv

  87. [116]

    Dpo meets ppo: Reinforced token optimization for rlhf.arXiv preprint arXiv:2404.18922,

    Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf.arXiv preprint arXiv:2404.18922,

    arXiv

  88. [117]

    Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36:55006–55021, 2023a

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36:55006–55021, 2023a. Denny Zhou, Nathanael Schärli, Le Hou, Jason ...

    2011 arXiv

  89. [1952]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818,

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818,

    arXiv

  90. [1973]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.arXiv preprint arXiv:2206.04615,

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.arXiv preprint ar...

    arXiv

  91. [1991]

    Retrieval-augmented thought process as sequential decision making.arXiv preprint arXiv:2402.07812,

    Thomas Pouplin, Hao Sun, Samuel Holt, and Mihaela Van der Schaar. Retrieval-augmented thought process as sequential decision making.arXiv preprint arXiv:2402.07812,

    arXiv

  92. [1992]

    Style over substance: Evaluation biases for large language models.arXiv preprint arXiv:2307.03025,

    Minghao Wu and Alham Fikri Aji. Style over substance: Evaluation biases for large language models.arXiv preprint arXiv:2307.03025,

    arXiv

  93. [1994]

    Contrastive prefence learning: Learning from human feedback without rl.arXiv preprint arXiv:2310.13639,

    Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W Bradley Knox, and Dorsa Sadigh. Contrastive prefence learning: Learning from human feedback without rl.arXiv preprint arXiv:2310.13639,

    arXiv

  94. [1995]

    Dense reward for free in reinforcement learning from human feedback.arXiv preprint arXiv:2402.00782,

    Alex J Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. Dense reward for free in reinforcement learning from human feedback.arXiv preprint arXiv:2402.00782,

    arXiv

  95. [1998]

    28 (AAAI 2025 and ACL

    2025

  96. [2007]

    A survey of temporal credit assignment in deep reinforcement learning.arXiv preprint arXiv:2312.01072,

    Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, Olivier Pietquin, and Laura Toni. A survey of temporal credit assignment in deep reinforcement learning.arXiv preprint arXiv:2312.01072,

    arXiv

  97. [2011]

    Training language models with natural language feedback.arXiv preprint arXiv:2204.14146, 8,

    Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with natural language feedback.arXiv preprint arXiv:2204.14146, 8,

    arXiv

  98. [2013]

    Dota 2 with large scale deep reinforcement learning

    Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680,

    1912 arXiv

  99. [2015]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  100. [2016]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    arXiv

  101. [2017]

    Sft memorizes, rl generalizes: A comparative study of foundation model post-training

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161,

    arXiv

  102. [2018]

    Off-policy deep reinforcement learning without exploration

    Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052–2062. PMLR,

  103. [2019]

    Robocat: A self-improving generalist agent for robotic manipulation

    Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X Lee, Maria Bauzá, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, et al. Robocat: A self-improving generalist agent for robotic manipulation. arXiv preprint arXiv:2306.11706,

    arXiv

  104. [2020]

    Maxmin-rlhf: Alignment with diverse human preferences

    Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. Maxmin-rlhf: Alignment with diverse human preferences. arXiv preprint arXiv:2402.08925,

    arXiv

  105. [2021]

    23 (AAAI 2025 and ACL

    2025

  106. [2022]

    Pilaf: Optimal human preference sampling for reward modeling.arXiv preprint arXiv:2502.04270,

    Yunzhen Feng, Ariel Kwiatkowski, Kunhao Zheng, Julia Kempe, and Yaqi Duan. Pilaf: Optimal human preference sampling for reward modeling.arXiv preprint arXiv:2502.04270,

    arXiv

  107. [2023]

    Scalable ensembling for mitigating reward overoptimisation.arXiv preprint arXiv:2406.01013,

    Ahmed M Ahmed, Rafael Rafailov, Stepan Sharkov, Xuechen Li, and Sanmi Koyejo. Scalable ensembling for mitigating reward overoptimisation.arXiv preprint arXiv:2406.01013,

    arXiv

  108. [2024]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022a

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022a. Yunt...

    arXiv

  109. [2025]

    Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model.arXiv preprint arXiv:2310.09520,

    Haikang Deng and Colin Raffel. Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model.arXiv preprint arXiv:2310.09520,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.