Pith. sign in

REVIEW 3 major objections 5 minor 31 references

Learning a Pessimistic Reward Model in RLHF

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a reward model made pessimistic by adversarial fine-tuning against rejection sampling lets offline RLHF greedily optimize policies without KL regularization.

desk verdict Novel and empirically interesting pessimistic-reward fine-tuning idea, but the central theorem's proof has a load-bearing gap and the no-regularization claim is not supported as written. read the letter →

arxiv 2505.20556 v1 pith:K3TXS2NU submitted 2025-05-26 cs.LG

Yinglun Xu , Hangoo Kang , Tarun Suresh , Yuxuan Wan , Gagandeep Singh This is my paper
classification cs.LG
keywords pessimisticrewardmodelhackingRLHFrejectionsamplingKLregularizationofflinereinforcementlearningadversarialtrainingTLDRsummarization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that reward hacking in offline RLHF can be prevented by making the reward model itself pessimistic, rather than by regularizing the policy toward the dataset. Its method, PET, fine-tunes a proxy reward so that it gives low relative scores to the policy produced by rejection sampling on that reward, while still fitting the preference data. The central claim is that once such a reward model is learned, a greedy policy optimizer (PPO with no KL term) is safe and can even find high-performing policies far from the dataset. On the TL;DR summarization task, unregularized PPO on the PET reward reaches a 40.8% win rate against human summaries at KL divergence 114.0, whereas the same greedy run on the proxy reward collapses to a 7.2% win rate. This matters because KL regularization is widely treated as necessary in RLHF, and this work argues it is a crude substitute for pessimism that excludes good out-of-distribution policies.

What carries the argument

The central object is the PET objective, Equation 2: minimize over reward models $r$ the relative score $V_r(\mathrm{RS}(r)) - V_r(\pi_{\mathrm{ref}})$ plus $\beta$ times the preference prediction loss $L_D(r)$. The rejection-sampling policy $\mathrm{RS}(r)$ is the actor; Proposition 2.1 guarantees that this particular policy maximizes $V_r$ within the RS policy class, so the original minimax formulation becomes a single minimization. The dataset-loss term $\beta L_D(r)$ is what keeps pessimism from becoming over-pessimism, and the coverage coefficient defined from the dataset distribution controls how far outside the data the guarantee extends.

What would settle it

Run unregularized PPO on a PET-trained reward on a held-out task, measure the true reward (or human preference) of the final policy as a function of KL from the reference policy; if win rate drops as KL grows, as it does with a proxy reward, the claim that pessimistic fine-tuning alone prevents reward hacking is false.

Watch

Extended reading notes

Core claim

PET solves a minimax problem in which the critic (reward model) tries to minimize the gap between the rejection-sampling policy and a reference policy, subject to keeping low prediction loss on the preference dataset. Proposition 2.1 collapses the game to a pure minimization over rewards: because for any reward the rejection-sampling policy is already optimal among rejection-sampling policies on that reward, the actor step can be folded into the reward update. The paper's Theorem 3.3 states that the resulting pessimistic reward, when used for rejection sampling, matches any dataset-covered rejection-sampling policy up to a gap that shrinks as $N^{-1/2}$ in the number of preference tuples. Empirically, the paper finds that greedy PPO on the PET reward outperforms the same on the proxy reward and matches or beats DPO, RPO, and chi-PO baselines, while running at a much higher KL divergence.

Load-bearing premise

The proof assumes PET returns a reward model that exactly minimizes the pessimistic objective (or otherwise satisfies the optimality condition used in the proof), but Algorithm 1 is only a stochastic gradient update with no convergence guarantee.

Editorial extensions

If this is right

  • Greedy policy optimization on a PET reward needs no KL term, so high-performing policies far from the dataset are no longer excluded.
  • Rejection sampling (best-of-N) on a PET reward beats the same procedure on the proxy reward, e.g., 39.2% versus 32.0% win rate on TL;DR.
  • PPO on the PET reward is competitive with or better than KL-regularized PPO, DPO, RPO, and chi-PO in the reported evaluations.
  • The theoretical bound gives a sample-efficiency guarantee: with dataset coverage, performance gap shrinks like $N^{-1/2}$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If PET rewards are trustworthy at high KL, the same fine-tuning could replace KL constraints in other alignment loops, such as iterative best-of-N or online preference collection, where exploration is currently capped.
  • Because the adversary is a rejection-sampling policy, the method's guarantees depend on that specific actor; extending the idea to arbitrary policy classes would need a different tractable 'best response'.
  • A direct test of pessimism is to measure the PET reward's ranking accuracy on deliberately adversarially sampled responses; if greedy search can still find overestimated responses, the method fails by its own criterion.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes PET, a pessimistic reward fine-tuning method for offline RLHF. PET is formulated as a minimax problem in which a reward model is trained to assign low relative scores to rejection-sampling policies induced by that same reward model, while keeping low prediction loss on the preference dataset. The authors state Theorem 3.3, which claims that rejection sampling on the PET reward attains performance comparable to any dataset-covered rejection-sampling policy without KL regularization. They then test PET with rejection sampling and with PPO on TL;DR summarization and IMDB, reporting win rates against human summaries and ground-truth sentiment likelihoods, and conclude that a greedy policy on the pessimistic reward avoids reward hacking without any regularization.

Significance. If Theorem 3.3 were sound, the paper would make a substantive contribution: it would show that pessimism can be injected at the reward-model stage rather than through KL regularization, with a finite-sample gap bound governed by dataset coverage. The paper also deserves credit for including a ground-truth evaluation on IMDB and direct pairwise comparisons against baselines, which provide a non-circular check on the central claim. However, the main proof contains a real derivation gap, the algorithm is not shown to solve the theoretical objective, and the headline empirical claim rests on a policy class and an experimental regime not covered by the theory and on very limited seed information.

major comments (3)
  1. [Appendix A.3 (proof of Theorem 3.3)] The proof requires the inequality V_{\hat r}(\hat\pi)-V_{\hat r}(\pi_\mathrm{ref})+\beta L_D(\hat r) \le V_{r^*}(\hat\pi)-V_{\hat r}(\pi_\mathrm{ref})+\beta L_D(r^*), which is then used to drop a bracket; equivalently the proof needs V_{\hat r}(\hat\pi)+\beta L_D(\hat r) \le V_{r^*}(\hat\pi)+\beta L_D(r^*). The paper justifies this by saying that \hat r minimizes Eq. (2). But Eq. (2) minimizes h(r)=V_r(\pi_{\mathrm{RS}}(r))-V_r(\pi_\mathrm{ref})+\beta L_D(r), and since \pi_{\mathrm{RS}}(r^*) maximizes V_{r^*}, optimality of \hat r gives only h(\hat r)\le V_{r^*}(\pi_{\mathrm{RS}}(r^*))-V_{r^*}(\pi_\mathrm{ref})+\beta L_D(r^*), whose right-hand side is at least V_{r^*}(\hat\pi)-V_{r^*}(\pi_\mathrm{ref})+\beta L_D(r^*). The needed inequality has the smaller right-hand side V_{r^*}(\hat\pi)-V_{\hat r}(\pi_\mathrm{ref})+\beta L_D(r^*), so it does not follow. The dropped term can be positive precisely when \pi_{\mathrm{RS}}(\hat r) is worse under the true reward than \pi_{\mathrm{RS}}(r^*), which is the situation the theorem is supposed to control. As written, the proof does not establish Theorem 3.3.
  2. [Appendix A.1 (PET as SGD on Eq. (2))] The derivation that PET is stochastic gradient descent on h(r) uses the claim that \nabla_\pi f(r,\pi)=0 at \pi=\pi_{\mathrm{RS}}(r) because \pi_{\mathrm{RS}} maximizes f over \Pi_{\mathrm{RS}}. Rejection sampling selects the argmax among n discrete samples, so the induced policy is not differentiable in the reward parameters and is not an unconstrained interior maximizer; the stationarity condition \nabla_\pi f=0 is therefore not justified. Additionally, Algorithm 1 provides only stochastic gradient updates with no convergence guarantee, so the returned reward model is not shown to be the exact minimizer of Eq. (2) used in Theorem 3.3. This severs the link between the practical algorithm and the theoretical guarantee.
  3. [Section 4.3 (Table 2) and Theorem 3.3] The headline empirical claim is that an unregularized PPO policy trained on the pessimistic reward achieves 40.8% win rate at KL 114.0, in contrast to 7.2% for unregularized PPO on the proxy reward. Theorem 3.3 applies only to rejection-sampling policies in \Pi_{\mathrm{RS}} that are covered by the dataset; PPO policies are not in that class, so the theory does not license the no-regularization claim for PPO. Moreover, Table 2 reports no standard errors or seed count; the two-seed analysis in Table 6 covers PET-PPO versus baselines but not the regularized-versus-unregularized comparison in Table 2. The central empirical contrast is thus supported by at most two seeds and is presented without any uncertainty quantification.
minor comments (5)
  1. [Table 1 caption] The caption says the results are on the IMDB dataset, but the evaluation metric is win rate against human responses and the surrounding text in Section 4.2 describes the TL;DR summarization setting; please reconcile the caption with the dataset actually used.
  2. [Tables 3 and 6] The single-run numbers in Table 3 differ from the two-seed means in Table 6 in the same metrics (for example DPO summarization 38.8 in Table 3 versus 37.5 +/- 1.3 in Table 6, and PET-PPO IMDB 98.3 versus 96.1 +/- 2.2). The paper should state explicitly which table reports which seeds and provide the per-seed values for all reported results.
  3. [Section 2.1 (sigmoid definition)] The text defines sigmoid(y):=1/(1-exp(-y)); the standard definition is 1/(1+exp(-y)), which is needed for the Bradley-Terry likelihood to be a valid probability.
  4. [Algorithm 1, line 7] The loss expression r_t(x_i, \pi_t(x_i, a_i)) mixes response variables with policy notation; it should be written in terms of the sampled responses a_i and a_ref,i, for example r_t(x_i,a_i) and r_t(x_i,a_ref,i).
  5. [Appendix A.3, first sentence] The opening of the proof is incomplete: 'Let \hat r be the reward solution to \hat r \in \arg\min_{r\in R} V...' cuts off and should be rewritten as a complete statement of the minimizer and the objective.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the proof has a rigor gap but the empirical and theoretical claims are not circular.

full rationale

The paper's central mechanism—PET trains a reward model by minimizing h(r)=V_r(pi_RS(r))-V_r(pi_ref)+beta*L_D(r) (Eq. 2)—does not presuppose the evaluation outcomes. The pessimistic objective is defined in terms of the reward model's own scores and prediction loss, not in terms of the external win rates or ground-truth sentiment used to validate the method. The fact that the trained reward gives low scores to its own induced RS policy is the intended mechanism of pessimism, not a disguised prediction; the claim that this prevents reward hacking is tested against independent targets (win rate vs. human summaries, IMDB ground-truth sentiment). Theorem 3.3's proof does rely on a saddle-point optimality condition for the minimax problem of Eq. 1, and the compressed proof in Appendix A.3 glosses over why an SGD solution to Eq. 2 would satisfy that saddle inequality; this is a rigor gap or omitted justification, not a circularity, because the needed inequality is a game-theoretic property of an exact minimax solution rather than a restatement of the theorem's conclusion. There are no load-bearing self-citations: the proof technique is attributed to Liu et al. (2024b), whose authors are disjoint from the present authors, and the assumptions are inherited from that external prior work rather than from a self-imported uniqueness claim. The empirical results in Section 4 are externally measured and would be falsifiable if PET merely shrank reward values without preserving alignment. Therefore no circular step is exhibited, and the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

PET introduces no new physical or theoretical entities. The central claim rests on standard offline RLHF assumptions (Bradley-Terry model, realizability of the true reward, bounded rewards, dataset coverage), a black-box concentration bound from Liu et al. (2024b), and two free hyperparameters (β and n) that are not ablated.

free parameters (2)
  • Pessimistic coefficient β = 10 (1/β = 0.1)
    Set in Section 4.1 'following the setups in [Rafailov et al., 2023]', with no sensitivity analysis. It controls the trade-off between pessimism and prediction loss.
  • Rejection sampling count n = 64 during PET; grid {16, 32, 64, 128} in Table 1
    n determines the degree of implicit KL regularization in RS policies and materially changes results (Table 1). The paper notes n=64 was chosen due to compute limits.
assumptions (6)
  • domain assumption Preferences follow a Bradley-Terry model with an underlying reward r*
    Section 2.1 assumes the preference signal is generated by a Bradley-Terry model.
  • domain assumption The true reward r* is in the reward model class R
    Required in Theorem 3.3 and Remark 3.4.
  • standard math Reward functions are bounded: |r| ≤ R for all r in R
    Assumed in Theorem 3.3 to control concentration terms.
  • domain assumption The coverage coefficient C_µD(R, π, π_ref) is finite for comparator policies
    Definition 3.2 and Theorem 3.3 require the comparator policy to be covered by the dataset.
  • standard math Theorem 5.3 from Liu et al. (2024b) concentration bound
    The proof in Appendix A.3 invokes this black-box result to bound the reward and loss differences.
  • standard math The dataset is sampled i.i.d. from µD
    Section 2.1 states the i.i.d. sampling assumption, which is needed for generalization bounds.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning a Pessimistic Reward Model in RLHF." pith.science (2026). https://pith.science/paper/K3TXS2NU

@misc{pith2026250520556,
  author       = {Pith},
  title        = {Pith review of: Learning a Pessimistic Reward Model in RLHF},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K3TXS2NU}},
  note         = {Machine review of arXiv:2505.20556}
}
read the original abstract

This work proposes `PET', a novel pessimistic reward fine-tuning method, to learn a pessimistic reward model robust against reward hacking in offline reinforcement learning from human feedback (RLHF). Traditional reward modeling techniques in RLHF train an imperfect reward model, on which a KL regularization plays a pivotal role in mitigating reward hacking when optimizing a policy. Such an intuition-based method still suffers from reward hacking, and the policies with large KL divergence from the dataset distribution are excluded during learning. In contrast, we show that when optimizing a policy on a pessimistic reward model fine-tuned through PET, reward hacking can be prevented without relying on any regularization. We test our methods on the standard TL;DR summarization dataset. We find that one can learn a high-quality policy on our pessimistic reward without using any regularization. Such a policy has a high KL divergence from the dataset distribution while having high performance in practice. In summary, our work shows the feasibility of learning a pessimistic reward model against reward hacking. The agent can greedily search for the policy with a high pessimistic reward without suffering from reward hacking.

Figures

Figures reproduced from arXiv: 2505.20556 by the authors.

Figure 1
Figure 1. A three-step reward-based learning framework. The first step is the traditional reward [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 4 canonical work pages

  1. [1]

    [2024b], with probability at least 1 − δ, the last term can be bound by V µ r∗ (π) − V µ r∗ (ˆπ) ≤ CµD (R, π, πref)2 8κ2 · β + 3β N log( Nϵ(R, ∥ · ∥∞) δ )

    Formally, based on the results of Theorem 5.3 in Liu et al. [2024b], with probability at least 1 − δ, the last term can be bound by V µ r∗ (π) − V µ r∗ (ˆπ) ≤ CµD (R, π, πref)2 8κ2 · β + 3β N log( Nϵ(R, ∥ · ∥∞) δ ). 15 Here, κ = 1 (1+exp(R))2 is a constant, and Nϵ(R, ∥ · ∥∞) is the ϵ-covering number for the reward model class Cheng et al. [2022]. Setting ...

    work page 2022

  2. [4]

    The rejection sampling process is effectively a policy as it takes a prompt as input and stochastically outputs a response

    In practice, the reward model ˆr is trained by minimizing the prediction loss, and the base policy π0 is usually set as a proxy policy for the dataset. The rejection sampling process is effectively a policy as it takes a prompt as input and stochastically outputs a response. An RS policy can be directly implemented on a policy model and a reward model wit...

    work page 2024

  3. [6]

    Helping or herd- ing? reward model ensembles mitigate but do not eliminate reward hacking

    Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herd- ing? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244,

    arXiv

  4. [7]

    Robust preference optimization through reward model distillation

    Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405.19316,

    arXiv

  5. [8]

    Mitigating preference hacking in policy optimization with pessimism

    Dhawal Gupta, Adam Fisch, Christoph Dann, and Alekh Agarwal. Mitigating preference hacking in policy optimization with pessimism. arXiv preprint arXiv:2503.06810,

    arXiv

  6. [9]

    Correcting the mythos of kl-regularization: Direct alignment without overoptimization via chi-squared preference optimization

    Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D Lee, Wen Sun, Akshay Krishnamurthy, and Dy- lan J Foster. Correcting the mythos of kl-regularization: Direct alignment without overoptimization via chi-squared preference optimization. arXiv preprint arXiv:2407.13399, 2024a. 10 Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Dylan J Foster, and Akshay Kri...

    arXiv

  7. [10]

    The n+ implementation details of rlhf with ppo: A case study on tl;dr summarization, 2024b

    Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The n+ implementation details of rlhf with ppo: A case study on tl;dr summarization, 2024b. URL https://arxiv.org/abs/2403.17031. Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, and Minlie Huang. To- wards efficient exact optimizati...

    arXiv 2024

  8. [11]

    Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models

    Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, and Prathap Ramachandra. Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. arXiv preprint arXiv:2402.10038,

    arXiv

Show all 31 references
  1. [12]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643,

    2005 arXiv

  2. [13]

    Statistical rejection sampling improves preference optimization

    Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657,

    arXiv

  3. [14]

    Rrm: Robust reward model training mitigates reward hacking

    Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, et al. Rrm: Robust reward model training mitigates reward hacking. arXiv preprint arXiv:2409.13156, 2024a. Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Gu...

    arXiv

  4. [16]

    Warm: On the benefits of weight averaged reward models

    Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401.12187,

    arXiv

  5. [17]

    Direct nash optimization: Teaching language models to self-improve with general preferences

    Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715,

    arXiv

  6. [18]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  7. [19]

    Boosting reward model with preference-conditional multi-aspect synthetic data generation

    Jiaming Shen, Ran Xu, Yennie Jun, Zhen Qin, Tianqi Liu, Carl Yang, Yi Liang, Simon Baumgartner, and Michael Bendersky. Boosting reward model with preference-conditional multi-aspect synthetic data generation. arXiv preprint arXiv:2407.16008,

    arXiv

  8. [20]

    Generalized preference optimization: A unified approach to offline alignment.arXiv preprint arXiv:2402.05749,

    Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment.arXiv preprint arXiv:2402.05749,

    arXiv

  9. [21]

    Causal confusion and reward misidentification in preference-based reward learning

    Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca D Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference-based reward learning. arXiv preprint arXiv:2204.06601,

    arXiv

  10. [22]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

    arXiv

  11. [23]

    TL;DR: Mining Reddit to learn automatic summarization

    Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL;DR: Mining Reddit to learn automatic summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu, editors, Proceedings of the Workshop on New Frontiers in Summarization , pages 59– 63, Cope...

    arXiv

  12. [24]

    Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf

    Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf. arXiv preprint arXiv:2405.21046,

    arXiv

  13. [25]

    Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. arXiv preprint arXiv:2312.11456, 2023a. Wei Xiong, Hanze Dong, Chenlu Ye, H...

    arXiv

  14. [26]

    Provable offline preference-based reinforcement learning

    Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D Lee, and Wen Sun. Provable offline preference-based reinforcement learning. arXiv preprint arXiv:2305.14816,

    arXiv

  15. [27]

    Self-exploring language models: Active preference elicitation for online alignment

    Shenao Zhang, Donghan Yu, Hiteshi Sharma, Han Zhong, Zhihan Liu, Ziyi Yang, Shuohang Wang, Hany Hassan, and Zhaoran Wang. Self-exploring language models: Active preference elicitation for online alignment. arXiv preprint arXiv:2405.19332,

    arXiv

  16. [28]

    Logarithmic regret for online kl-regularized reinforcement learning

    Heyang Zhao, Chenlu Ye, Wei Xiong, Quanquan Gu, and Tong Zhang. Logarithmic regret for online kl-regularized reinforcement learning. arXiv preprint arXiv:2502.07460,

  17. [30]

    1 will increase the bound on the performance gap analysis, which is undesired for sampling efficiency

    Adding a KL regularization multiplied by any positive coefficient to Eq. 1 will increase the bound on the performance gap analysis, which is undesired for sampling efficiency. B Policy Optimization Methods KL regularized proximal policy optimization (KL-PPO): For a prompt dist...

    2023

  18. [1952]

    Value-incentivized preference optimization: A unified approach to online and offline rlhf

    Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, and Bo Dai. Value-incentivized preference optimization: A unified approach to online and offline rlhf. arXiv preprint arXiv:2405.19320,

    arXiv

  19. [2017]

    Mitigating reward over- optimization in rlhf via behavior-supported regularization

    Juntao Dai, Taiye Chen, Yaodong Yang, Qian Zheng, and Gang Pan. Mitigating reward over- optimization in rlhf via behavior-supported regularization. arXiv preprint arXiv:2503.18130 ,

    arXiv

  20. [2022]

    Sample-efficient reinforcement learn- ing from human feedback via information-directed sampling

    Han Qi, Haochen Yang, Qiaosheng Zhang, and Zhuoran Yang. Sample-efficient reinforcement learn- ing from human feedback via information-directed sampling. arXiv preprint arXiv:2502.05434,

    arXiv

  21. [2023]

    Ralph Allan Bradley and Milton E Terry

    URL https://arxiv.org/abs/2304.01373. Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345,

    arXiv

  22. [2024]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. Yun...

    arXiv

  23. [2025]

    Rlhf workflow: From reward modeling to online rlhf

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.