Pith. sign in

REVIEW 4 major objections 4 minor 29 references

BPO: Revisiting Preference Modeling in Direct Preference Optimization

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BPO swaps DPO's relative reward margin for a balanced margin $\min(r_w, -\alpha r_l)$, keeping chosen responses from degrading and adding about ten points of math accuracy on the tested models.

desk verdict A novel balanced-margin objective with real potential, but the printed loss has a sign error that inverts the method, and the headline gain is overstated. read the letter →

arxiv 2506.03557 v1 pith:XEFLY7S2 submitted 2025-06-04 cs.CL

Lin Sun , Chuang Liu , Peng Liu , Bingyang Li , Weijia Lu , Ning Wu This is my paper
classification cs.CL
keywords directpreferenceoptimizationdegradedchosenresponsesbalancedrewardmargingapadaptorlanguagemodelalignmentmathematicalreasoninglosses
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Direct preference optimization (DPO) is vulnerable to a failure mode this paper names Degraded Chosen Responses: because the loss only rewards the gap between chosen and rejected response scores, the model can satisfy the gap by making the chosen response less likely. The paper proposes Balanced Preference Optimization (BPO), which replaces the reward gap $r_w - r_l$ with a balanced reward margin $\min(r_w, -\alpha r_l)$, so the loss also cares that the chosen response's absolute reward stays positive while the rejected response is pushed down. The required change is one line in an existing DPO loss. On five competition-level math benchmarks, BPO improves average accuracy over DPO by 10.1 points on Llama-3.1-8B-Instruct and 11.7 points on Qwen2.5-Math-7B. If correct, this gives a simpler explanation of why some earlier DPO fixes worked: they indirectly enforced what BPO enforces directly.

What carries the argument

The central object is the balanced reward margin $\rho^b_\theta = \min(r_w, -\alpha r_l)$, with the gap adaptor $\alpha \in (0,1]$. It carries the argument by turning the pairwise difference objective into a weakest-link objective: whichever of the two rewards is smaller sets the loss, so the chosen response's reward cannot silently collapse while the rejected response is pushed down. The resulting gradient update touches only one side at a time, which is the mechanism that prevents degraded chosen responses and also reduces gradient computation in the chosen-dominated regime.

What would settle it

Select $\alpha$ on a held-out split of the five math benchmarks and then measure BPO's gain over DPO; the empirical claim falls if the gain vanishes, and the theoretical claim falls if $\min(r_w, -\alpha r_l)$ never becomes positive during training.

Watch

Extended reading notes

Core claim

The paper's central claim is that the Degraded Chosen Responses issue is caused by the relative reward margin itself, and that replacing it with the balanced reward margin $\rho^b_\theta = \min(r_w, -\alpha r_l)$ resolves it. Here $r_w$ and $r_l$ are the implicit rewards $\beta\log(\pi_\theta(y_w|x)/\pi_{\mathrm{ref}}(y_w|x))$ and $\beta\log(\pi_\theta(y_l|x)/\pi_{\mathrm{ref}}(y_l|x))$, and $\alpha \in (0,1]$ is a gap adaptor that controls how hard rejected responses are suppressed. When $r_w < -\alpha r_l$ the gradient updates only the chosen response, and otherwise only the rejected response, so the loss is always driven by the weaker side. The paper proves that if the optimized margin satisfies $\rho^b_\theta \ge \gamma$, then $\pi_\theta(y_w|x) \ge \exp(\gamma/\beta)\,\pi_{\mathrm{ref}}(y_w|x)$, which rules out degradation of chosen responses. Empirically, BPO keeps the chosen reward positive and rising during training across several loss functions and delivers the reported accuracy gains.

Load-bearing premise

The argument rests on the assumption that the balancing parameter $\alpha$ chosen on the five evaluation benchmarks transfers to new problems, and that training actually reaches a positive margin rather than merely assuming one.

Editorial extensions

If this is right

  • Under BPO, the likelihood of the chosen response is bounded below by $\exp(\gamma/\beta)\,\pi_{\mathrm{ref}}(y_w|x)$ whenever the optimized margin $\gamma$ is positive, so chosen-response degradation cannot occur in that regime.
  • The balanced margin applies to any preference optimization loss $f$, not just the logistic loss of DPO; the paper reports gains over the relative margin under hinge, squared, exponential, truncated quadratic, and Savage losses.
  • Because the balanced margin is a drop-in replacement for the margin term, adopting BPO in an existing DPO pipeline requires changing one line of code.
  • Across the five competition-level math benchmarks, BPO improves over DPO by 10.1 points with Llama-3.1-8B-Instruct and by 11.7 points with Qwen2.5-Math-7B, and it also outperforms IPO, SLiC, and Cal-DPO in the same setup.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to make $\alpha$ adaptive per pair or per batch; the paper's own sensitivity analysis shows the optimal balance shifts with the loss, so a fixed scalar may not transfer across data distributions.
  • The same margin swap is worth testing in on-policy preference learning; the authors list this as a limitation, but the gradient mechanism itself does not depend on offline data.
  • The DCR diagnosis suggests that earlier fixes such as DPOP and Cal-DPO are indirect ways of enforcing that the chosen reward stays positive; BPO achieves the same end by construction, which may explain why those methods only partially closed the gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Balanced Preference Optimization (BPO), a modification of the DPO family of preference optimization losses that replaces the relative reward margin r_w - r_l with a 'balanced reward margin' min(r_w, -α r_l), where α is a gap adaptor. The authors argue that this change preserves the absolute likelihood of chosen responses and mitigates the 'Degraded Chosen Responses' (DCR) issue. They provide a gradient analysis, a theorem claiming a lower bound on πθ(yw|x), and experiments on five competition-level math benchmarks using Llama-3.1-8B-Instruct and several Qwen2.5 models, reporting consistent gains over DPO and other preference optimization baselines.

Significance. If correct, the proposed change is attractive: it is a one-line modification of existing losses, is general across preference optimization functions, and the paper includes a broad empirical sweep (four base models, six loss forms) plus released code. However, the paper's main formal and empirical claims are currently not supported as stated: the printed loss in Eq. (5) has a sign error relative to its own gradient in Eq. (6); Theorem 1 is a tautological unpacking of the definition of min rather than an optimization guarantee; the abstract's headline gain of +11.7% compares BPO against the untrained base model rather than against DPO; and α is tuned on the evaluation benchmarks without error bars or held-out validation. With these issues fixed, the core idea remains a plausible, testable contribution.

major comments (4)
  1. [§2.1, Eq. (5); Appendix A.1, Eq. (9)] The printed objective and the gradient are inconsistent. For the logistic log function f(z)=log(1+e^{-z}), which is decreasing in z, minimizing L = -E[f(min(βr_w, -αβr_l))] drives the margin toward -∞: when r_w is the minimum this decreases r_w, and when -αr_l is the minimum this increases r_l. Eq. (6) is instead the gradient of L' = E[f(min(...))], since the first case gives -E[σ(-βr_w)β∇log πθ(yw|x)]. Thus a reader implementing Eq. (5) trains the opposite objective to the one described. Please correct Eq. (5) (and Eq. (9)) or the gradient derivation so the method is unambiguously defined.
  2. [§2.2, Theorem 1; Appendix A.2] Theorem 1's proof only unpacks the definition: if min(r_w, -αr_l) ≥ γ then r_w ≥ γ and -αr_l ≥ γ, so the claimed lower bound on πθ(yw|x) is exactly the assumption, not a consequence of optimizing the BPO loss. The theorem also does not establish that the optimizer attains any positive γ; the proof in Appendix A.2 simply assumes ρb_θ ≥ γ. Please restate the result as an assumption or prove that the loss minimization achieves a positive margin.
  3. [Abstract and Table 2] The abstract's headline '+11.7%' over DPO for Qwen2.5-Math-7B is contradicted by Table 2: DPO's average is 41.0 and BPO's is 46.7, a +5.7 gain, while 35.0 is the accuracy of Qwen2.5-Math-7B-Base without any preference optimization. Please correct all gain claims to compare against the appropriate baseline.
  4. [§3.2, §3.3, and Figure 5] The gap adaptor α is selected on the same evaluation benchmarks: Figure 5 reports optimal α=0.5 for logistic and α=0.3 for hinge, and the gains in Table 3 use these choices. No held-out validation, seeds, or error bars are reported. This makes it difficult to assess whether the reported differences are statistically meaningful or an artifact of tuning on the test sets. Please provide a validation-based selection procedure or report error bars and confidence intervals.
minor comments (4)
  1. [§2.1, Eq. (4), and Figure 2] The explanation of α's effect appears reversed: with ρb = min(r_w, -αr_l), a larger α makes the rejected term more negative and thus more likely to be the active minimum, so larger α emphasizes updates to the rejected response, not 'improving chosen responses' as stated in the text and Figure 2 caption.
  2. [Abstract and §2.2] There are several editorial slips: 'Direct Preference Optimization have emerged' in the abstract, 'This lead us' in the introduction, and 'BDO loss' in §2.2 should be 'BPO loss'.
  3. [Eq. (1) and Table 3] The preference optimization function f is described loosely in the text as 'the logistic log function'; please define f explicitly (f(z)=log(1+e^{-z})) before Eq. (5) and use consistent notation in the equations.
  4. [References] Reference [6] is cited for both Qwen2.5-Math-7B-Base-SFT and Qwen2.5-7B-RAFT-Zero, but it is listed as 'The llama 3 herd of models'; please correct this citation.

Circularity Check

2 steps flagged · score 6.0 of 10

Theorem 1 is a definitional restatement of its own min-margin assumption rather than an independent guarantee, and the gap adaptor alpha is tuned on the evaluation benchmarks whose scores are then reported as BPO's results; the core benchmark comparison itself is not circular.

  1. self definitional [Section 2.2, Theorem 1 and Appendix A.2, Eqs. (17)-(22)]
    "Let γ be the maximized margin in the balanced reward margin constraint ρbθ ≥ γ. Under BPO, the likelihood of the chosen response satisfies: πθ(yw |x) ≥ exp(γ/β) πref(yw |x). ... For this minimum to be at least γ, both components must individually satisfy: rw ≥ γ and −αrl ≥ γ. ... log πθ(yw|x)/πref(yw|x) ≥ γ/β."

    The theorem's premise 'ρbθ ≥ γ' already contains the conclusion. By the definition of the balanced margin in Eq. (4), ρbθ = min(rw, -αrl), so rw ≥ ρbθ ≥ γ; exponentiation then gives Eq. (8) verbatim. Appendix A.2 only unpacks the definition of min and never shows that optimizing the BPO loss attains any positive γ. The claimed lower bound is thus equivalent to the imposed constraint, not a consequence of the BPO objective, so the 'theoretical guarantee' reduces by construction to its own assumption.

  2. fitted input called prediction [Section 3.3, Figure 5 and Table 3 caption]
    "As shown in Figure 5, BPO performance initially increases and then decreases as α increases. For the logistic log loss, the best performance is achieved when α=0.5. In contrast, for the hinge loss, the optimal performance is observed at α=0.3. ... Gap adaptor is set to 0.3 in this experiment."

    The gap adaptor α is selected by maximizing the reported average accuracy on the same five evaluation benchmarks (AIME2024, MATH500, AMC2023, MinervaMath, OlympiadBench) that are later presented as BPO's headline gains. No held-out or development split is described for choosing α, so the values are fitted to the test benchmarks. The resulting +10.1% and +11.7% improvements are therefore partly a test-set-selected optimum of a BPO hyperparameter presented as a fixed predictive result, rather than an out-of-sample prediction. This is selection via fitted input, not algebraic circularity, and the benchmark comparison retains empirical content.

full rationale

The core algorithmic idea—replacing the relative margin rw - rl with the balanced margin min(rw, -αrl) and the corresponding benchmark comparisons—is a genuine, externally testable modification, and the reported tables are not themselves products of a self-citation chain. No load-bearing self-citation or imported uniqueness theorem appears. However, the paper's stated 'theoretical guarantee' is circular in the strict sense: Theorem 1 assumes ρbθ ≥ γ and then derives rw ≥ γ from the definition of min, so Eq. (8) is a restatement of the assumption rather than an independently established property of BPO. Additionally, the gap adaptor α is tuned directly on the evaluation benchmarks whose scores are reported, making the headline gains partially a product of test-set selection. The separate sign inconsistency between Eq. (5) and Eq. (6) is a correctness concern, not a circularity, and is not counted here. Overall, the central empirical finding has independent content, so the score is 6 rather than higher.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method rests on the DPO/BT reward parameterization (standard), an unproven difficulty-mismatch heuristic, an assumed attainment of the margin in Theorem 1, and a test-set-tuned hyperparameter α. No new physical entities are introduced.

free parameters (2)
  • α (gap adaptor) = 0.3 (hinge) / 0.5 (logistic)
    Chosen by scanning on the evaluation benchmarks (Figure 5 and Table 3); different values per loss function, no held-out selection.
  • β (DPO inverse temperature) = not reported
    Standard DPO hyperparameter controlling reward scale; value not given in the paper, yet it affects the min switch and the loss scale.
assumptions (4)
  • domain assumption Preference distribution is modeled by p(yw ≻ yl | x) = f(β log(πθ(yw)/πref(yw)) − β log(πθ(yl)/πref(yl))) with f = logistic log (Bradley-Terry).
    Standard DPO assumption inherited from Rafailov et al.; the paper restates it in Eq (1).
  • ad hoc to paper Optimizing the balanced loss achieves a finite margin γ such that ρb ≥ γ (Theorem 1).
    The proof assumes γ is 'the maximized margin' but no argument shows the optimization reaches any positive margin; the guarantee is conditional on an unproven attainment.
  • ad hoc to paper Lowering the probability of rejected responses is far easier than raising chosen-response probability (difficulty mismatch).
    This heuristic in Section 1 motivates DCR but is not measured or proven; the entire motivation rests on it.
  • domain assumption α in (0,1] adequately controls the balance; no theory prescribes its value.
    α is an added hyperparameter; the paper provides no principled selection rule, only empirical tuning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BPO: Revisiting Preference Modeling in Direct Preference Optimization." pith.science (2026). https://pith.science/paper/XEFLY7S2

@misc{pith2026250603557,
  author       = {Pith},
  title        = {Pith review of: BPO: Revisiting Preference Modeling in Direct Preference Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XEFLY7S2}},
  note         = {Machine review of arXiv:2506.03557}
}
read the original abstract

Direct Preference Optimization (DPO) have emerged as a popular method for aligning Large Language Models (LLMs) with human preferences. While DPO effectively preserves the relative ordering between chosen and rejected responses through pairwise ranking losses, it often neglects absolute reward magnitudes. This oversight can decrease the likelihood of chosen responses and increase the risk of generating out-of-distribution responses, leading to poor performance. We term this issue Degraded Chosen Responses (DCR).To address this issue, we propose Balanced Preference Optimization (BPO), a novel framework that dynamically balances the optimization of chosen and rejected responses through two key components: balanced reward margin and gap adaptor. Unlike previous methods, BPO can fundamentally resolve DPO's DCR issue, without introducing additional constraints to the loss function. Experimental results on multiple mathematical reasoning tasks show that BPO significantly outperforms DPO, improving accuracy by +10.1% with Llama-3.1-8B-Instruct (18.8% to 28.9%) and +11.7% with Qwen2.5-Math-7B (35.0% to 46.7%). It also surpasses DPO variants by +3.6% over IPO (43.1%), +5.0% over SLiC (41.7%), and +3.1% over Cal-DPO (43.6%) on the same model. Remarkably, our algorithm requires only a single line of code modification, making it simple to implement and fully compatible with existing DPO-based frameworks.

Figures

Figures reproduced from arXiv: 2506.03557 by the authors.

Figure 1
Figure 1. Overall performance across five competition-level benchmarks (AIME2024, MATH500, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. In DPO, the rewards for chosen responses can drop below zero, whereas in our BPO, they [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The gradient distributions under the logistic log preference optimization function are [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Relative Reward Margin vs. Balanced Reward Margin under different preference optimiza [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Influence of the gap adaptor α on model performance. Optimal performance is achieved when the gap adaptor is set to a moderate value. Moderately relaxing the constraints on re￾jected responses can enhance model perfor￾mance. We analyzed the impact of the gap adaptor (α…
Figure 6
Figure 6. Figure 6: The reward dynamics of both chosen and rejected responses, along with the evolution of [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 14 canonical work pages

  1. [1]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Rémi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, International Conference on Artificial Intelligence and Statistics, 2-4 May 2024, Palau de Con...

    2024

  2. [2]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, and et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  3. [3]

    Bradley and Milton E

    Ralph A. Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika, 39(3/4):324–345, 1952

    work page 1952

  4. [4]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Garnett, 8 editors,Advances in Neural Information Processing Systems 30: Annual Conference on Neura...

    work page 2017

  5. [6]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and et al. The llama 3 herd of models. CoRR, abs/2407.21783, 2024

    arXiv 2024

  6. [7]

    Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024

    work page 2024

  7. [8]

    Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceed...

    work page 2024

  8. [9]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung, editors,Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021...

    2021

Show all 29 references
  1. [10]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290, 2025

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290, 2025

    2025 arXiv

  2. [11]

    Goucher, and et al

    Aaron Hurst, Adam Lerer, Adam P. Goucher, and et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  3. [12]

    Smith, Yejin Choi, and Hanna Hajishirzi

    Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, and Hanna Hajishirzi. Unpacking DPO and PPO: disentangling best practices for learning from preference feedback. In Amir Globersons, Lester Mackey, Danielle Belgr...

    2024

  4. [13]

    Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V . Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning prob- lems with language mo...

    2022

  5. [14]

    Mitigating the alignment tax of RLHF

    Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, and Tong Zhang. Mitigating the alignment tax of RLHF. In Yaser Al- Onaizan, Mohit Bansal, ...

    2024

  6. [15]

    Liu, and Jialu Liu

    Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11,

    2024

  7. [16]

    Online merg- ing optimizers for boosting rewards and mitigating tax in alignment.CoRR, arXiv preprint arXiv:2405.17931, 2024

    Keming Lu, Bowen Yu, Fei Huang, Yang Fan, Runji Lin, and Chang Zhou. Online merg- ing optimizers for boosting rewards and mitigating tax in alignment.CoRR, arXiv preprint arXiv:2405.17931, 2024

    2024 arXiv

  8. [17]

    Mankowitz, Doina Precup, and Bilal Piot

    Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, an...

    2024

  9. [18]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, and et al. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Information Processing Systems 35: Annual Conference on ...

    2022

  10. [19]

    Smaug: Fixing failure modes of preference optimisation with dpo-positive.arXiv preprint arXiv:2402.13228, 2024

    Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive.arXiv preprint arXiv:2402.13228, 2024

    2024 arXiv

  11. [20]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, e...

    2023

  12. [21]

    Insights into alignment: Evaluating DPO and its variants across multiple tasks.arXiv preprint arXiv:2404.14723, 2024

    Amir Saeidi, Shivanshu Verma, and Chitta Baral. Insights into alignment: Evaluating DPO and its variants across multiple tasks.arXiv preprint arXiv:2404.14723, 2024

    2024 arXiv

  13. [22]

    Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin...

    2020

  14. [23]

    Generalized preference optimization: A unified approach to offline alignment

    Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. InForty-first International C...

    2024

  15. [24]

    Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G. Honavar. Cal-dpo: Calibrated direct preference optimization for language model alignment. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors...

    2024

  16. [25]

    Contrastive preference optimization: Pushing the bound- aries of LLM performance in machine translation

    Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the bound- aries of LLM performance in machine translation. InForty-first International Conference on Machine Learn...

    2024

  17. [26]

    Is DPO superior to PPO for LLM alignment? A comprehensive study

    Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO superior to PPO for LLM alignment? A comprehensive study. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. Ope...

    2024

  18. [27]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    An Yang, Baosong Yang, Beichen Zhang, and et al. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024. 10

    2024 arXiv

  19. [28]

    Limitations

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. Slic- hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023. 11 A Derivations and Proofs A.1 Gradient of BPO Loss We consider the following BPO loss fu...

    2023 arXiv

  20. [30]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2025

  21. [2024]

    OpenReview.net, 2024. 9

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.