REVIEW 4 major objections 6 minor 1 cited by
DPO-Shift: Shifting the Distribution of Direct Preference Optimization
T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read DPO-Shift adds a tunable factor $f(\lambda)$ to the rejected term of the DPO loss and proves this converts likelihood displacement into a controlled trade-off: chosen-probability gains at the price of a smaller reward margin.
desk verdict A one-line DPO variant with solid empirical ablations but a broken proof and an overstated abstract; the trade-off is real, the theory as stated is not. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the modified Bradley-Terry loss $L_{\mathrm{DPO-Shift}}(\pi)=-\mathbb{E}[\log \sigma(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} - f(\lambda) \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)})]$, which inserts the parameter function $0<f(\lambda)<1$ as a multiplier on the rejected response's reward. The analysis proceeds by comparing one gradient step of this objective against one step of DPO: the gap in chosen log-probability and the gap in the smoothed reward-margin indicator factor as $(1-f(\lambda))u_1$ and $(1-f(\lambda))u_2$, with $u_1$ and $u_2$ defined as expectations of gradient inner products $\nabla_\theta \log \pi_\theta(y_l|x)^\top \nabla_\theta \log \pi_\theta(y_w|x)$ and related terms. Because the empirically measured signs are $u_1>0$ and $u_2<0$ on the SFTed Llama 3-8B / UltraFeedback configuration, the factor $(1-f(\lambda))>0$ directly yields the paper's core conclusion: an improvement in chosen probability that costs reward margin, with the size of both effects set by $f(\lambda)$.
What would settle it
Compute $u_1$ and $u_2$ on a different model family and preference dataset—say a Pythia-style checkpoint with the HH-RLHF dataset—at several training checkpoints; if any setting yields $u_1 \le 0$ or $u_2 \ge 0$, the paper's guarantee that $f(\lambda)<1$ improves chosen probability while reducing reward margin fails for that setting, and the recommended $f(\lambda)=0.95$ would need to be revisited.
Extended reading notes
Core claim
On its own terms, the paper establishes that the likelihood displacement of DPO can be mitigated by a one-line change to the loss: replace the DPO objective's rejected log-ratio with $f(\lambda)\log(\pi_\theta(y_l|x)/\pi_{\mathrm{ref}}(y_l|x))$, keeping $0<f(\lambda)<1$. Theorem 2.1 shows that after a single gradient step, the gap between DPO-Shift and DPO in chosen log-probability is exactly $(1-f(\lambda))u_1$, and the gap in the smoothed reward-margin indicator is $(1-f(\lambda))u_2$, where $u_1$ and $u_2$ are expectations over inner products of the chosen and rejected gradient log-probabilities. Measured on an SFTed Llama 3-8B with UltraFeedback, $u_1>0$ and $u_2<0$, so every $f(\lambda)<1$ improves the chosen probability at the expense of the reward margin. The paper further shows the trade-off magnitude scales with $1-f(\lambda)$: values close to 1 give near-DPO reward margins with a visible chosen-probability gain, while small values push chosen probability up but reduce reward accuracy and can overfit to chosen responses. The authors conclude the method mitigates likelihood displacement, is controlled by a scalar schedule that can be fixed or varied linearly over training, and outperforms DPO in downstream win-rate comparisons and perplexity.
Load-bearing premise
The conclusion that any $f(\lambda)<1$ improves the chosen probability rests on the measured sign pattern $u_1>0$ and $u_2<0$, which was obtained on a single model (SFTed Llama 3-8B) and a single dataset (UltraFeedback test set); if that sign pattern is not generic across models, datasets, and training stages, the recommended choice of $f(\lambda)<1$ could reduce chosen probability instead of improving it.
Editorial extensions
If this is right
- With $f(\lambda)$ close to 1 (e.g., 0.95–0.99), DPO-Shift raises the chosen probability relative to DPO while keeping the reward margin and reward accuracy nearly unchanged, directly mitigating likelihood displacement.
- Smaller $f(\lambda)$ increases the chosen probability further but progressively reduces the reward margin and reward accuracy; the paper observes that very small values can overfit to chosen responses and degrade preference alignment.
- The trade-off is explicitly quantified by $(1-f(\lambda))u_1$ and $(1-f(\lambda))u_2$, so practitioners can choose $f(\lambda)$—or a linear schedule over training—to hit a desired balance between likelihood and margin.
- On downstream evaluation, DPO-Shift beats DPO in a designed win-rate comparison (e.g., 72.15% vs 27.85% with linear_decrease $\lambda_{min}=0.95$ on Llama 3-8B / UltraFeedback) and produces lower perplexity on chosen responses (4.475 vs 18.996).
- The modification requires no dataset filtering, no reference-model retraining, and no extra training stages; it is a drop-in change to the DPO loss.
Reading between the lines
- The sign of $u_1$ should track the semantic similarity of chosen/rejected pairs: on datasets with mostly dissimilar pairs, $u_1<0$ may hold, and the paper's own analysis implies $f(\lambda)>1$ could then improve both chosen probability and reward margin simultaneously—a regime worth testing systematically rather than writing off as a crash risk.
- A per-sample adaptive choice of $f(\lambda_i)$, assigned according to the sign of $u^i_1$ for each training pair, is the natural next step the authors leave as future work; one could test it by computing $u^i_1$ on the training set and applying $f(\lambda_i)<1$ where $u^i_1>0$ and $f(\lambda_i)>1$ where $u^i_1<0$.
- The sign-pattern measurement is the paper's empirical linchpin, so independently reproducing $u_1>0$, $u_2<0$ across architectures and preference datasets would either widen or shrink the domain where the recommended $f(\lambda)<1$ is safe.
- Because likelihood displacement is also addressed by data filtering approaches, DPO-Shift's soft weighting of the rejected term may compound with filtering—e.g., DPO-Shift on a filtered DPO-Positive-style dataset might recover margin loss while keeping chosen-probability gains.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DPO-Shift, a modification of the DPO objective in which the rejected-response log-ratio term is multiplied by a function f(λ)∈(0,1). The central theoretical claim (Theorem 2.1, contribution C.2) is that after one gradient step the gap in chosen log-probability and in a smoothed reward-margin indicator between DPO-Shift and DPO factorizes as (1−f(λ))u1 and (1−f(λ))u2; combined with empirically measured signs u1>0 and u2<0, this is used to conclude that f(λ)<1 improves chosen probability at the cost of a reduced reward margin. The paper validates the predicted trade-off with ablations on Llama 3-8B and Qwen 2-7B over UltraFeedback and Capybara, and reports win-rate comparisons against DPO and other preference optimization methods.
Significance. If the theoretical claim were established, DPO-Shift would be a simple and practically useful remedy for likelihood displacement, a well-documented failure mode of DPO. The paper's strengths are its simple formulation, the released code, and the unusually thorough ablation over f(λ) values and multiple model/dataset combinations; the downstream comparison against SimPO, KTO, and IPO using their released checkpoints is also informative. However, the advertised 'theoretically grounded' guarantee is the main added value over the empirical observation, and the proof of Theorem 2.1 does not support the theorem as stated. The empirical sign pattern used to draw the main conclusion is measured on a single model/dataset pair, and the win-rate evaluation is partly circular, so the strongest claims in the paper are not yet supported.
major comments (4)
- [Theorem 2.1 / Appendix B] The theorem as stated is not a consequence of the proof in Appendix B. Using the proof's own update θ_{t+1}=θ_t+η(c1∇logπθ(yw|x)−c2∇logπθ(yl|x)), the one-step difference in ω1 between DPO-Shift and DPO is ηE[(c1^f−c1^0)||∇logπθ(yw|x)||² − (f c1^f−c1^0)∇logπθ(yl|x)^T∇logπθ(yw|x)], where the superscripts denote f(λ) and f=1. The proof drops the first term and thereby obtains g1=η(c1−c2)∇logπ(yl)^T∇logπ(yw); this is only legitimate when f≈1, not for all 0<f(λ)<1 as the theorem states. In addition, the theorem's g1 omits the factor η present in the proof, and the theorem's u2 in Eq. (9) omits the c(θ) factor that the proof's g2 expression contains. Finally, c(θ) in the theorem uses γ while the proof's c1 uses β, and γ is never assigned a value in the main text; the proof's remark 'use β → +∞' is not a derivation. The theorem therefore needs either a corrected statement with explicit f≈1, η, and constant factors, or a complete derivation under its stated hypotheses.
- [Section 2.2] The sign conclusion u1>0 and u2<0 is empirical, measured on one SFTed Llama 3-8B model on the UltraFeedback test set, and is not a consequence of the theorem. The manuscript does not verify the sign pattern at initialization for Qwen 2-7B, Capybara, or later training stages, so the recommendation f(λ)<1 is not established as generic. Moreover, the main text and Appendix B contradict each other: Section 2.2 reports 81.7% of the u_i^2 values as negative, while Appendix B reports 81.7% of the bracket term ∇logπ(yl)^T∇logπ(yw)−||∇logπ(yl)||² as positive; since the main text sets η1 to 1 when computing u2, these two statements cannot both be correct. This internal inconsistency weakens the empirical basis for the sign of u2.
- [Section 3.3 / Appendix A.2] The win-rate experiment is partly circular. The judge prompt instructs that 'the most important criterion is to select the response whose meaning is essentially closer to the reference answer,' and the reference answer is the chosen response whose likelihood DPO-Shift is specifically designed to increase. The evaluation metric therefore directly rewards the quantity DPO-Shift optimizes, so Table 1 does not provide an independent demonstration of better alignment quality. The perplexity comparison (4.475 vs 18.996) is similarly computed on the chosen responses of the test split, which again rewards the objective being shifted. Please re-run with a judge that does not have access to the reference answer, or report a judge based on general quality only.
- [Abstract / Contribution C.3] The abstract states that DPO-Shift demonstrates superiority on downstream tasks 'such as MT-Bench and a designed win rate experiment,' but no MT-Bench results appear anywhere in the main text or appendices. Either the MT-Bench results should be reported, or the abstract should be revised to mention only the win-rate experiment, which is the downstream evaluation actually presented.
minor comments (6)
- [Section 3.2] The sentence beginning 'A carefully chosen For example' is grammatically broken and should be rewritten.
- [Section 2.2] The notation u_i^1 and u_i^2 is hard to read; using a sample index as a subscript (e.g., u_1,i) would clarify the definitions.
- [Section 2.2] The phrase 'the first one is to fix it all along the optimization process, i.e., f(λ)=λ' is confusing because λ is never defined as a variable separate from the function argument; please define λ and the fixed strategy explicitly.
- [Appendix B] The phrase 'we abuse notation and use β → +∞ as hyper parameter' should be replaced with a precise statement about the smoothing factor γ; as written it suggests taking an infinite temperature limit that is not part of the training procedure.
- [Appendix B] The proof states that 81.7% of the computed bracket values 'turn out to be positive,' which is inconsistent with the main text's claim that 81.7% of the u_i^2 values are negative; please reconcile these numbers.
- [Table 1] Entries such as 'linear_increase 0.95' should specify whether 0.95 is λ_min or λ_max; currently the reader must infer the convention from the text.
Circularity Check
The central trade-off claim is not circularly derived: u1 and u2 are empirically measured, not defined to force the conclusion. However, the proof of Theorem 2.1 drops a nonzero term, and the downstream win-rate/perplexity evaluations are constructed around the exact chosen-likelihood quantity DPO-Shift is designed to increase.
-
other
[Appendix B, Eqs. (12)-(13) and Theorem 2.1 (Section 2.2)]
"w1(θt+1) =w1(θt) +η (c1∇θ log πθt (yw|x) − c2∇θ log πθt (yl|x))⊤ (∇θ log πθt (yw|x)) (12) = w1(θt) +η (c1 ||∇θ log πθt (yw|x)||2 − c2∇θ log πθt (yl|x)⊤ ∇θ log πθt (yw|x)) (13) then g1(t + 1) =η(c1 − c2)∇θ log π (yl|x)⊤ ∇θ log π (yw|x)"
For the DPO baseline f(λ)=1, the same expansion gives c1=c2=c, so subtracting the DPO update from Eq. (13) yields η[(c1−c)||∇ log πθ(yw)||² + (c−f c1)∇ log πθ(yl)ᵀ∇ log πθ(yw)], not the printed η(c1−c2)∇ log πθ(yl)ᵀ∇ log πθ(yw). The stated factorization g1=(1−f)u1 is obtained only after suppressing the nonzero η(c1−c)||∇ log πθ(yw)||² term and identifying c with c1, which is an f≈1 assumption. The theorem is claimed for all 0<f<1, so the proof does not establish the result; the 'without imposing additional assumptions' guarantee in C.2 is unsupported.
-
self definitional
[Section 3.3 (perplexity paragraph)]
"Perplexity quantifies how well a probability model predicts a sample of data... The perplexity of DPO and DPO-Shift trained on UltraFeedback with fixed f (λ) = 0.95 is evaluated on the chosen responses from the test split of the UltraFeedback dataset. The results are 4.475 for DPO-Shift and 18.996 for DPO, further demonstrating the potential advantage of DPO-Shift."
Perplexity of an autoregressive policy on the chosen responses is exp(−E log πθ(yw|x)/|yw|) up to a constant, so 'lower perplexity on chosen responses' is a monotone transform of the chosen log-probability that DPO-Shift is defined to increase (g1>0 under Theorem 2.1's claimed sign pattern). Reporting it as an additional downstream advantage is therefore a restatement of the optimized quantity, not an independent evaluation of the method.
1 more flagged steps
-
other
[Appendix A.2 (judge prompt) with Section 3.3 (win-rate experiment)]
"The most important criterion is to select the response whose meaning is essentially closer to the reference answer."
The reference answer is the chosen response yw of the preference dataset, and DPO-Shift's construction (Eq. (3)) plus the claimed g1>0 is specifically designed to raise log πθ(yw|x). Instructing the judge to prefer semantic closeness to the reference answer makes the win-rate comparison largely a measure of whether DPO-Shift has adopted the chosen-response distribution, which is the exact quantity that was deliberately shifted. The experiment is therefore partially self-confirming rather than an independent test of general quality.
full rationale
The core derivation is not circular by construction: u1 and u2 are measured empirically on the SFTed Llama 3-8B / UltraFeedback test set, and the claimed signs of g1 and g2 are not baked into the definitions of ω1 and ω2. However, the proof of Theorem 2.1 in Appendix B does not establish the stated factorization: from the paper's own Eq. (13), the one-step DPO-Shift vs DPO gap contains an extra η(c1−c)||∇ log πθ(yw)||² term that is silently dropped, and the remaining coefficient (c−f c1) is not (1−f)c1 unless c=c1. The theorem's 'no additional assumptions' statement is therefore unsupported; this is a proof gap rather than a circular definition, but it is the load-bearing step of claim C.2. Separately, the downstream evaluation is partially circular: perplexity on chosen responses is a monotone function of the chosen log-probability the method was built to increase, and the judge prompt makes closeness to the reference answer the primary win criterion, which rewards that same shifted distribution. The distributional and reward-margin experiments in Section 3.2 still provide independent, non-circular evidence for the empirical trade-off; the flaws are concentrated in the theoretical guarantee and in the framing of the downstream metrics. Overall circularity is moderate: the headline contribution has independent empirical content, but two of its advertised confirmations reduce to the design objective, and the claimed theorem is not derived as written.
Assumptions & free parameters
free parameters (3)
- f(λ) =
ablation range 0.5 to 0.99
- β (DPO temperature) =
not specified in text; follows SimPO config
- γ (smoothing factor in Theorem 2.1) =
unspecified; set to 1 in one empirical check
assumptions (4)
- domain assumption Bradley-Terry model of preferences
- domain assumption First-order Taylor expansion with small learning rate η
- domain assumption Semantic similarity between chosen and rejected responses in preference datasets
- ad hoc to paper The empirical sign pattern u1>0, u2<0 computed on one SFTed model and dataset generalizes
Cite this review
Pith. "Pith review of DPO-Shift: Shifting the Distribution of Direct Preference Optimization." pith.science (2026). https://pith.science/paper/43HVHJB2
@misc{pith2026250207599,
author = {Pith},
title = {Pith review of: DPO-Shift: Shifting the Distribution of Direct Preference Optimization},
year = {2026},
howpublished = {\url{https://pith.science/paper/43HVHJB2}},
note = {Machine review of arXiv:2502.07599}
}
read the original abstract
Direct Preference Optimization (DPO) and its variants have become increasingly popular for aligning language models with human preferences. These methods aim to teach models to better distinguish between chosen (or preferred) and rejected (or dispreferred) responses. However, prior research has identified that the probability of chosen responses often decreases during training, and this phenomenon is known as likelihood displacement. To tackle this challenge, in this work we introduce DPO-Shift to controllably shift the distribution of the chosen probability. Then, we show that DPO-Shift exhibits a fundamental trade-off between improving the chosen probability and sacrificing the reward margin, as supported by both theoretical analysis and experimental validation. Furthermore, we demonstrate the superiority of DPO-Shift over DPO on downstream tasks such as MT-Bench and a designed win rate experiment. We believe this study shows that the likelihood displacement issue of DPO can be effectively mitigated with a simple, theoretically grounded solution. Our code is available at https://github.com/Meaquadddd/DPO-Shift.
Figures
Figures from the paper (19 more)
Forward citations
Cited by 1 Pith paper
-
LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs
LeanPO improves Video-LLM alignment by using a reference-free average-likelihood reward, self-generated winning/losing pairs, and dynamic label smoothing, yielding gains on six video benchmarks.
Reference graph
Works this paper leans on
-
[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. ArXiv, abs/2303.08774, 2023
arXiv 2023
-
[2]
Llama 3 model card
AI@Meta. Llama 3 model card. 2024
2024
- [3]
-
[4]
A general theoretical paradigm to understand learning from human preferences
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024
2024
-
[5]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023
arXiv 2023
-
[6]
Training a helpful and harmless assistant with reinforcement learning from human feedback
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. 10
arXiv 2022
-
[7]
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324, 1952
1952
-
[8]
Evaluation metrics for language models
Stanley F Chen, Douglas Beeferman, and Roni Rosenfeld. Evaluation metrics for language models. 1998
work page 1998
Show all 41 references
-
[9]
Deep reinforcement learning from human preferences
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017
2017
-
[10]
Perplexed: Understanding when large language models are confused
Nathan Cooper and Torsten Scholak. Perplexed: Understanding when large language models are confused. arXiv preprint arXiv:2404.06634, 2024
2024 arXiv
-
[11]
UltraFeedback: Boosting language models with high-quality feedback
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. UltraFeedback: Boosting language models with high-quality feedback. In ICML, 2024
2024
-
[12]
Enhancing chat language models by scaling high-quality instructional conversations
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In EMNLP, 2023
2023
-
[14]
Kto: Model alignment as prospect theoretic optimization
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024
2024 arXiv
-
[15]
ORPO: Monolithic preference optimization without reference model
Jiwoo Hong, Noah Lee, and James Thorne. ORPO: Monolithic preference optimization without reference model. ArXiv, abs/2403.07691, 2024
2024 arXiv
-
[16]
Sdpo: Segment-level direct preference optimization for social agents
Aobo Kong, Wentao Ma, Shiwan Zhao, Yongbin Li, Yuchuan Wu, Ke Wang, Xiaoqian Liu, Qicheng Li, Yong Qin, and Fei Huang. Sdpo: Segment-level direct preference optimization for social agents. arXiv preprint arXiv:2501.01821, 2025
2025 arXiv
-
[17]
Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405.16436, 2024
2024 arXiv
-
[18]
Simpo: Simple preference optimization with a reference-free reward
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024
2024 arXiv
-
[19]
Entropy controllable direct preference optimization
Motoki Omura, Yasuhiro Fujita, and Toshiki Kataoka. Entropy controllable direct preference optimization. arXiv preprint arXiv:2411.07595, 2024
2024 arXiv
-
[20]
Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan...
2022
-
[21]
Smaug: Fixing failure modes of preference optimisation with DPO-positive
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with DPO-positive. arXiv preprint arXiv:2402.13228, 2024
2024 arXiv
-
[22]
Pre-dpo: Improving data utilization in direct preference optimization using a guiding reference model
Junshu Pan, Wei Shen, Shulin Huang, Qiji Zhou, and Yue Zhang. Pre-dpo: Improving data utilization in direct preference optimization using a guiding reference model. arXiv preprint arXiv:2504.15843, 2025
2025
-
[23]
Iterative reasoning preference optimization
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733, 2024
2024 arXiv
-
[24]
Disentangling length from quality in direct preference optimization
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. ArXiv, abs/2403.19159, 2024. 11
2024 arXiv
-
[25]
From r to Q∗: Your language model is secretly a Q-function
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to Q∗: Your language model is secretly a Q-function. arXiv preprint arXiv:2404.12358, 2024
2024 arXiv
-
[26]
Manning, and Chelsea Finn
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, 2023
2023
-
[27]
Unintentional unalignment: Likelihood displacement in direct preference optimization
Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization. arXiv preprint arXiv:2410.08847, 2024
2024 arXiv
-
[28]
Learning to summarize with human feedback
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020
2020
-
[29]
Preference fine-tuning of llms should leverage suboptimal, on-policy data
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of llms should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404.14367, 2024
2024 arXiv
-
[30]
Generalized preference optimization: A unified approach to offline alignment
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024
2024 arXiv
-
[31]
Llama 2: Open foundation and fine-tuned chat models, 2023
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...
2023
-
[32]
Rush, and Thomas Wolf
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alvaro Bartolome, Alexander M. Rush, and Thomas Wolf. The Alignment Handbook
-
[33]
Simper: A minimalist approach to preference alignment without hyperpa- rameters
Teng Xiao, Yige Yuan, Zhengyu Chen, Mingxiao Li, Shangsong Liang, Zhaochun Ren, and Vasant G Honavar. Simper: A minimalist approach to preference alignment without hyperpa- rameters. arXiv preprint arXiv:2502.00883, 2025
2025 arXiv
-
[34]
Cal-dpo: Calibrated direct preference optimization for language model alignment
Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G Honavar. Cal-dpo: Calibrated direct preference optimization for language model alignment. arXiv preprint arXiv:2412.14516, 2024
2024 arXiv
-
[35]
A systematic evaluation of large language models of code
Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1–10, 2022
2022
-
[36]
Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Ken- ton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. ArXiv, abs/2401.08417, 2024
2024 arXiv
-
[37]
Full-step-dpo: Self-supervised preference optimization with step-wise rewards for mathematical reasoning
Huimin Xu, Xin Mao, Feng-Lin Li, Xiaobao Wu, Wang Chen, Wei Zhang, and Anh Tuan Luu. Full-step-dpo: Self-supervised preference optimization with step-wise rewards for mathematical reasoning. arXiv preprint arXiv:2502.14356, 2025
2025 arXiv
-
[38]
Advancing llm reasoning generalists with preference trees
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing llm reasoning generalists with preference trees. arXiv preprint arXiv:2404.02078, 2024. 12
2024 arXiv
-
[39]
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In NeurIPS Datasets and Benchmarks Track, 2023. 13 Contents 1 Introduction 1 2 DPO- ...
2023
-
[40]
The most important criterion is to select the response whose meaning is essentially closer to the reference answer
-
[41]
Do not judge the quality of the two responses based on their length
-
[42]
Evaluate the responses based on their helpfulness, relevance, accuracy, depth, and conciseness. User’s Question: {question} Reference Answer: {ref_answer} Assistant A’s Response: {response_compare} Assistant B’s Response: {response_baseline} 15 A.3 Ablation Studies for fixed 1...
2000
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.