Pith. sign in

REVIEW 4 major objections 7 minor 61 references

Preference learning made easy: Everything should be understood through win rate

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Preference learning has a unique defensible evaluation: win rate, up to any strictly increasing transformation.

desk verdict A genuinely useful framework paper: the win-rate uniqueness theorem is correct given its axioms, but the axioms do the heavy lifting and a few appendix numbers don't reproduce. read the letter →

arxiv 2502.10505 v2 pith:WSY7RHB2 submitted 2025-02-14 cs.LG cs.CLstat.ML

Lily H. Zhang , Rajesh Ranganath This is my paper
classification cs.LGcs.CLstat.ML
keywords preferencelearningwinratewin-rateoptimizationdirectRLHFgenerativemodelalignmentprevalence-consistencyevaluationgroundedness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Preference learning trains generative models from pairwise comparison data, but unlike classification it has lacked a single conceptual anchor. This paper tries to supply one by asking what 'good' can mean using only the sampling distribution of preference data. It proves that the only evaluations consistent with both the preference signal and the prevalence of queries, model outputs, and anchors are win rates, possibly transformed by a strictly increasing function. It then reads existing methods such as RLHF, DPO, and SFT as better or worse approximations to win-rate optimization, proving that DPO lacks a key alignment property and that SFT's win-rate gain is capped by the initial model's preference diversity. The upshot is that optimization success, not the choice of the objective's target distribution, is the main predictor of practical performance.

What carries the argument

The load-bearing object is the sampling distribution of pairwise preferences (Definition 3.1) together with the axiomatic definition of a grounded evaluation (Definition 3.2). The two axioms, preference-consistency and prevalence-consistency, do the work: the first anchors the evaluation to the preference probability in the single-point case, and the second forces the evaluation to be a linear mixture-respecting functional of the query, generator, and anchor distributions. Together they imply the representation of any grounded evaluation as an expectation of $h(p(\ell=1\mid x,y_0,y))$, that is, a win rate. The framework then introduces two derived properties, win-rate correspondence and win-rate consistency, to classify objectives, and uses the variance formula of Theorem 5.1 as the exact characterization of SFT's limited gain.

What would settle it

Synthetic check: fix three responses with known preference probabilities and an initial model, compute the exact SFT target distribution from Equation (5), and measure its win rate against the initial model; if the measured win rate differs from $0.5 + 2\,\mathrm{Var}(\mathbb{E}_{p(y_0\mid x)}[p(\ell=1\mid x,y_0,y_1)])$, Theorem 5.1 is wrong. For the uniqueness claim, one would need to exhibit an evaluation that satisfies both consistency axioms and is not equal to any h-win rate; Proposition 3.3 asserts that none exists.

Watch

Extended reading notes

Core claim

The paper's central claim is that within the pairwise preference sampling setup, defined by a query distribution $p(x)$, two candidate generators $p(y_0\mid x)$ and $p(y_1\mid x)$, and a preference classifier $p(\ell=1\mid x,y_0,y_1)$, the only evaluation $\phi$ of a generative model that is grounded in the preference data itself is the h-win rate. Grounded means two things: preference-consistency, so that in the deterministic base case the evaluation is a strictly increasing function $h$ of the preference probability, and prevalence-consistency, so that the evaluation is linear in mixtures of the generator, the query distribution, and the anchor distribution. Proposition 3.3 states that for discrete $p$, such an evaluation exists if and only if $\phi_{p(y_0\mid x)}(p(y\mid x), E) = \mathbb{E}_{p(x)}\mathbb{E}_{p(y\mid x)}\mathbb{E}_{p(y_0\mid x)}[h(p(\ell=1\mid x,y_0,y))]$. The paper then classifies preference learning methods as win-rate optimization (WRO) or not: RLHF is KL-regularized WRO with $h=\mathrm{logit}$ under the Bradley-Terry assumption, NLHF is a WRO game, while DPO and SFT on preferred samples are not WRO. DPO fails win-rate correspondence, SFT fails both correspondence and consistency, and Theorem 5.1 gives SFT's exact win-rate gain over the starting model as $0.5 + 2\,\mathbb{E}_x\,\mathrm{Var}_{p(y_1\mid x)}(\mathbb{E}_{p(y_0\mid x)}[p(\ell=1\mid x,y_0,y_1)])$. Empirically, the paper reports that WRO variants underperform relative to their theoretical promise because of optimization difficulty, and that training loss correlates with win rate far more strongly than design choices like $h$, $\beta$, or the preference-classifier estimator.

Load-bearing premise

The uniqueness theorem depends on prevalence-consistency, which requires the evaluation to be linear in mixtures of the generator, query distribution, and anchor distribution; if an evaluation may respond non-linearly to these distributions, evaluations other than h-win rate can be grounded, and the proof no longer applies.

Editorial extensions

If this is right

  • If the uniqueness result is right, win rate, or any strictly increasing transform of it, should replace ad hoc rewards and ranking accuracies as the canonical evaluation of aligned generative models unless extra assumptions are imported.
  • All objectives of the form in Equation (2), including KL-regularized RLHF, inherit win-rate correspondence and consistency; under the Bradley-Terry assumption they share a common optimal solution that maximizes win rate over every anchor distribution.
  • DPO and other direct alignment algorithms can improve their own loss while decreasing win rate, even in fully online settings, so their training curves should be monitored with win-rate checkpoints rather than loss.
  • SFT on preferred samples cannot reach maximum win rate when the initial model has more than two supported responses; its gain is exactly the variance term in Theorem 5.1, so diversity of candidates in preference space is the main lever.
  • Empirically, better optimization, measured by lower training loss, predicts higher win rate across WRO-KL variants, so effort spent improving policy-gradient optimization should matter more than tuning $h$, $\beta$, or the classifier estimator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the theorem leaves the choice of $h$ free, so model rankings can in principle change under different strictly increasing transforms; a robust evaluation practice should check win-rate comparisons across several $h$ rather than one.
  • Editorial inference: the SFT variance formula suggests a concrete recipe, sampling many diverse candidates before filtering, that should raise win rate according to the covariance term in Theorem 5.2.
  • Editorial inference: if optimization difficulty is the binding constraint, variance-reduced or variational estimators for WRO objectives should improve win rates more than any change to the objective's target distribution; this is directly testable.
  • Editorial inference: the uniqueness proof applies to pairwise comparisons, and whether an analogous 'only X is grounded' theorem holds for single-item ratings, listwise rankings, or continuous feedback is not settled by this paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes a conceptual framework for preference learning centered on win rate. It defines an evaluation function over a generative model, an anchor distribution, and a query-preference environment, and imposes two axioms: preference-consistency (at deterministic atoms, the evaluation is a strictly increasing transform of the preference probability) and prevalence-consistency (the evaluation is linear in mixtures of the model, query, and anchor distributions). The central theoretical result, Proposition 3.3, states that the only grounded evaluations are h-win rates, i.e., expectations of h applied to the preference probability under the query, model, and anchor distributions. The paper then classifies preference learning algorithms as win-rate optimization (WRO) or non-WRO, proves benefits of WRO such as win-rate correspondence and consistency, shows that DPO and SFT fail these properties, derives formulas for SFT's win-rate improvement in terms of preference variance, and reports experiments comparing RLHF, DPO, SFT, and many WRO-KL variants. The empirical conclusion is that optimization success, rather than the choice of target distribution, is the main driver of win rate.

Significance. If the axiomatic step is accepted, the paper provides a clean justification for win rate as the primary evaluation in preference learning and offers a useful taxonomy of methods. The proof of Proposition 3.3 is straightforward, but the characterization is conceptually sharp and likely to be influential. The paper also contributes a general family of WRO objectives beyond RLHF and NLHF, a theorem characterizing SFT's win-rate ceiling in terms of preference variance, and a systematic experimental comparison of WRO variants. The empirical claim that training loss predicts win rate better than choices of h, beta, or anchor is practically relevant if it holds. The main weaknesses are that the uniqueness theorem is conditional on the linearity/independence axiom, which is not derived from the sampling distribution itself, and that the empirical evaluation uses the same learned judge model both to define the preference environment and to measure win rate. Neither weakness invalidates the theoretical result, but both need to be stated more carefully.

major comments (4)
  1. [Appendix B, Lemma B.1 and Proposition 3.3] The uniqueness claim is conditional on the prevalence-consistency axioms 2a-2c, which require the evaluation to be linear in mixtures of the generator, query, and anchor distributions. The paper's language elsewhere, such as 'the only evaluation that can matter' and 'grounded in the preference data sampling distribution itself,' overstates what is proven. The theorem establishes uniqueness within the class of evaluations satisfying the stated axioms, but it does not follow from the sampling distribution alone; the linearity axiom is a substantive normative assumption similar to an independence axiom in expected-utility theory. Many monotone but nonlinear evaluations, such as risk-sensitive or rank-dependent functionals, would satisfy preference-consistency and an intuitive monotonicity requirement while not being h-win rates. I recommend that the authors explicitly state that the theorem characterizes evaluations satisfying Definition 3.2, and that they discuss whether those axioms should be regarded as canonical rather than merely convenient.
  2. [Section 6.1 and Figure 2] The proof of Lemma B.1 is by induction on the size of a finite support, but Proposition 3.3 is stated for discrete support, which normally includes countably infinite sets. The reverse direction of the proof applies prevalence-consistency repeatedly to decompose the evaluation over the supports of the query, generator, and anchor distributions; for countable support this requires a continuity or dominated-convergence assumption. The appendix notes that a continuous-density extension would need 'some notion of convergence,' but it does not address the countable-support case. Either restrict Proposition 3.3 to finite support or add an explicit continuity condition to Definition 3.2 and extend the proof to countable support.
  3. [Section 6.3] The empirical evaluation uses the same learned judge model to relabel the preference data, to define the preference classifier used in training, and to compute win rate after training. The paper calls this judge an 'oracle,' but it is a Pythia-2.8b model finetuned on the same pairwise data, with 68.8% per-row agreement with human labels. The empirical results therefore measure performance with respect to the judge's preferences, not human preferences. The conclusions about practical performance, such as RLHF underperforming relative to expectations and optimization success being a first-order factor, should be qualified as statements about this proxy preference environment. The label 'oracle' is misleading and should be changed or explicitly defined.
  4. [Section 6.3] The claim that training loss correlates more with test win rate than any of the target-distribution design choices is supported by a Spearman rank-correlation p-value for loss and Kolmogorov-Smirnov p-values for the factor levels. These are not directly comparable quantities: a low Spearman p-value for loss does not by itself show that loss is a stronger predictor than the design choices. To substantiate the claim that optimization success predicts performance better than choices affecting the objective's solution, the paper should report an effect size or fit a joint model that includes both training loss and the design choices as predictors of win rate.
minor comments (7)
  1. [Abstract and Section 1] The abstract and introduction should qualify 'the only evaluation' with 'under Definition 3.2' or 'within the proposed axiomatic framework,' since the uniqueness result depends on the prevalence-consistency axioms.
  2. [Definition 3.2] The quantifier on h in condition 1 is ambiguous. The definition should say 'there exists a strictly increasing h such that for every singleton atom...' so that h is a global function rather than an input to the definition.
  3. [Equation (1)] The notation h · p(ℓ = 1 | x, y0, y) should be explicitly introduced as function composition h(p(ℓ = 1 | x, y0, y)), to avoid confusion with scalar multiplication.
  4. [Appendix F.1] The expression p(x) = 1 [1] [x = x'] contains a stray [1]; it should be p(x) = 1[x = x'] or a point-mass notation.
  5. [Figure 1] The sentence 'Any pair of points that would form a line segment with a positive slope form a setting of θ, θ′ such that win rate correspondence does not hold' is hard to parse; it should clarify that the failure occurs when one point has lower DPO loss and lower win rate than another.
  6. [Section 7] The ordering RLHF > DPO > SFT mixes unregularized WRO properties for RLHF with regularized properties for DPO; the sentence should state that the ordering is based on the specific theoretical properties defined in Sections 4 and 5, not on all variants of each method.
  7. [Throughout] There are several typos, including 'pariwise' in Section 3, 'prevalance' in Appendix B, and 'curent' in Section 7; these should be corrected in a final pass.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the uniqueness theorem is an axiomatic representation result, not a reduction to its inputs.

full rationale

The paper's central claim (Proposition 3.3) is an axiomatic characterization: it proves that any evaluation satisfying the stated preference-consistency and prevalence-consistency axioms must be an h-win rate. The axioms in Definition 3.2 do not mention win rate, and the proof in Appendix B derives the expectation form via Lemma B.1 rather than assuming it. The linearity conditions (2a-2c) are explicit structural assumptions; whether they are the right assumptions for 'grounded' is a modeling or scope question, not a circular step. The proof is self-contained and does not rely on any cited result for its main equivalence. The empirical section uses the same judge model to relabel preferences and to evaluate win rate, which is a methodological limitation and a self-referential evaluation loop, but it is not a fitted parameter renamed as a prediction and does not force the empirical conclusions by construction. Self-citations, such as Chen et al. (2024) and Ranganath et al. (2014), appear only as supporting references for context or auxiliary characterizations and are not load-bearing for the theorems. Theorems 5.1 and 5.2 generalize prior best-of-n and filtering results rather than renaming them. Overall, the derivation chain is logically sound relative to its stated axioms, with no step that reduces to its own inputs.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

No new physical or ontological entities are introduced. h-win rate is a definitional variant of existing win rate, not a new entity. The free parameters listed are experimental hyperparameters, not fitted constants in the theoretical claims.

free parameters (3)
  • KL regularization strength β = swept 1, 0.1, 0.01, 0.001
    Chosen by the authors in experiments; not fitted to a target, but the empirical conclusions depend on the sweep across β.
  • Judge temperature T = 0.2
    Hand-chosen sharpening of judge preference probabilities in Appendix K; affects the preference environment used for both relabeling and win rate evaluation.
  • DPO β in counterexample = 1
    Set to 1 in Appendix F.2; the numerical counterexample values depend on this choice.
assumptions (6)
  • domain assumption Preference-consistency: in the atomic case the evaluation is an increasing function h of the true preference probability (Definition 3.2.1).
    This equates 'respecting preferences' with being monotonically aligned with p(ℓ=1) when everything is deterministic; acceptable but normative.
  • domain assumption Prevalence-consistency: the evaluation is linear in mixtures of generator, query, and anchor distributions (Definition 3.2.2a-c).
    This is the load-bearing axiom for Proposition 3.3; non-linear evaluations are excluded by fiat. The paper gives intuitive defenses for each sub-condition but does not derive linearity.
  • standard math Discrete support of all distributions for Proposition 3.3; continuous extension requires an additional weak-convergence assumption on ϕ.
    Noted at the end of Appendix B. The main applications to language models are discrete-sequence distributions, so this is a technical caveat.
  • domain assumption Bradley-Terry assumption: p(ℓ=1|x,y0,y1) = σ(r(x,y1)-r(x,y0)) for some reward r.
    Used in Proposition 4.1 (all h-WRO share the same optimum) and in the derivation that RLHF is a WRO instance. The HH dataset is said not to satisfy it.
  • domain assumption Realizability: the hypothesis class induced by θ contains all optimal distributions (Proposition 4.1 formal proof).
    Without realizability, the equivalence of optima across h could fail due to approximation error.
  • domain assumption Anchor equals the initial model p(y0|x)=p(y1|x) in Theorems 5.1 and 5.2.
    This is stated in the theorem conditions; it sets the win rate baseline at 0.5 and yields the variance and covariance formulas.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Preference learning made easy: Everything should be understood through win rate." pith.science (2026). https://pith.science/paper/WSY7RHB2

@misc{pith2026250210505,
  author       = {Pith},
  title        = {Pith review of: Preference learning made easy: Everything should be understood through win rate},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WSY7RHB2}},
  note         = {Machine review of arXiv:2502.10505}
}
read the original abstract

Preference learning, or the task of aligning generative models to preference comparison data, has yet to reach the conceptual maturity of classification, density estimation, etc. To close this gap, this work presents a framework to understand preference learning starting from the sampling distribution of pairwise preference data. First, we prove that the only evaluation of a generative model that respects both preferences and prevalences in the data distribution is a form of win rate, justifying win rate as the focal point to understand preference learning. We then analyze preference learning methods as win rate optimization (WRO) or non-WRO. We present novel instances of WRO beyond existing examples (RLHF, NLHF) and identify two key theoretical benefits of all such methods. We prove that common non-WRO methods like DPO and SFT on preferred samples lack these properties and suggest ways to mitigate such theoretical limitations. We also show that WRO underperforms in practice due optimization difficulties and that optimization success predicts performance better than choices which affect the objective's solution. Our analysis highlights best practices for existing methods and provides recommendations for future research, guided by the principle that one should either align non-WRO methods more closely with WRO or improve the optimization of WRO objectives.

Figures

Figures reproduced from arXiv: 2502.10505 by the authors.

Figure 1
Figure 1. Plot of win rate vs. DPO loss for different [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Expected versus observed win rates of RLHF, [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Train loss vs. test win rate across all settings [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Different preference learning objectives have different target distributions. Consider the initial setting in (a). [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Plot of win rate vs. DPO loss for different [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 40 canonical work pages

  1. [1]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms

    Ahmadian, A., Cremer, C., Gall \'e , M., Fadaee, M., Kreutzer, J., \"U st \"u n, A., and Hooker, S. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv, 2024

    work page 2024

  2. [2]

    Calibration and consistency of adversarial surrogate losses

    Awasthi, P., Frank, N., Mao, A., Mohri, M., and Zhong, Y. Calibration and consistency of adversarial surrogate losses. Advances in Neural Information Processing Systems, 34: 0 9804--9815, 2021

    work page 2021

  3. [3]

    Multi-class h -consistency bounds

    Awasthi, P., Mao, A., Mohri, M., and Zhong, Y. Multi-class h -consistency bounds. NeurIPS, 2022

    work page 2022

  4. [4]

    G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R

    Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. A general theoretical paradigm to understand learning from human preferences. arXiv, 2023

    work page 2023

  5. [5]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv, 2022

    work page 2022

  6. [6]

    G., Bradley, H., O'Brien, K., Hallahan, E., Khan, M

    Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and van der Wal, O. Pythia: A suite for analyzing large language models across training and scaling. ArXiv, abs/2304.01373, 2023. URL https://api.semanticscholar.org/CorpusID:257921893

    arXiv 2023

  7. [7]

    Quantile Filtered Imitation Learning

    Brandfonbrener, D., Whitney, W. F., Ranganath, R., and Bruna, J. Quantile filtered imitation learning. arXiv preprint arXiv:2112.00950, 2021

    work page Pith review arXiv 2021

  8. [8]

    Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition

    Bridle, J. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. Neurocomputing, 1990

    work page 1990

Show all 61 references
  1. [9]

    A., Richemond, P

    Calandriello, D., Guo, D., Munos, R., Rowland, M., Tang, Y., Pires, B. A., Richemond, P. H., Lan, C. L., Valko, M., Liu, T., Joshi, R., Zheng, Z., and Piot, B. Human alignment of large language models through online preference optimisation, 2024. URL https://arxiv.org/abs/2403.08635

    2024 arXiv

  2. [10]

    H., Chen, X., Zhang, Q., Ranganath, R., and Cho, K

    Chen, A., Malladi, S., Zhang, L. H., Chen, X., Zhang, Q., Ranganath, R., and Cho, K. Preference learning algorithms do not learn preference rankings. NeurIPS, 2024

    2024

  3. [11]

    F., Leike, J., Brown, T

    Christiano, P. F., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. ArXiv, abs/1706.03741, 2017

    2017 arXiv

  4. [12]

    Raft: Reward ranked finetuning for generative foundation model alignment

    Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Reward ranked finetuning for generative foundation model alignment. TMLR, 2023

    2023

  5. [13]

    Understanding dataset difficulty with v-usable information

    Ethayarajh, K., Choi, Y., and Swayamdipta, S. Understanding dataset difficulty with v-usable information. In ICML, 2022

    2022

  6. [14]

    Bonbon alignment for large language models and the sweetness of best-of-n sampling

    Gui, L., Gârbacea, C., and Veitch, V. Bonbon alignment for large language models and the sweetness of best-of-n sampling. NeurIPS, 2024

    2024

  7. [15]

    L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., Macherey, W., Doucet, A., Firat, O., and de Freitas, N

    Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., Macherey, W., Doucet, A., Firat, O., and de Freitas, N. Reinforced self-training (rest) for language modeling. arXiv, 2023

    2023

  8. [16]

    Direct language model alignment from online ai feedback, 2024

    Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., Ferret, J., and Blondel, M. Direct language model alignment from online ai feedback, 2024. URL https://arxiv.org/abs/2402.04792

    2024 arXiv

  9. [17]

    f -po: Generalizing preference optimization with f -divergence minimization

    Han, J., Jiang, M., Song, Y., Leskovec, J., Ermon, S., and Xu, M. f -po: Generalizing preference optimization with f -divergence minimization. arXiv preprint arXiv:2410.21662, 2024

    2024 arXiv

  10. [18]

    D., Sun, W., Krishnamurthy, A., and Foster, D

    Huang, A., Zhan, W., Xie, T., Lee, J. D., Sun, W., Krishnamurthy, A., and Foster, D. J. Correcting the mythos of kl-regularization: Direct alignment without overoptimization via chi-squared preference optimization, 2024. URL https://arxiv.org/abs/2407.13399

    2024 arXiv

  11. [19]

    A., Choi, Y., and Hajishirzi, H

    Ivison, H., Wang, Y., Liu, J., Wu, Z., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., and Hajishirzi, H. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. NeurIPS, 2024

    2024

  12. [20]

    Towards efficient and exact optimization of language model alignment

    Ji, H., Lu, C., Niu, Y., Ke, P., Wang, H., Zhu, J., Tang, J., and Huang, M. Towards efficient and exact optimization of language model alignment. arXiv preprint arXiv:2402.00856, 2024

    2024 arXiv

  13. [21]

    A survey on human preference learning for large language models, 2024

    Jiang, R., Chen, K., Bai, X., He, Z., Li, J., Yang, M., Zhao, T., Nie, L., and Zhang, M. A survey on human preference learning for large language models, 2024. URL https://arxiv.org/abs/2406.11191

    2024 arXiv

  14. [22]

    A survey of reinforcement learning from human feedback

    Kaufmann, T., Weng, P., Bengs, V., and H \"u llermeier, E. A survey of reinforcement learning from human feedback. ArXiv, abs/2312.14925, 2023

    2023

  15. [23]

    Understanding the effects of rlhf on llm generalisation and diversity

    Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., and Raileanu, R. Understanding the effects of rlhf on llm generalisation and diversity. arXiv, 2024

    2024

  16. [24]

    R., Stevens, K., Barhoum, A., Duc, N

    Kopf, A., Kilcher, Y., von Rutte, D., Anagnostidis, S., Tam, Z. R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., Shahul, E., Suri, S., Glushkov, D., Dantuluri, A., Maguire, A., Schuhmann, C., Nguyen, H., and Mattick, A. Openassistant conversations - democrati...

    2023 arXiv

  17. [25]

    Huggingface h4 stack exchange preference dataset

    Lambert, N., Tunstall, L., Rajani, N., and Thrush, T. Huggingface h4 stack exchange preference dataset. 2023. URL https://huggingface.co/datasets/HuggingFaceH4/stack-exchange-preferences

    2023

  18. [26]

    Self-alignment with instruction backtranslation

    Li, X., Yu, P., Zhou, C., Schick, T., Levy, O., Zettlemoyer, L., Weston, J., and Lewis, M. Self-alignment with instruction backtranslation. arXiv preprint arXiv:2308.06259, 2023 a

    2023 arXiv

  19. [27]

    Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacaeval: An automatic evaluator of instruction-following models. GitHub repository, 2023 b

    2023

  20. [28]

    A., Socher, R., Amatriain, X., and Gao, J

    Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M. A., Socher, R., Amatriain, X., and Gao, J. Large language models: A survey. ArXiv, abs/2402.06196, 2024

    2024 arXiv

  21. [29]

    Monte carlo gradient estimation in machine learning

    Mohamed, S., Rosca, M., Figurnov, M., and Mnih, A. Monte carlo gradient estimation in machine learning. JMLR, 2020 a

    2020

  22. [30]

    Monte carlo gradient estimation in machine learning

    Mohamed, S., Rosca, M., Figurnov, M., and Mnih, A. Monte carlo gradient estimation in machine learning. JMLR, 2020 b

    2020

  23. [31]

    G., Rowland, M., Guo, Z

    Munos, R., Valko, M., Calandriello, D., Azar, M. G., Rowland, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T., Michi, A., Selvi, M., Girgin, S., Momchev, N., Bachem, O., Mankowitz, D. J., Precup, D., and Piot, B. Nash learning from human feedback. arXiv, 2023

    2023

  24. [32]

    A., Lindsten, F., and Blei, D

    Naesseth, C. A., Lindsten, F., and Blei, D. Markovian score climbing: Variational inference with kl(p || q). NeurIPS, 2020

    2020

  25. [33]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv, 2024

    2024

  26. [34]

    L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L. E., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. J. Training language mo...

    2022 arXiv

  27. [35]

    Smaug: Fixing failure modes of preference optimisation with dpo-positive, 2024

    Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: Fixing failure modes of preference optimisation with dpo-positive, 2024. URL https://arxiv.org/abs/2402.13228

    2024 arXiv

  28. [36]

    Scaling laws for reward model overoptimization in direct alignment algorithms, 2024 a

    Rafailov, R., Chittepu, Y., Park, R., Sikchi, H., Hejna, J., Knox, B., Finn, C., and Niekum, S. Scaling laws for reward model overoptimization in direct alignment algorithms, 2024 a . URL https://arxiv.org/abs/2406.02900

    2024 arXiv

  29. [37]

    From r to q^* : Your language model is secretly a q-function

    Rafailov, R., Hejna, J., Park, R., and Finn, C. From r to q^* : Your language model is secretly a q-function. COLM, 2024 b

    2024

  30. [38]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 36, 2024 c

    2024

  31. [39]

    Black box variational inference

    Ranganath, R., Gerrish, S., and Blei, D. Black box variational inference. In Artificial intelligence and statistics, pp.\ 814--822. PMLR, 2014

    2014

  32. [40]

    Unintentional unalignment: Likelihood displacement in direct preference optimization, 2024 a

    Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., and Hanin, B. Unintentional unalignment: Likelihood displacement in direct preference optimization, 2024 a . URL https://arxiv.org/abs/2410.08847

    2024 arXiv

  33. [41]

    Vanishing gradients in reinforcement finetuning of language models

    Razin, N., Zhou, H., Saremi, O., Thilak, V., Bradley, A., Nakkiran, P., Susskind, J., and Littwin, E. Vanishing gradients in reinforcement finetuning of language models. ICLR, 2024 b

    2024

  34. [42]

    Direct nash optimization: Teaching language models to self-improve with general preferences, 2024

    Rosset, C., Cheng, C.-A., Mitra, A., Santacroce, M., Awadallah, A., and Xie, T. Direct nash optimization: Teaching language models to self-improve with general preferences, 2024

    2024

  35. [43]

    Direct alignment with heterogeneous preferences

    Shirali, A., Nasr-Esfahany, A., Alomar, A., Mirtaheri, P., Abebe, R., and Procaccia, A. Direct alignment with heterogeneous preferences. ArXiv, abs/2502.16320, 2025. URL https://api.semanticscholar.org/CorpusID:276574796

    2025 arXiv

  36. [44]

    A long way to go: Investigating length correlations in rlhf, 2024

    Singhal, P., Goyal, T., Xu, J., and Durrett, G. A long way to go: Investigating length correlations in rlhf, 2024. URL https://arxiv.org/abs/2310.03716

    2024 arXiv

  37. [45]

    Distributional preference learning: Understanding and accounting for hidden context in rlhf

    Siththaranjan, A., Laidlaw, C., and Hadfield-Menell, D. Distributional preference learning: Understanding and accounting for hidden context in rlhf. ICLR, 2024

    2024

  38. [46]

    How to compare different loss functions and their risks

    Steinwart, I. How to compare different loss functions and their risks. Constructive Approximation, 26 0 (2): 0 225--287, 2007. doi:10.1007/s00365-006-0668-8

    2007 doi

  39. [47]

    M., Lowe, R

    Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R. J., Voss, C., Radford, A., Amodei, D., and Christiano, P. Learning to summarize from human feedback. ArXiv, abs/2009.01325, 2020

    2009 arXiv

  40. [48]

    S., and Agarwal, A

    Swamy, G., Dann, C., Kidambi, R., Wu, Z. S., and Agarwal, A. A minimaximalist approach to reinforcement learning from human feedback. ICML, 2024

    2024

  41. [49]

    Preference fine-tuning of llms should leverage suboptimal, on-policy data

    Tajwar, F., Singh, A., Sharma, A., Rafailov, R., Schneider, J., Xie, T., Ermon, S., Finn, C., and Kumar, A. Preference fine-tuning of llms should leverage suboptimal, on-policy data. ICML, 2024

    2024

  42. [50]

    D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P

    Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Avila Pires, B., and Piot, B. Generalized preference optimization: A unified approach to offline alignment. ICML, 2024

    2024

  43. [51]

    Trl: Transformer reinforcement learning

    von Werra, L., Belkada, Y., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., and Huang, S. Trl: Transformer reinforcement learning. GitHub repository, 2020

    2020

  44. [52]

    Enabling language models to implicitly learn self-improvement

    Wang, Z., Hou, L., Lu, T., Wu, Y., Li, Y., Yu, H., and Ji, H. Enabling language models to implicitly learn self-improvement. ICLR, 2024 a

    2024

  45. [53]

    Transforming and combining rewards for aligning large language models

    Wang, Z., Nagpal, C., Berant, J., Eisenstein, J., D'Amour, A., Koyejo, S., and Veitch, V. Transforming and combining rewards for aligning large language models. arXiv, 2024 b

    2024

  46. [54]

    Policy gradient algorithms

    Weng, L. Policy gradient algorithms. lilianweng.github.io, 2018. URL https://lilianweng.github.io/posts/2018-04-08-policy-gradient/

    2018

  47. [55]

    V., Murray, K., and Kim, Y

    Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Durme, B. V., Murray, K., and Kim, Y. J. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. ICML, 2024 a

    2024

  48. [56]

    Some things are more cringe than others: Iterative preference optimization with the pairwise cringe loss, 2024 b

    Xu, J., Lee, A., Sukhbaatar, S., and Weston, J. Some things are more cringe than others: Iterative preference optimization with the pairwise cringe loss, 2024 b . URL https://arxiv.org/abs/2312.16682

    2024 arXiv

  49. [57]

    Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J

    Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language models. arXiv, 2024

    2024

  50. [58]

    Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J. Slic-hf: Sequence likelihood calibration with human feedback. arXiv, 2023

    2023

  51. [59]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. NeurIPS, 36, 2024

    2024

  52. [60]

    I., and Jiao, J

    Zhu, B., Jordan, M. I., and Jiao, J. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. arXiv, 2024

    2024

  53. [61]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.