Pith. sign in

REVIEW 4 major objections 5 minor 54 references

Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DPO can miss the true RLHF-optimal policy because its underlying Bradley-Terry model can have multiple maximum-likelihood fits; the proposed energy-based model EPA has a unique fit and outperforms DPO on open benchmarks.

desk verdict Solid negative result on DPO's non-uniqueness, a real but unproven theoretical bridge to EPA, and enough empirical consistency to merit a serious referee. read the letter →

arxiv 2412.13862 v1 pith:6CUDVGOG submitted 2024-12-18 cs.LG cs.CL

Yuzhong Hong , Hanshan Zhang , Junwei Bao , Hongfei Jiang , Yang Song This is my paper · ORCID
classification cs.LGcs.CL
keywords DirectPreferenceOptimization(DPO)Bradley-Terrymodelenergy-basedalignmentRLHFcontrastivelossmaximumlikelihoodestimationoffline
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Offline preference alignment via DPO is supposed to find the unique policy that minimizes the KL-constrained RLHF objective, which is equivalent to a log-ratio reward $r_\theta(x,y)=\beta\log(\pi_\theta(y|x)/\pi_{\mathrm{ref}}(y|x))$ sitting in slope-1 linearity with the true reward. The paper's core claim is that this target can be unattainable for DPO, because the Bradley-Terry model behind the DPO loss can have multiple maximum-likelihood estimators over the infinite space of responses, and only one of those estimators satisfies the required linearity. The authors propose an energy-based model, the Infinite Preference Model, whose maximum-likelihood estimator is provably unique and exactly matches the RLHF minimizer, together with a tractable contrastive loss, EPA, that contrasts each preferred response with strong rejected responses and many weak mismatched responses. They prove that under a symmetric negative-sampling condition the approximation error of EPA vanishes almost surely as the number of negatives grows, and they show empirically that EPA-trained policies come closer to slope-1 linearity and score higher than DPO-trained policies on MT-Bench and Alpaca-Eval 2.0. If the argument is right, offline alignment should move from pairwise Bradley-Terry fitting to energy-based modeling over the full response space.

What carries the argument

The load-bearing object is the Infinite Preference Model (IPM), an energy-based model that assigns each response a Boltzmann probability $p(y|x)\propto\exp(r_{\mathrm{true}}(x,y))$ over the effectively infinite set of possible responses. Its MLE, when the model is parameterized by the log-ratio reward $r_\theta(x,y)=\beta\log(\pi_\theta(y|x)/\pi_{\mathrm{ref}}(y|x))$, is shown to exist uniquely and to coincide exactly with the minimizer of the KL-constrained RLHF objective (Theorem 3.1). The second piece of machinery is the energy-discrepancy functional, whose unique global minimizer is the IPM itself; the EPA loss is a finite-sample estimator of that functional, and its approximation error vanishes almost surely when the negative-sampling source satisfies the symmetry condition $\sum_y p(z|y)f(y)=\sum_y p(y|z)f(y)$. The practical engine is the contrast between one preferred response, one or more strong rejected responses, and many weak mismatched responses, which the paper argues reflects the theorem's requirement that negatives be informative of the positive and high-variance.

What would settle it

A direct test would train DPO and EPA on synthetic data with a known true reward, deliberately including for each prompt a response that never appears in any preference pair, then measure on held-out responses whether the learned log-ratio reward satisfies $r_\theta(x,y)=r_{\mathrm{true}}(x,y)+C(x)$. If DPO also converges to a unique slope-1 solution, the claimed failure mode is not triggered in that regime; if EPA's closeness to slope-1 linearity does not improve as weak negatives are added, the practical negative-sampling assumption is the weak link.

Watch

Extended reading notes

Core claim

The paper's central discovery is a failure mode in DPO and a replacement that avoids it. Because the space of possible responses to a prompt is effectively infinite, the Bradley-Terry preference model's maximum-likelihood estimator is not guaranteed to be unique; the DPO loss then has multiple minimizers, and the unique minimizer of the KL-constrained RLHF loss is only one of them. The proposed replacement, the Infinite Preference Model (IPM), treats $p(y|x)$, the probability that response $y$ is preferred over all other responses, as a Boltzmann distribution over the true reward, $p(y|x)\propto\exp(r_{\mathrm{true}}(x,y))$. Parametrized by the log-ratio reward, IPM has a unique MLE, and that MLE is reached exactly when $r_\theta(x,y)=r_{\mathrm{true}}(x,y)+C(x)$ holds, which is the same condition that characterizes the RLHF minimizer. The practical loss EPA approximates the IPM's MLE by maximizing the softmax probability of the preferred response against strong negatives (the rejected responses) and weak negatives (mismatched responses from other prompts); the authors show that the approximation error of this estimator almost surely vanishes when negatives are drawn from a source satisfying the energy-discrepancy symmetry condition, and that EPA-trained models are measurably closer to slope-1 linearity than DPO-trained models.

Load-bearing premise

The load-bearing premise is that the mismatched-response negatives used in EPA training behave enough like the carefully chosen perturbation source for which the convergence proof works; if that fails, the nearness of EPA's optimum to the RLHF minimizer is not guaranteed.

Editorial extensions

If this is right

  • Offline alignment methods should be evaluated by whether the learned log-ratio reward achieves $r_\theta(x,y)=r_{\mathrm{true}}(x,y)+C(x)$ on held-out responses, not only by pairwise ranking accuracy.
  • Having multiple strong negatives per prompt, as in multi-response preference datasets, improves the approximation to the IPM's MLE and is a practical reason to collect or generate several rejected responses per prompt.
  • Adding more preference pairs cannot by itself guarantee that DPO reaches the RLHF optimum: the pair-sampling distribution over the infinite response space must be structured so that the Bradley-Terry MLE is unique.
  • Loss-modification tricks such as margins and on-policy weighting improve both DPO and EPA, but they do not remove the underlying non-uniqueness, and EPA with the same tricks remains ahead.
  • The weak-contrast term in EPA's gradient acts as a regularizer, making training less prone to the rapid overfitting seen with DPO after the first epoch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the non-uniqueness argument should transfer to any pairwise or Plackett-Luce preference loss, so IPO, DPO-PL, and similar methods may carry the same unattainability risk in infinite response spaces.
  • Beyond the paper: the paper's slope-1 linearity probe could be adopted as a general diagnostic for offline alignment, independent of preference-model family, which would make the theoretical target directly measurable.
  • Beyond the paper: testing EPA on synthetic data with known rewards and varying number of weak negatives would reveal whether the approximation error decreases monotonically toward zero or plateaus because of the mismatched-negative source; Theorem 3.3 predicts the former only under its symmetry condition.
  • Beyond the paper: the symmetry condition behind the almost-sure convergence could serve as a principled criterion for negative selection in other contrastive learning settings, such as metric learning or sentence embedding, where hard-negative mining is currently heuristic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that DPO, as maximum likelihood estimation of the Bradley-Terry model, can have multiple minimizers in the infinite response space, so the unique RLHF-optimal policy may be unattainable. It proposes the Infinite Preference Model (IPM), an energy-based model whose MLE is unique and equivalent to the RLHF minimizer, and derives a contrastive loss, Energy Preference Alignment (EPA), which uses strong and weak negatives. The empirical section reports that EPA outperforms DPO and other baselines on AlpacaEval 2.0 and MT-Bench with Mistral-7B on Ultrafeedback data.

Significance. The central negative claim about DPO is valuable and mostly independent of the positive proposal: Proposition B.5 gives a concrete construction showing that an unsampled response breaks uniqueness of the DPO minimizer, which is a real caveat for offline alignment in infinite output spaces. The positive claim, that the IPM has a unique MLE equal to the RLHF minimizer, is a clean theoretical identity, though largely a restatement of the Boltzmann/energy parameterization. The empirical gains of EPA over DPO and several other baselines on standard benchmarks are consistent across tables and ablations, and the paper includes helpful controls such as adding weak negatives to DPO (Table 5), slope-1 linearity probing (Table 7), and an alignment-tax table (Table 6). The main weakness is the gap between the theoretical convergence guarantee (Theorem 3.3) and the actual EPA loss defined in Eq. (11)/(12), where the negative sampling distribution is not shown to satisfy the balance condition.

major comments (4)
  1. [Theorem 3.3 and Eq. (11)/(12)] Theorem 3.3 states that the contrastive estimate in Eq. (9) converges almost surely to the energy discrepancy only when Z satisfies the balance condition. In Eq. (11), weak negatives are mismatched responses from other prompts, drawn from a distribution that is not specified as a perturbation kernel p(z|y) for the positive y, and no argument is given that the balance condition holds. Therefore the almost-sure convergence guarantee does not formally apply to the implemented EPA loss, and the paper's own statement in Section 3.2 that the condition is only sufficient does not fill this gap. The theoretical bridge from the IPM's unique MLE to the practical EPA loss is thus not established by the manuscript as written.
  2. [Assumption B.1 and Section 3.1] Assumption B.1, which is required for the IPM partition function to be finite, states that the set of possibly preferred responses for each prompt is finite. In the later justification in Section 4.1, the authors appeal to this assumption to argue that a dataset preferred y_w is a good approximation of a positive sample from p(y|x). However, Theorem 3.1 as stated in the paper is essentially a restatement of the uniqueness of the Boltzmann MLE within that finite support; the discussion in Section 3.1 frames the contribution as taking infinity into account, so the reader should be told explicitly that the formal theorem only covers the finite-support case and that the infinite-support case is handled only by Assumptions B.1/B.2 as a sufficient condition for the partition function to exist.
  3. [Table 1 and Section 5.2.1] The empirical comparison in Table 1 reports EPA "a new state of the art", but the significance of the gains on MT-Bench is small (7.71 vs. 7.55 for DPO, and 7.58 for EPA-general vs. 7.57 for DPO-PL), with no variance reported for the single-seed runs in Table 1. Since the central claim is that EPA is closer to the RLHF optimum, the paper would be strengthened by reporting multiple seeds or a significance test for at least the headline comparisons; as it stands, the AE 2.0 gains are large but the MT-Bench gains are borderline.
  4. [Section 4.3 and Tables 2, 5] The gradient analysis in Section 4.3 (Eq. 13) interprets the weak contrast as a regularization term and states that DPO's gradient is recovered when there is no weak contrast. This is correct, but the empirical results in Tables 2 and 5 suggest that the benefit of EPA may come substantially from this regularization effect rather than from the IPM uniqueness mechanism. The paper does not clearly separate these two explanations; a concrete test would be to evaluate whether a DPO loss with the same weak-negative regularization achieves the same or similar AE 2.0 performance, which Table 5 partially addresses but only with added weak negatives as data rather than as in-batch regularizers.
minor comments (5)
  1. [Eq. (9)] In Eq. (9), the notation uses N as both the sample count and the number of weak negatives in later notation; the subscript/superscript conventions for N_weak^- and N_strong^- are introduced only in Table 2, which is fine, but Eq. (9) itself should clarify whether M is the number of negatives per positive.
  2. [Figure 1 and Section 1.1] The statement in Section 1.1 that the reward r_theta for weak negatives is "not substantially lower" than for strong negatives in Figure 1 is hard to verify from the figure as presented; labeling the axes and adding a caption that explains the y-axis (log ratio reward) would help.
  3. [Table 5] In Table 5, the notation "+ UF-weak x1" is not defined before the table; a sentence in Appendix C explaining that it adds a copy of UF-binarized with y_l replaced by a random weak negative would improve readability.
  4. [Appendix B.3] The proof sketch for Theorem 3.3 in Appendix B.3 says the balance condition enables a statistic estimate but does not state which form of the Strong Law of Large Numbers is used for the inner logarithm; citing the specific result and stating the integrability conditions would make the proof easier to check.
  5. [Section 2.2] In the related work, the comparison with infoNCA claims infoNCA is "a worse-performing ablation version of EPA", but this claim is not backed by a formal derivation; either add a short derivation or soften the claim.

Circularity Check

1 steps flagged · score 2.0 of 10

Definitional uniqueness theorem; independent DPO critique and benchmarks keep overall circularity low.

  1. self definitional [Section 3.1, Eq. (6)-(7) and Theorem 3.1; Appendix B.1, Definition B.3 and Theorem B.4]
    "we define the Infinite Preference Model (IPM) to be the one that posits that p(y|x) is a Boltzmann distribution induced by the corresponding true reward ... p(y|x) = exp[rtrue(x, y)] / Σ∞ y' exp[rtrue(x, y')] (6) ... Theorem 3.1. when we parameterize the IPM as follows, the unique existence of the IPM's MLE is guaranteed and it will be reached if and only if the slope-1 linearity (i.e., Eq.(3)) holds between the log ratio reward and the true reward. qθ(y|x) = exp[rθ(x, y)] / Σ∞ y' exp[rθ(x, y')] (7)"

    By Definition A.2, slope-1 linearity is rθ(x,y) = rtrue(x,y) + C(x). Since Eq. (6) defines the target p(y|x) as exp(rtrue)/Z and Eq. (7) defines the model qθ as exp(rθ)/Zθ, the condition qθ = p is algebraically the same as rθ = rtrue + C(x). Theorem B.4 then proves the MLE is unique by Gibbs inequality (KL[p||q] minimized at q = p), so the 'guarantee' is an identity built into the model definition: the data-generating distribution is stipulated to be the Boltzmann distribution of the very reward the MLE is said to recover. The link to the RLHF minimizer is still imported from Lemma A.1/Korbak et al., so the uniqueness claim itself is self-definitional rather than an independent derivation.

full rationale

The only circular-adjacent step is Theorem 3.1: the IPM is defined as the Boltzmann distribution of r_true, so its MLE (q_theta = p) is equivalent to r_theta = r_true + C by the model's own equations, making the uniqueness statement an identifiability tautology. This is real but mild, because the paper's central negative claim about DPO is independent: Proposition B.5 constructs a second minimizer of the Bradley-Terry likelihood without assuming the conclusion, and Lemma A.1 imports the RLHF minimizer from prior work. The empirical EPA-versus-DPO comparisons on AlpacaEval 2.0 and MT-Bench are external benchmarks and do not reduce to the definition. The EPA-to-MLE transfer does have a genuine gap: Eq. (11)/(12) use weak negatives from mismatched responses, and the paper only asserts the Theorem 3.3 balance condition is sufficient, not necessary, so the almost-sure convergence guarantee is not proven for the implemented loss. That is an unproved sufficiency claim and a correctness risk, not circularity. No load-bearing self-citation chain is present; the energy-discrepancy framework is cited from Schröder et al. with no author overlap. Overall score is 2 rather than 0 because the flagship uniqueness theorem is largely a restatement of the model definition, but the DPO critique and empirical content remain independent.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central claim leans on the IPM model definition, the finite-support assumptions, and the realizability of the optimal policy in the parameterized family. The practical EPA loss additionally depends on an unverified relationship between the negative-sampling distribution and the theoretical Z condition. Main free parameters are the number of strong and weak negatives and the hyperparameters beta, batch size, and LoRA rank.

free parameters (3)
  • N_weak^- (number of weak negatives per positive) = 2 for main EPA (1:1:2); varied 0 to 10 in ablations
    Chosen by hand in Eq. (11); ablation Table 2 shows performance depends strongly on it, e.g., 1:1:1 gives MT-Bench 7.29 while 1:1:2 gives 7.71.
  • N_strong^- (number of strong negatives per positive) = 1 or 3, depending on dataset setup
    Uses the available paired negatives, 1 in UF-binarized and up to 3 in UF-all; ablations show it changes results, e.g., 1:3:8 wins AlpacaEval 24.54 but has MT-Bench 7.19.
  • beta (KL penalty coefficient) = 0.01 for main comparisons; varied 0.01 to 0.5 for frontiers; 2.0 for some tricks
    The paper calls beta a control variable of the RLHF objective, but for tricks it is tuned per method, and the main comparison fixes it at 0.01 without sensitivity analysis.
assumptions (5)
  • standard math Gibbs' inequality (KL divergence is non-negative and zero iff distributions match)
    Used in Lemma A.1 and Theorem B.4 to identify the minimizer of the RLHF and IPM likelihood objectives.
  • domain assumption Assumption B.1: for each prompt x, the set of responses with positive preference probability p(y|x) is finite.
    Invoked in Section B.1 to make the IPM partition function finite. It is not validated for natural language, where many semantically different responses can be preferred.
  • domain assumption Assumption B.2: r_true(x,y) tends to negative infinity for any y outside the finite support and is finite inside.
    Needed for the IPM to be a well-defined Boltzmann distribution; it also implies the slope-1 solution has a finite normalizer, but this is assumed rather than derived.
  • ad hoc to paper Realizability: the policy class pi_theta can represent the RLHF-optimal policy pi_r proportional to pi_ref * exp(r_true / beta).
    Theorem 3.1's claim that the IPM MLE is reached iff slope-1 linearity holds requires that some theta in the parameterized family realizes r_true(x,y)+C(x); the paper never states this assumption.
  • ad hoc to paper The perturbation source Z used for negative sampling satisfies the detailed-balance-like condition Sum_y p(z|y)f(y) = Sum_y p(y|z)f(y).
    Theorem 3.3's almost-sure convergence depends on it; EPA's practical weak negatives are not shown to satisfy it, only a rule-of-thumb argument is given in Section 3.2.
invented entities (1)
  • Infinite Preference Model (IPM)
    purpose: A Boltzmann preference distribution p(y|x) proportional to exp[r_true(x,y)] that is claimed to have a unique MLE, used as the theoretical basis for EPA.
    The model is defined so that its MLE coincides with the RLHF optimum; it has no falsifiable handle outside the paper beyond the preference probabilities it posits, and the finite-support assumptions needed to make it well-defined are unverified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model." pith.science (2026). https://pith.science/paper/6CUDVGOG

@misc{pith2026241213862,
  author       = {Pith},
  title        = {Pith review of: Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6CUDVGOG}},
  note         = {Machine review of arXiv:2412.13862}
}
read the original abstract

Since the debut of DPO, it has been shown that aligning a target LLM with human preferences via the KL-constrained RLHF loss is mathematically equivalent to a special kind of reward modeling task. Concretely, the task requires: 1) using the target LLM to parameterize the reward model, and 2) tuning the reward model so that it has a 1:1 linear relationship with the true reward. However, we identify a significant issue: the DPO loss might have multiple minimizers, of which only one satisfies the required linearity condition. The problem arises from a well-known issue of the underlying Bradley-Terry preference model: it does not always have a unique maximum likelihood estimator (MLE). Consequently,the minimizer of the RLHF loss might be unattainable because it is merely one among many minimizers of the DPO loss. As a better alternative, we propose an energy-based model (EBM) that always has a unique MLE, inherently satisfying the linearity requirement. To approximate the MLE in practice, we propose a contrastive loss named Energy Preference Alignment (EPA), wherein each positive sample is contrasted against one or more strong negatives as well as many free weak negatives. Theoretical properties of our EBM enable the approximation error of EPA to almost surely vanish when a sufficient number of negatives are used. Empirically, we demonstrate that EPA consistently delivers better performance on open benchmarks compared to DPO, thereby showing the superiority of our EBM.

Figures

Figures reproduced from arXiv: 2412.13862 by the authors.

Figure 1
Figure 1. Samples are off from the slope-1 linearity (yellow lines) after training with DPO. Given an extremely undesirable y − weak (i.e., it has very small rtrue), its rθ has to be as sufficiently small as rtrue to attain the linearity. canonical RLHF objective (Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022; Perez et al., 2022) is defined as follows (given x): LRLHF = − E πθ(y|x) [rtrue(x, y)] + βKL[πθ(y|… view at source ↗
Figure 2
Figure 2. An illustration of the contributions of the paper. Our core argument is that an Energy-Based model (EBM) is a better alternative to the Bradley-Terry model (BTM) due to its guaranteed unique existence of maximum likelihood estimator (MLE) (which is identical to the minimizer of the RLHF loss). The advantage of our EBM comes from its intrinsic consideration of the infinity in the size of the space of y|x, whereas BTM… view at source ↗
Figure 3
Figure 3. DPO vs. EPA (1:1:2) from the perspective of (a) KL-Reward frontier and (b) training dynamics. tricks in consideration are those used in SimPO, R-DPO, CPO and WPO (Details in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance of modified DPO (N − weak = 0) and modi￾fied EPA (N − weak > 0) with a margin mc added to rθ(yl|x). Solid lines represent the length-controlled win-rates, and dotted lines represent the raw win-rates. we only consider two of them when applying to EPA. The f…
Figure 6
Figure 6. Figure 6: EPA vs DPO in terms of the slope-1 linearity. Each average slope-1 linear regression error ϵˆ is computed on 500 × 10% = 50 prompts. Although the difference between EPA and DPO only becomes noticeable for the “WORST 10%” group in the visualization (i.e., how close the …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 21 canonical work pages

  1. [1]

    Direct preference optimization with an offset

    Amini, A., Vieira, T., and Cotterell, R. Direct preference optimization with an offset. arXiv preprint arXiv:2402.10571, 2024

    arXiv 2024

  2. [2]

    G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R

    Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310.12036, 2023

    arXiv 2023

  3. [3]

    and Rinaldo, A

    Bong, H. and Rinaldo, A. Generalized results for the existence and consistency of the MLE in the bradley-terry-luce model. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.\ 2160--2177. PM...

    work page 2022

  4. [4]

    Noise contrastive alignment of language models with explicit rewards

    Chen, H., He, G., Su, H., and Zhu, J. Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402.05369, 2024 a

    arXiv 2024

  5. [5]

    Self-play fine-tuning converts weak language models to strong language models

    Chen, Z., Deng, Y., Yuan, H., Ji, K., and Gu, Q. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024 b

    arXiv 2024

  6. [6]

    F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D

    Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  7. [7]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168

    arXiv 2021

  8. [8]

    Ultrafeedback: Boosting language models with scaled ai feedback

    Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., et al. Ultrafeedback: Boosting language models with scaled ai feedback. In Forty-first International Conference on Machine Learning, 2024

    2024

Show all 54 references
  1. [9]

    Learning discrete energy-based models via auxiliary-variable local exploration

    Dai, H., Singh, R., Dai, B., Sutton, C., and Schuurmans, D. Learning discrete energy-based models via auxiliary-variable local exploration. Advances in Neural Information Processing Systems, 33: 0 10443--10455, 2020

    2020

  2. [10]

    Residual energy-based models for text generation

    Deng, Y., Bakhtin, A., Ott, M., Szlam, A., and Ranzato, M. Residual energy-based models for text generation. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=B1l4SgHKDH

    2020

  3. [11]

    Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024

    2024 arXiv

  4. [12]

    R., Elsahar, H., and Dymetman, M

    Eikema, B., Kruszewski, G., Dance, C. R., Elsahar, H., and Dymetman, M. An approximate sampler for energy-based models with divergence diagnostics. Transactions on Machine Learning Research, 2022

    2022

  5. [13]

    Kto: Model alignment as prospect theoretic optimization, 2024

    Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization, 2024

    2024

  6. [14]

    Ford, L. R. Solution of a ranking problem from binary comparisons. The American Mathematical Monthly, 64 0 (8): 0 28--33, 1957. ISSN 00029890, 19300972. URL http://www.jstor.org/stable/2308513

    1957

  7. [15]

    Asymptotic theory of sparse Bradley–Terry model

    Han, R., Ye, R., Tan, C., and Chen, K. Asymptotic theory of sparse Bradley–Terry model . The Annals of Applied Probability, 30 0 (5): 0 2491 -- 2515, 2020. doi:10.1214/20-AAP1564. URL https://doi.org/10.1214/20-AAP1564

    2020 doi

  8. [16]

    Minimax rate for learning from pairwise comparisons in the BTL model

    Hendrickx, J., Olshevsky, A., and Saligrama, V. Minimax rate for learning from pairwise comparisons in the BTL model. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Researc...

    2020

  9. [17]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. CoRR, abs/2009.03300, 2020. URL https://arxiv.org/abs/2009.03300

    2009 arXiv

  10. [18]

    Orpo: Monolithic preference optimization without reference model

    Hong, J., Lee, N., and Thorne, J. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2 0 (4): 0 5, 2024

    2024 arXiv

  11. [19]

    Some extensions of score matching

    Hyv \"a rinen, A. Some extensions of score matching. Computational statistics & data analysis, 51 0 (5): 0 2499--2512, 2007

    2007

  12. [20]

    W., and On, K.-W

    Jung, S., Han, G., Nam, D. W., and On, K.-W. Binary classifier optimization for large language model alignment. arXiv preprint arXiv:2404.04656, 2024

    2024 arXiv

  13. [21]

    and Langford, J

    Kakade, S. and Langford, J. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pp.\ 267--274, 2002

    2002

  14. [22]

    A distributional approach to controlled text generation

    Khalifa, M., Elsahar, H., and Dymetman, M. A distributional approach to controlled text generation. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=jWkw45-9AbL

    2021

  15. [23]

    On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting

    Korbak, T., Elsahar, H., Kruszewski, G., and Dymetman, M. On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting. Advances in Neural Information Processing Systems, 35: 0 16203--16220, 2022 a

    2022

  16. [24]

    Rl with kl penalties is better viewed as bayesian inference

    Korbak, T., Perez, E., and Buckley, C. Rl with kl penalties is better viewed as bayesian inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp.\ 1083--1091, 2022 b

    2022

  17. [25]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp.\ 1207--1216, Stanford, CA, 2000. Morgan Kaufmann

    2000

  18. [26]

    Perturb-and-max-product: Sampling and learning in discrete energy-based models

    Lazaro-Gredilla, M., Dedieu, A., and George, D. Perturb-and-max-product: Sampling and learning in discrete energy-based models. Advances in Neural Information Processing Systems, 34: 0 928--940, 2021

    2021

  19. [27]

    A tutorial on energy-based learning

    LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., and Huang, F. A tutorial on energy-based learning. Predicting structured data, 1 0 (0), 2006

    2006

  20. [28]

    Conditional strong law of large number

    Majerek, D., Nowak, W., and Zieba, W. Conditional strong law of large number. Int. J. Pure Appl. Math, 20 0 (2): 0 143--156, 2005

    2005

  21. [29]

    Concrete score matching: Generalized score matching for discrete data

    Meng, C., Choi, K., Song, J., and Ermon, S. Concrete score matching: Generalized score matching for discrete data. Advances in Neural Information Processing Systems, 35: 0 34532--34545, 2022

    2022

  22. [30]

    Simpo: Simple preference optimization with a reference-free reward

    Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024

    2024 arXiv

  23. [31]

    A note on dpo with noisy preferences & relationship to ipo, 2023

    Mitchell, E. A note on dpo with noisy preferences & relationship to ipo, 2023. URL https://ericmitchell.ai/cdpo.pdf

    2023

  24. [32]

    and Szepesv \'a ri, C

    Munos, R. and Szepesv \'a ri, C. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9 0 (5), 2008

    2008

  25. [33]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  26. [34]

    Disentangling length from quality in direct preference optimization

    Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159, 2024

    2024 arXiv

  27. [35]

    Distributional reinforcement learning for energy-based sequential models

    Parshakova, T., Andreoli, J.-M., and Dymetman, M. Distributional reinforcement learning for energy-based sequential models. Optimization Foundations for Reinforcement Learning Workshop at NeurIPS 2019, 2019. URL https://optrl2019.github.io/assets/accepted_papers/34.pdf

    2019

  28. [36]

    Red teaming language models with language models

    Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.\ 3419--3448, 2022

    2022

  29. [37]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=HPuSIXJaa9

    2023

  30. [38]

    Schr \"o der, T., Ou, Z., Li, Y., and Duncan, A. B. Training discrete EBM s with energy discrepancy. In ICML 2023 Workshop: Sampling and Optimization in Discrete Space, 2023. URL https://openreview.net/forum?id=kFMpJh75Wo

    2023

  31. [39]

    and Yao, Y.-C

    Simons, G. and Yao, Y.-C. Asymptotics when the number of parameters tends to infinity in the bradley-terry model for paired comparisons. The Annals of Statistics, 27 0 (3): 0 1041--1060, 1999

    1999

  32. [40]

    Preference ranking optimization for human alignment

    Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., and Wang, H. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 18990--18998, 2024

    2024

  33. [41]

    and Kingma, D

    Song, Y. and Kingma, D. P. How to train your energy-based models. arXiv preprint arXiv:2101.03288, 2021

    2021 arXiv

  34. [42]

    Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33: 0 3008--3021, 2020

    2020

  35. [43]

    D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P

    Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Pires, B. \'A ., and Piot, B. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024

    2024 arXiv

  36. [44]

    and Ryabinin, M

    Tikhonov, A. and Ryabinin, M. It's all in the heads: Using attention heads as a baseline for cross-lingual transfer in commonsense reasoning. CoRR, abs/2106.12066, 2021. URL https://arxiv.org/abs/2106.12066

    2021 arXiv

  37. [45]

    Zephyr: Direct distillation of lm alignment

    Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944, 2023

    2023 arXiv

  38. [46]

    W., and Niezink, N

    Wu, W., Junker, B. W., and Niezink, N. Asymptotic comparison of identifying constraints for bradley-terry models. arXiv preprint arXiv:2205.04341, 2022

    2022 arXiv

  39. [47]

    Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Van Durme, B., Murray, K., and Kim, Y. J. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401.08417, 2024

    2024 arXiv

  40. [48]

    Q., Salamatian, S., Sun, Z., Suresh, A

    Yang, J. Q., Salamatian, S., Sun, Z., Suresh, A. T., and Beirami, A. Asymptotics of language model alignment, 2024

    2024

  41. [49]

    RRHF : Rank responses to align language models with human feedback

    Yuan, H., Yuan, Z., Tan, C., Wang, W., Huang, S., and Huang, F. RRHF : Rank responses to align language models with human feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=EdIGMCHk4l

    2023

  42. [50]

    Offline reinforcement learning with realizability and single-policy concentrability

    Zhan, W., Huang, B., Huang, A., Jiang, N., and Lee, J. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, pp.\ 2730--2775. PMLR, 2022

    2022

  43. [51]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024

    2024

  44. [52]

    R., Zhao, S., Song, K., Xu, S., and Zhu, C

    Zhou, W., Agrawal, R., Zhang, S., Indurthi, S. R., Zhao, S., Song, K., Xu, S., and Zhu, C. Wpo: Enhancing rlhf with weighted preference optimization. arXiv preprint arXiv:2406.11827, 2024

    2024 arXiv

  45. [53]

    M., Stiennon, N., Wu, J., Brown, T

    Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

  46. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.