{"id":"0633cfec-c9da-4928-858d-5f94f054854d","arxiv_id":"2412.13862","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"EPA, a contrastive preference loss based on an energy-based model with a unique maximum likelihood estimator, outperforms DPO on offline LLM alignment benchmarks.","lead":"An LLM alignment method called EPA uses an energy-based preference model instead of the Bradley-Terry model behind DPO, and reports better results on AlpacaEval 2.0 and MT-Bench. A reader might care because the paper argues DPO can fail to reach the true RLHF solution when its preference model has multiple equally good fits.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central theoretical claim is not supported for the implemented EPA loss: Eq. (11)/(12) samples weak negatives from mismatched responses, a source not shown to satisfy Theorem 3.3's condition, so the almost-sure convergence to the IPM MLE cannot be invoked.","rationale":"After reading the full manuscript, the strongest version of the central claim is the chain: IPM has a unique MLE equivalent to slope-1 linearity (Theorem 3.1); energy discrepancy has the same unique minimizer (Theorem 3.2); EPA's contrastive loss is a consistent estimator of the energy discrepancy (Theorem 3.3); therefore EPA recovers the RLHF optimum. The least secure link is the last one. Theorem 3.3 is conditional on a precise balance condition for the perturbation source Z. The actual EPA loss uses y_l and mismatched batch responses as negatives; no perturbation kernel p(z|y) is specified, and for mismatched responses the negative is essentially independent of the positive, so the balance condition is not verified. The manuscript explicitly flags this condition as sufficient but not necessary (Section 3.2) and falls back on a heuristic from Schroder et al. (2023). That admission matters because the almost-sure guarantee does not cover the implemented loss. The empirical results are consistently positive but single-seed and checkpoint-selected, so they do not substitute for the missing theoretical link. A controlled synthetic test can settle whether the actual sampling scheme recovers r_true + C; absent such a test, the paper's theoretical superiority claim should be regarded as unproven even though the empirical contribution may remain useful. This is an addressable gap, so a CONDITIONAL verdict remains appropriate rather than moving to rejection.","tokens_in":22149,"tokens_out":10377,"duration_ms":97531,"concrete_test":"Run a controlled synthetic experiment with a finite response space, known true reward r_true, and known preference distribution p(y|x) proportional to exp(r_true(x,y)). Optimize the EPA loss with (a) weak negatives drawn as independent random responses, mimicking mismatched batch items, and (b) negatives from a perturbation kernel p(z|y) that satisfies the Theorem 3.3 balance condition, using the same numbers of negatives. Measure whether the recovered r_theta equals r_true(x,y) + C(x) up to numerical tolerance. If (a) deviates substantially while (b) converges, the EPA-to-IPM-MLE link fails for the actual weak-negative source. Additionally, report the residual magnitude of Sum_y p(z|y)f(y) - Sum_y p(y|z)f(y) for the mismatched sampler to quantify the condition violation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's load-bearing bridge from the IPM's unique MLE (Theorem 3.1) to the implemented algorithm is Theorem 3.3: the contrastive estimate in Eq. (9) converges almost surely to the energy discrepancy only when the negative variable Z obeys the balance condition Sum_y p(z|y) f(y) = Sum_y p(y|z) f(y). The EPA losses in Eq. (11) and (12) are presented as practical versions of this estimate, with y_l as strong negatives and mismatched responses from other prompts as weak negatives. However, no perturbation kernel p(z|y) is specified for either negative source, and for the weak negatives z is a response to a different prompt, effectively independent of the positive y. The balance condition is therefore not shown to hold, and the paper itself states only that the condition is sufficient, not necessary, citing a rule of thumb from Schroder et al. (2023). Consequently, the almost-sure convergence guarantee does not apply to the actual EPA loss, and the unique-MLE/RLHF-optimality property of the IPM does not transfer to EPA. EPA may still be a useful regularized contrastive loss, and the gradient analysis in Section 4.3 supports that reading, but the theoretical claim that EPA approximates the IPM MLE is unsubstantiated. The empirical improvements on AlpacaEval 2.0 and MT-Bench are consistent with a regularization benefit and do not by themselves confirm the uniqueness mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that DPO, as maximum likelihood estimation of the Bradley-Terry model, can have multiple minimizers in the infinite response space, so the unique RLHF-optimal policy may be unattainable. It proposes the Infinite Preference Model (IPM), an energy-based model whose MLE is unique and equivalent to the RLHF minimizer, and derives a contrastive loss, Energy Preference Alignment (EPA), which uses strong and weak negatives. The empirical section reports that EPA outperforms DPO and other baselines on AlpacaEval 2.0 and MT-Bench with Mistral-7B on Ultrafeedback data.","tokens_in":22468,"tokens_out":3124,"duration_ms":23513,"significance":"The central negative claim about DPO is valuable and mostly independent of the positive proposal: Proposition B.5 gives a concrete construction showing that an unsampled response breaks uniqueness of the DPO minimizer, which is a real caveat for offline alignment in infinite output spaces. The positive claim, that the IPM has a unique MLE equal to the RLHF minimizer, is a clean theoretical identity, though largely a restatement of the Boltzmann/energy parameterization. The empirical gains of EPA over DPO and several other baselines on standard benchmarks are consistent across tables and ablations, and the paper includes helpful controls such as adding weak negatives to DPO (Table 5), slope-1 linearity probing (Table 7), and an alignment-tax table (Table 6). The main weakness is the gap between the theoretical convergence guarantee (Theorem 3.3) and the actual EPA loss defined in Eq. (11)/(12), where the negative sampling distribution is not shown to satisfy the balance condition.","major_comments":[{"comment":"Theorem 3.3 states that the contrastive estimate in Eq. (9) converges almost surely to the energy discrepancy only when Z satisfies the balance condition. In Eq. (11), weak negatives are mismatched responses from other prompts, drawn from a distribution that is not specified as a perturbation kernel p(z|y) for the positive y, and no argument is given that the balance condition holds. Therefore the almost-sure convergence guarantee does not formally apply to the implemented EPA loss, and the paper's own statement in Section 3.2 that the condition is only sufficient does not fill this gap. The theoretical bridge from the IPM's unique MLE to the practical EPA loss is thus not established by the manuscript as written.","section":"Theorem 3.3 and Eq. (11)/(12)"},{"comment":"Assumption B.1, which is required for the IPM partition function to be finite, states that the set of possibly preferred responses for each prompt is finite. In the later justification in Section 4.1, the authors appeal to this assumption to argue that a dataset preferred y_w is a good approximation of a positive sample from p(y|x). However, Theorem 3.1 as stated in the paper is essentially a restatement of the uniqueness of the Boltzmann MLE within that finite support; the discussion in Section 3.1 frames the contribution as taking infinity into account, so the reader should be told explicitly that the formal theorem only covers the finite-support case and that the infinite-support case is handled only by Assumptions B.1/B.2 as a sufficient condition for the partition function to exist.","section":"Assumption B.1 and Section 3.1"},{"comment":"The empirical comparison in Table 1 reports EPA \"a new state of the art\", but the significance of the gains on MT-Bench is small (7.71 vs. 7.55 for DPO, and 7.58 for EPA-general vs. 7.57 for DPO-PL), with no variance reported for the single-seed runs in Table 1. Since the central claim is that EPA is closer to the RLHF optimum, the paper would be strengthened by reporting multiple seeds or a significance test for at least the headline comparisons; as it stands, the AE 2.0 gains are large but the MT-Bench gains are borderline.","section":"Table 1 and Section 5.2.1"},{"comment":"The gradient analysis in Section 4.3 (Eq. 13) interprets the weak contrast as a regularization term and states that DPO's gradient is recovered when there is no weak contrast. This is correct, but the empirical results in Tables 2 and 5 suggest that the benefit of EPA may come substantially from this regularization effect rather than from the IPM uniqueness mechanism. The paper does not clearly separate these two explanations; a concrete test would be to evaluate whether a DPO loss with the same weak-negative regularization achieves the same or similar AE 2.0 performance, which Table 5 partially addresses but only with added weak negatives as data rather than as in-batch regularizers.","section":"Section 4.3 and Tables 2, 5"}],"minor_comments":[{"comment":"In Eq. (9), the notation uses N as both the sample count and the number of weak negatives in later notation; the subscript/superscript conventions for N_weak^- and N_strong^- are introduced only in Table 2, which is fine, but Eq. (9) itself should clarify whether M is the number of negatives per positive.","section":"Eq. (9)"},{"comment":"The statement in Section 1.1 that the reward r_theta for weak negatives is \"not substantially lower\" than for strong negatives in Figure 1 is hard to verify from the figure as presented; labeling the axes and adding a caption that explains the y-axis (log ratio reward) would help.","section":"Figure 1 and Section 1.1"},{"comment":"In Table 5, the notation \"+ UF-weak x1\" is not defined before the table; a sentence in Appendix C explaining that it adds a copy of UF-binarized with y_l replaced by a random weak negative would improve readability.","section":"Table 5"},{"comment":"The proof sketch for Theorem 3.3 in Appendix B.3 says the balance condition enables a statistic estimate but does not state which form of the Strong Law of Large Numbers is used for the inner logarithm; citing the specific result and stating the integrability conditions would make the proof easier to check.","section":"Appendix B.3"},{"comment":"In the related work, the comparison with infoNCA claims infoNCA is \"a worse-performing ablation version of EPA\", but this claim is not backed by a formal derivation; either add a short derivation or soften the claim.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The reader's report and my own reading agree on the main technical gap: the practical EPA loss in Eq. (11)/(12) is not shown to satisfy the balance condition of Theorem 3.3, so the almost-sure convergence guarantee does not transfer. This is a load-bearing issue for the paper's central theoretical claim. However, the empirical contribution is solid and the negative result on DPO (Proposition B.5) is a useful caveat. I recommend major revision rather than rejection, since the empirical claims are clearly strong enough to be a useful contribution and the theoretical gap could be addressed in principle by either (a) proving convergence under a weaker condition, (b) showing the balance condition approximately holds for the specific sampling scheme, or (c) reframing the paper's claims to present EPA as a regularized contrastive loss and the IPM uniqueness as motivation rather than as a proven convergence guarantee."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — this paper is worth your time, but keep the theoretical claims and the empirical claims on separate shelves.\n\nWhat's genuinely new: the observation that DPO's Bradley-Terry backbone can have multiple MLEs in infinite candidate spaces, imported from learning-to-rank with proper citations (Ford 1957, Bong & Rinaldo 2022), and the construction of an energy-based alternative with a unique MLE tied to slope-1 linearity. Proposition B.5 is a clean, self-contained demonstration that one unsampled response breaks uniqueness of the DPO minimizer. The EPA loss — contrasting each positive against strong and weak negatives — is a real artifact, and the ablations show the weak negatives help. The gradient analysis in Section 4.3 gives a plausible mechanism: weak contrast as regularization. On the empirical side, EPA consistently beats DPO and other baselines on AlpacaEval 2.0 and MT-Bench, and the slope-1 probing in Table 7 is a nice direct check.\n\nWhere it gets soft: the bridge from IPM's unique MLE to the implemented EPA loss is not established. Theorem 3.3's almost-sure convergence requires the negative perturbation source Z to satisfy the balance condition, and the paper never specifies a kernel for which mismatched responses from other prompts satisfy it. The paper openly calls the condition sufficient, not necessary — that's honest, but it means the a.s. claim in the abstract doesn't transfer to the actual loss. The IPM uniqueness theorem also rests on Assumption B.1 (finite support of p(y|x)) which is stated but not defended for language. The empirical evidence is real but thin: main tables are single-seed, checkpoints are selected by MT-Bench, and no code is released. The alignment tax table shows EPA slightly worse on GSM8k/MMLU, which is acknowledged but not resolved.\n\nIf the uniqueness mechanism were the only story, I'd call this a theory paper with a broken bridge. But the empirical gains are consistent and the weak-negative regularization story is independently plausible. This deserves a serious referee: the referee should push for either a proof (or explicit relaxation) of the balance condition for the actual negative sources, multi-seed runs, and code release.\n\nRecommendation: send to peer review, conditional on those experiments. I'd cite the DPO non-uniqueness point myself.","headline":"Solid negative result on DPO's non-uniqueness, a real but unproven theoretical bridge to EPA, and enough empirical consistency to merit a serious referee.","tokens_in":23047,"tokens_out":2270,"would_cite":true,"duration_ms":20521,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DPO can miss the true RLHF-optimal policy because its underlying Bradley-Terry model can have multiple maximum-likelihood fits; the proposed energy-based model EPA has a unique fit and outperforms DPO on open benchmarks.","keywords":["Direct Preference Optimization (DPO)","Bradley-Terry model","energy-based model","preference alignment","RLHF","contrastive loss","maximum likelihood estimation","offline alignment"],"falsifier":"A direct test would train DPO and EPA on synthetic data with a known true reward, deliberately including for each prompt a response that never appears in any preference pair, then measure on held-out responses whether the learned log-ratio reward satisfies $r_\\theta(x,y)=r_{\\mathrm{true}}(x,y)+C(x)$. If DPO also converges to a unique slope-1 solution, the claimed failure mode is not triggered in that regime; if EPA's closeness to slope-1 linearity does not improve as weak negatives are added, the practical negative-sampling assumption is the weak link.","tokens_in":21863,"feed_emoji":"🎯","tokens_out":13275,"duration_ms":105759,"temperature":0.7,"pith_summary":"Offline preference alignment via DPO is supposed to find the unique policy that minimizes the KL-constrained RLHF objective, which is equivalent to a log-ratio reward $r_\\theta(x,y)=\\beta\\log(\\pi_\\theta(y|x)/\\pi_{\\mathrm{ref}}(y|x))$ sitting in slope-1 linearity with the true reward. The paper's core claim is that this target can be unattainable for DPO, because the Bradley-Terry model behind the DPO loss can have multiple maximum-likelihood estimators over the infinite space of responses, and only one of those estimators satisfies the required linearity. The authors propose an energy-based model, the Infinite Preference Model, whose maximum-likelihood estimator is provably unique and exactly matches the RLHF minimizer, together with a tractable contrastive loss, EPA, that contrasts each preferred response with strong rejected responses and many weak mismatched responses. They prove that under a symmetric negative-sampling condition the approximation error of EPA vanishes almost surely as the number of negatives grows, and they show empirically that EPA-trained policies come closer to slope-1 linearity and score higher than DPO-trained policies on MT-Bench and Alpaca-Eval 2.0. If the argument is right, offline alignment should move from pairwise Bradley-Terry fitting to energy-based modeling over the full response space.","feed_headline":"Energy-based model beats DPO with a guaranteed unique optimum","feed_subtitle":"EPA's contrastive loss with weak negatives lands closer to the true RLHF reward than DPO on MT-Bench and Alpaca-Eval 2.0.","key_machinery":"The load-bearing object is the Infinite Preference Model (IPM), an energy-based model that assigns each response a Boltzmann probability $p(y|x)\\propto\\exp(r_{\\mathrm{true}}(x,y))$ over the effectively infinite set of possible responses. Its MLE, when the model is parameterized by the log-ratio reward $r_\\theta(x,y)=\\beta\\log(\\pi_\\theta(y|x)/\\pi_{\\mathrm{ref}}(y|x))$, is shown to exist uniquely and to coincide exactly with the minimizer of the KL-constrained RLHF objective (Theorem 3.1). The second piece of machinery is the energy-discrepancy functional, whose unique global minimizer is the IPM itself; the EPA loss is a finite-sample estimator of that functional, and its approximation error vanishes almost surely when the negative-sampling source satisfies the symmetry condition $\\sum_y p(z|y)f(y)=\\sum_y p(y|z)f(y)$. The practical engine is the contrast between one preferred response, one or more strong rejected responses, and many weak mismatched responses, which the paper argues reflects the theorem's requirement that negatives be informative of the positive and high-variance.","core_discovery":"The paper's central discovery is a failure mode in DPO and a replacement that avoids it. Because the space of possible responses to a prompt is effectively infinite, the Bradley-Terry preference model's maximum-likelihood estimator is not guaranteed to be unique; the DPO loss then has multiple minimizers, and the unique minimizer of the KL-constrained RLHF loss is only one of them. The proposed replacement, the Infinite Preference Model (IPM), treats $p(y|x)$, the probability that response $y$ is preferred over all other responses, as a Boltzmann distribution over the true reward, $p(y|x)\\propto\\exp(r_{\\mathrm{true}}(x,y))$. Parametrized by the log-ratio reward, IPM has a unique MLE, and that MLE is reached exactly when $r_\\theta(x,y)=r_{\\mathrm{true}}(x,y)+C(x)$ holds, which is the same condition that characterizes the RLHF minimizer. The practical loss EPA approximates the IPM's MLE by maximizing the softmax probability of the preferred response against strong negatives (the rejected responses) and weak negatives (mismatched responses from other prompts); the authors show that the approximation error of this estimator almost surely vanishes when negatives are drawn from a source satisfying the energy-discrepancy symmetry condition, and that EPA-trained models are measurably closer to slope-1 linearity than DPO-trained models.","pith_inferences":["Beyond the paper: the non-uniqueness argument should transfer to any pairwise or Plackett-Luce preference loss, so IPO, DPO-PL, and similar methods may carry the same unattainability risk in infinite response spaces.","Beyond the paper: the paper's slope-1 linearity probe could be adopted as a general diagnostic for offline alignment, independent of preference-model family, which would make the theoretical target directly measurable.","Beyond the paper: testing EPA on synthetic data with known rewards and varying number of weak negatives would reveal whether the approximation error decreases monotonically toward zero or plateaus because of the mismatched-negative source; Theorem 3.3 predicts the former only under its symmetry condition.","Beyond the paper: the symmetry condition behind the almost-sure convergence could serve as a principled criterion for negative selection in other contrastive learning settings, such as metric learning or sentence embedding, where hard-negative mining is currently heuristic."],"forward_implications":["Offline alignment methods should be evaluated by whether the learned log-ratio reward achieves $r_\\theta(x,y)=r_{\\mathrm{true}}(x,y)+C(x)$ on held-out responses, not only by pairwise ranking accuracy.","Having multiple strong negatives per prompt, as in multi-response preference datasets, improves the approximation to the IPM's MLE and is a practical reason to collect or generate several rejected responses per prompt.","Adding more preference pairs cannot by itself guarantee that DPO reaches the RLHF optimum: the pair-sampling distribution over the infinite response space must be structured so that the Bradley-Terry MLE is unique.","Loss-modification tricks such as margins and on-policy weighting improve both DPO and EPA, but they do not remove the underlying non-uniqueness, and EPA with the same tricks remains ahead.","The weak-contrast term in EPA's gradient acts as a regularizer, making training less prone to the rapid overfitting seen with DPO after the first epoch."],"supporting_citations":[{"why":"Establishes DPO and its claim that the Bradley-Terry MLE satisfies the slope-1 linearity; this is the method and target the paper challenges.","marker":"Rafailov et al. (2023)"},{"why":"Derives the analytical minimizer of the KL-constrained RLHF objective that underlies the slope-1 linearity equivalence.","marker":"Korbak et al. (2022a)"},{"why":"Introduces energy discrepancy and the negative-sampling symmetry condition from which EPA and its convergence guarantee are built.","marker":"Schröder et al. (2023)"},{"why":"Classic analysis of Bradley-Terry maximum likelihood in ranking problems, cited as the origin of the uniqueness problem.","marker":"Ford (1957)"},{"why":"Supplies known sufficient conditions for unique BTL MLEs, highlighting the gap between those conditions and DPO's unconstrained setting.","marker":"Bong & Rinaldo (2022)"},{"why":"Shows pairwise losses can lose correlation with the RLHF objective under off-policy data, supporting the need for a different preference model.","marker":"Tang et al. (2024)"},{"why":"Provides the on-policy weighting trick used to compare BTM-based and EBM-based methods under identical loss modifications.","marker":"Zhou et al. (2024)"}],"fun_headline_variants":["Energy-based model fixes DPO's multiple-minima issue","Unique MLE preference model beats DPO in alignment","EPA: energy model with unique optimum outperforms DPO","Guaranteed unique optimum: energy model beats DPO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the mismatched-response negatives used in EPA training behave enough like the carefully chosen perturbation source for which the convergence proof works; if that fails, the nearness of EPA's optimum to the RLHF minimizer is not guaranteed.","fun_headline_variants_meta":{"raw":{"variants":["Energy-based model fixes DPO's multiple-minima issue","Unique MLE preference model beats DPO in alignment","EPA: energy model with unique optimum outperforms DPO","Guaranteed unique optimum: energy model beats DPO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000275,"raw_usage":{"total_tokens":1731,"prompt_tokens":1123,"completion_tokens":608,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":739,"completion_tokens_details":{"reasoning_tokens":551}},"tokens_in":739,"tokens_out":608,"duration_ms":5699,"temperature":1.0,"reasoning_tokens":551,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:43:34.790291+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would train DPO and EPA on synthetic data with a known true reward, deliberately including for each prompt a response that never appears in any preference pair, then measure on held-out responses whether the learned log-ratio reward satisfies $r_\\theta(x,y)=r_{\\mathrm{true}}(x,y)+C(x)$. If DPO also converges to a unique slope-1 solution, the claimed failure mode is not triggered in that regime; if EPA's closeness to slope-1 linearity does not improve as weak negatives are added, the practical negative-sampling assumption is the weak link.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Classic analysis of Bradley-Terry maximum likelihood in ranking problems, cited as the origin of the uniqueness problem."},{"cited_title":"and Rinaldo, A","cited_arxiv_id":null,"evidence_quote":"Supplies known sufficient conditions for unique BTL MLEs, highlighting the gap between those conditions and DPO's unconstrained setting."}],"review_version":1}