Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimator

Fang Liu; Mohammad Ghavamzadeh; Xuan Zhu; Yanjun Qi; Zhuotong Chen

REVIEW 3 major objections 5 minor 1 cited by

Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimator

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Preference optimization, the paper claims, is equivalent to estimating an intractable normalization constant; sampling dispreferred completions by reward via a contrastive-divergence kernel improves alignment over existing baselines.

desk verdict A useful empirical recipe for reward-proportional hard-negative sampling in preference optimization, but the theoretical narrative does not cover the implemented algorithm; revise and resubmit. read the letter →

arxiv 2502.04567 v1 pith:H2WOAKZ7 submitted 2025-02-06 cs.AI

Zhuotong Chen , Fang Liu , Xuan Zhu , Yanjun Qi , Mohammad Ghavamzadeh This is my paper · ORCID

classification cs.AI

keywords preferenceoptimizationcontrastivedivergencenegativelog-likelihoodestimationnormalizationconstanthardsamplingMarkovchainMonteCarlodirectonline

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most preference-optimization methods pick the dispreferred completion by a heuristic, such as maximizing the margin between preferred and rejected responses. This paper argues that those rejected completions are doing statistical work: they are samples used to estimate the normalization constant of the unnormalized model $p_\theta(y|x)=\mu(y|x)\exp(\beta r_\theta(x,y))/Z_\theta(x)$, with the preferred completion as the observed data. On that reading, choosing the dispreferred completion is a sampling problem, and the paper proposes MC-PO, which uses a contrastive-divergence Markov-chain kernel to sample a candidate proportional to the current reward model, plus OnMC-PO, its online extension. If the claim is right, negative-sample selection stops being an algorithmic afterthought and becomes a controllable component of alignment; MC-PO beats existing preference-optimization baselines in five of six comparisons and OnMC-PO improves further on AlpacaEval 2 and Arena-Hard.

What carries the argument

The load-bearing object is the unnormalized reward-weighted model $p_\theta(y|x)=\mu(y|x)\exp(\beta r_\theta(x,y))/Z_\theta(x)$ with $r_\theta(x,y)=\log(\pi_\theta(y|x)/\pi_{\mathrm{ref}}(y|x))$, together with the identity $\nabla_\theta\log Z_\theta(x)=\mathbb{E}_{p_\theta(y|x)}[\nabla_\theta \beta r_\theta(x,y)]$, which turns normalization into a sampling problem. The mechanism that carries the argument is the contrastive-divergence Monte Carlo kernel: given a preferred completion $y_0$, draw $L$ candidates from a proposal $\mu$, set weights $w_i\propto\exp(\beta r_\theta(x,y_i))$, and sample the dispreferred completion from this categorical distribution. This kernel runs one MCMC step, targets the high-reward region where the gradient integrand is largest, and produces hard negatives; Lemma 3.2 shows such a pair yields a larger gradient magnitude than an easy pair. The same kernel is used online in OnMC-PO, where the preferred completion is re-sampled from the current policy to approximate sampling from $p_\theta$.

What would settle it

Fix a prompt, reference policy, and checkpoint; using many Monte Carlo samples from $\mu$, compute an accurate estimate of $\nabla_\theta\log Z_\theta(x)$, and compare it with the OnMC-PO estimator whose $y_0$ is the best of $k$ draws from $\pi_\theta$ with the MC-kernel dispreferred samples. If the two gradient estimates disagree beyond sampling noise, the unbiasedness claim fails even if the trained model still performs well.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the log-margin objective of DPO is a special case of negative-log-likelihood estimation over $p_\theta(y|x)=\mu(y|x)\exp(\beta r_\theta(x,y))/Z_\theta(x)$, with the preferred completion as the observation and the dispreferred completions as noise samples that estimate $Z_\theta(x)$. With one noise sample, the sampling-based loss $-\beta r_\theta(x,y_0)+\log\sum_{i=0}^{1}\exp(\beta r_\theta(x,y_i))$ is exactly the DPO loss, so DPO is the $M=1$ case of a broader sampling scheme. The paper then takes the sampling seriously: the gradient of the log-normalization constant is an expectation under $p_\theta$, and contrastive divergence's Monte Carlo kernel samples a dispreferred completion from a categorical distribution over candidates weighted by $\exp(\beta r_\theta(x,y_i))$, i.e., a hard negative. In the online setting, the paper proves that if the preferred completion is sampled from $p_\theta$, the resulting gradient estimator is unbiased, and it builds OnMC-PO by re-sampling completions from the current policy and keeping the best-ranked one. Across two model families and three setups, MC-PO outperforms previous preference-optimization baselines in five of six comparisons, and OnMC-PO gives further gains.

Load-bearing premise

The online method's guarantee relies on the preferred completion being drawn from the reward-weighted distribution, but in practice it is just the best of several completions sampled from the current policy, and the paper does not show these two ways of sampling agree.

Editorial extensions

If this is right

If the equivalence holds, every preference-optimization loss is implicitly choosing a sampler for $Z_\theta(x)$; improving that sampler should improve alignment without collecting new preferences.
Sampling dispreferred completions proportionally to $\exp(\beta r_\theta)$ produces harder negatives and larger, more effective policy updates than random or max-margin selection.
Increasing the number of dispreferred candidates $M$ helps MC-PO in the reported runs (AlpacaEval 35.84 to 37.40; Arena-Hard 63.77 to 66.16 on Llama-3.1-8B-SFT), whereas random sampling does not improve with $M$.
Because DPO is the $M=1$ case, MC-PO extracts more signal from a fixed candidate set; on the paper's comparison, training DPO on a four-fold augmented dataset (34.18 winrate) did not match MC-PO (35.84) on AlpacaEval.
OnMC-PO's batched re-sampling gives the largest reported winrates (72.63 on AlpacaEval with Llama-3.1-8B-Instruct), so online regeneration of candidates is a direct lever for alignment quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the normalization-estimator reading is right, comparing negative-selection heuristics reduces to comparing estimator variance for $\nabla_\theta\log Z_\theta$; a heuristic's variance ranking on a fixed candidate pool should predict its alignment winrate, which is testable on the paper's own datasets.
The unbiasedness proof in Proposition 3.3 assumes $y_0\sim p_\theta$, yet OnMC-PO samples $y_0$ as the best of several draws from $\pi_\theta$; a direct check would replace that heuristic with an actual $p_\theta$ sampler (for instance a short MCMC chain) and ask whether gains grow, stay, or shrink.
Because the paper notes only one MCMC step is used, a natural extension is a multi-step chain; the framework predicts multi-step sampling gives still-better negatives and larger alignment gains, at extra compute.
If DPO is $M=1$ NLL estimation, then preference data collection itself becomes a design target: collecting more ranked candidates per prompt is not just more data but a better estimator of the partition function, which could lower annotation cost.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea here is worth taking seriously: reformulate preference optimization as NLL estimation, then use a contrastive-divergence MC kernel to pick dispreferred completions proportionally to the current reward. That is a clean way to think about negative sampling, and it is genuinely new as an algorithmic recipe. The derivation that DPO is the M=1 case of the RNCE/InfoNCA loss is correct, and the paper is honest that Eq. (7) is close to InfoNCA. The empirical work is also solid: multiple seeds, standard deviations, three model setups, and an ablation (Max vs Min vs MC kernel) that supports the claim that reward-proportional sampling helps. Table 3's result that MC-PO improves with more negatives while RNCE does not is a real signal.

The soft spots are mostly in the theory-to-algorithm correspondence, and they are not minor. Proposition 3.3 proves unbiasedness when the preferred completion is sampled from pθ, but OnMC-PO samples from πθ. With µ=πref and β=0.01, pθ ∝ πref^0.99 πθ^0.01, which is very far from πθ, so the theorem does not apply to the implementation. Just as importantly, the paper analyzes the objective in Eq. (7) with M+1 samples all used in the softmax, while Algorithm 1 draws a single sample from the categorical and then uses that sample as the negative. The expected gradient of that sampled DPO loss differs from the gradient of Eq. (7) by an extra sigmoid factor (except for M=1), so the unbiasedness/optimality claims do not cover the actual training loss. If instead the authors intend the loss to be Eq. (7) with all L candidates included, then the kernel is not doing any selection and the framing collapses. The paper never clarifies this, and Table 5's objective with M negatives does not resolve it because the sampling procedure for M>1 is not described. It is also true that no formal proof of the claimed MC-kernel optimality appears anywhere in the text.

None of this kills the empirical finding. The algorithm plausibly works because it is a reasonable heuristic for hard-negative mining, and the results are internally consistent. But the theory as written is decorative rather than load-bearing. The paper needs a major revision that either analyzes the actual loss being optimized or changes the algorithm to match the theory.

Who gets value: researchers working on preference optimization and alignment, particularly anyone interested in how to select dispreferred completions. It also serves as a practical counterpoint to the usual random or margin-based choices. I would send it to a serious referee, with the expectation that the theoretical claims will need to be substantially scaled back or reworked. I would probably also cite it if I work on negative sampling for PO; the empirical recipe is more durable than the framing.

Referee Report

3 major / 5 minor

Summary. The paper proposes a framework that reinterprets preference optimization (PO) as negative log-likelihood (NLL) estimation of a probability model pθ(y|x) ∝ μ(y|x) exp(β rθ(x,y)) with rθ(x,y) = log(πθ(y|x)/πref(y|x)), so that dispreferred completions act as importance samples for the intractable normalization constant. The authors show that with one negative sample the NLL-based loss reduces exactly to the DPO loss, and they propose MC-PO, which uses a Monte Carlo kernel (Algorithm 1) to sample hard negatives proportionally to exp(βrθ), and OnMC-PO, an online variant that samples preferred completions from the current policy. Experiments on AlpacaEval 2 and Arena-Hard with Llama-3.1-8B and Mistral-7B report consistent gains over several PO baselines, with ablations on the number of negatives and on sampling strategies.

Significance. If the theoretical claims were fully established, this would be a valuable contribution: it would turn the choice of dispreferred completions in PO from a heuristic into a sampling-estimation problem and would connect PO to the contrastive-divergence literature. The algebraic identity between Eq. (7) with M=1 and DPO is correct and is cleanly presented; the paper also provides extensive, multi-seed experiments with several baselines, and the ablations in Section 5.3 and Table 3 are informative. The main weakness is that the theoretical machinery does not actually analyze the algorithm that is implemented: the MC kernel of Algorithm 1 is not the estimator analyzed in Lemma 3.1, and the online unbiasedness claim in Proposition 3.3 relies on a sampling assumption that the OnMC-PO implementation does not satisfy. These gaps are load-bearing for the paper's central claim that contrastive divergence provides theoretical guidance for sampling dispreferred completions, though the empirical results may still stand on their own as evidence that reward-proportional hard-negative selection helps.

major comments (3)

[Sec. 3.1 / Algorithm 1] The implemented MC-PO objective does not match the estimator analyzed in the theory. Eq. (7) is the full softmax loss over all M candidates, whose gradient is β Σ_{i=1}^M w_i (∇r_i − ∇r_0) with w_i ∝ exp(βr_i). Algorithm 1, however, draws a single candidate z from the categorical distribution over the candidate set and the paper's text describes this as selecting one hard negative. If training then uses a pairwise DPO loss with the selected negative, the expected gradient is Σ_i p_i σ(β(r_i−r_0)) β(∇r_i−∇r_0), where p_i is the categorical probability. This differs from the Eq. (7) gradient by the sigmoid factor σ(β(r_i−r_0)) and is not proportional to it in general (already for M=1 with a candidate set larger than one). If, instead, training uses Eq. (7) over all candidates, then the categorical draw in Algorithm 1 has no effect on the gradient and the kernel is not actually selecting negatives. The manuscript does not specify which loss is used when M>1 or how the kernel output enters the loss, so the theoretical optimality/unbiasedness claims about the MC kernel do not apply to the reported procedure.
[Sec. 3.1 / Lemma 3.1 / Appendix A.3] Lemma 3.1 is proven for a different estimator than the one in Algorithm 1. The proof in Appendix A.3 starts from E_μ[(pθ/μ)∇βrθ] and applies self-normalized importance sampling using all M+1 samples: it approximates ∇ log Z by Σ_{i=0}^M [exp(βr_i)/Σ_j exp(βr_j)] ∇βr_i. This is the gradient of log Σ_{i=0}^M exp(βr_i), i.e., it corresponds to using every candidate as a weighted sample. Algorithm 1, in contrast, draws a single output from a categorical distribution over the candidates and does not use a weighted average over all samples. One-step contrastive divergence, as classically defined, would generate one MCMC transition from the current observation, not resample from a categorical distribution over an i.i.d. proposal batch. Thus the statement that 'CD shares the same objective function with RNCE' is not established for the kernel that the paper actually deploys, and the interpretation of the drawn sample as a hard negative is not justified by the provided derivation.
[Sec. 3.2 / Proposition 3.3] The unbiased-gradient claim of Proposition 3.3 requires that the preferred completion y0 be sampled from the probability model pθ(y|x) ∝ μ(y|x) exp(βrθ(x,y)). With the paper's choice μ = π_ref and rθ = log(πθ/π_ref), this model equals pθ ∝ π_ref^{1−β} πθ^β. In the experiments β is set to 0.01, so pθ is overwhelmingly dominated by π_ref, not by the current policy πθ. The OnMC-PO implementation, however, samples several completions from πθ and selects the judge-best one, with no argument that this approximates pθ. Consequently, the unbiasedness result does not apply to the reported OnMC-PO procedure. The text's assertion in Section 3.2 that online PO algorithms generate preferred completions 'from the target policy that is proportional to the probability model pθ' is incorrect unless β=1 and μ=π_ref; the paper gives no such condition.

minor comments (5)

[Sec. 4] The word 'contastive' should be 'contrastive', and 'Bradely-Terry' should be 'Bradley-Terry' in the related-work section.
[Appendix A.3] The derivation of Lemma 3.1 ends with 'Threfore' instead of 'Therefore'.
[Appendix A.5] In the proof of Proposition A.5, the expression '∇θ exp(β log rθ(x, yi))' appears; since rθ already denotes a log-ratio, the argument should almost certainly be 'β rθ(x, yi)', not 'β log rθ(x, yi)'.
[Table 2] The table formatting for DPO(−) shows '1.08(±0.6)' and '3.17(±0.9)' where the standard deviations are missing the leading '±' symbol in the second entry; this is likely a typographical issue.
[Sec. 3.1] The contribution bullet states that the MC kernel samples 'proportionally to the log-likelihood of the target policy', but Algorithm 1 samples proportionally to exp(βrθ(x,y)), which equals (πθ/π_ref)^β; the wording is imprecise because the proposal μ also enters the unnormalized model.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the NLL-to-DPO mapping is an algebraic identity, the MC-kernel claims are conditional on stated sampling assumptions, and the empirical claims are benchmarked against external data and baselines.

full rationale

The central derivation is self-contained and algebraic. Equation (5) defines p_theta(y|x) proportional to mu(y|x) exp(beta r_theta(x,y)) with r_theta = log(pi_theta/pi_ref), and Proposition 2.1 derives Eq. (7) through the standard RNCE posterior-classification calculation; setting M=1 reduces Eq. (7) to the DPO logistic loss. This is an identity, not a circular reduction, and it reproduces a known external result (InfoNCA/RNCE) rather than importing a conclusion from the authors' own prior work. The MC kernel in Algorithm 1 is an importance/resampling scheme for the intractable gradient in Eq. (8), and Proposition 3.3 is a conditional mathematical statement: if y0 is sampled from p_theta, then the self-normalized estimate of the log-partition gradient is unbiased. That theorem is not used as an input to define the objective; it is a derived property under an explicit assumption. The empirical evaluation uses external benchmarks (AlpacaEval 2 and Arena-Hard) and standard baselines, so the reported winrates are not fitted values recycled as predictions. The only self-citation that appears (Chen et al., 2024b) is for the iterative pairwise ranking data-generation procedure in the instruct setup; it is not load-bearing for the theoretical claims or for the benchmark comparisons. Two nontrivial concerns exist, but they are implementation-assumption gaps rather than circularity: OnMC-PO's practical implementation samples preferred completions from pi_theta and keeps the judge-best, whereas Proposition 3.3 requires y0 ~ p_theta, and these distributions differ (with mu=pi_ref and beta=0.01, p_theta is close to pi_ref, not pi_theta). Similarly, Algorithm 1's single sampled negative is not obviously equivalent to the all-candidate loss in Eq. (7) when M>1, so the theoretical justification may not exactly cover the implemented loss. These are validity and correctness risks, not instances of a derivation reducing to its own inputs. The paper also honestly states its limitation that only one MCMC step is used, which is the opposite of hiding a gap. Therefore no circular step is exhibited, and the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The derivation rests on standard preference-modeling assumptions plus support and proposal-distribution assumptions. No new physical entities are introduced; pθ is a parametric model only. The proposal distribution in the base setup is an implicit mixture of LLMs, which is not fully specified and is a practical weakness.

free parameters (3)

beta = 0.01
KL and reward scaling in Eq. (5) and all preference-optimization losses; it controls how much pθ differs from µ and directly affects the theoretical claim about πθ versus pθ in the online setting.
M (number of sampled dispreferred completions) = 1 in main results; 2 and 3 in ablations
Algorithm design choice that sets the number of negatives in the sampled loss; varied in Sec. 5.4, Table 3.
Candidate set size L = 5 from Nectar (after removing rank-2); 6 from UltraFeedback regeneration
The MC kernel draws from this finite candidate set; the choice affects sampling quality and is a data-processing decision.

assumptions (5)

domain assumption Bradley-Terry model of pairwise preferences.
Used to derive DPO in Eq. (3) and throughout; standard but not derived from first principles.
domain assumption Closed-form solution of KL-constrained RLHF in Eq. (2).
Invoked in Sec. 2.1 as π*(y|x) proportional to πref exp(r/β); cited to Go et al. (2023).
domain assumption Support coverage: pθ(y|x) > 0 whenever π*(y|x) > 0 for all x and all θ.
Stated in Sec. 2.2; needed for the NLL minimization objective to be well-posed.
domain assumption Proposal distribution µ is known and samplable.
Algorithm 1 and Prop. 3.3 require sampling from µ; Sec. B.1 defines µ as the reference policy in the instruct setup and an implicit mixture of LLMs in the base setup.
domain assumption Transitivity and symmetry of pairwise comparisons for iterative ranking.
Appendix B.1, used to reduce L candidate comparisons to L-1 when constructing instruct-set labels via Chen et al. (2024b).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimator." pith.science (2026). https://pith.science/paper/H2WOAKZ7

@misc{pith2026250204567,
  author       = {Pith},
  title        = {Pith review of: Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimator},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H2WOAKZ7}},
  note         = {Machine review of arXiv:2502.04567}
}

read the original abstract

Existing studies on preference optimization (PO) have centered on constructing pairwise preference data following simple heuristics, such as maximizing the margin between preferred and dispreferred completions based on human (or AI) ranked scores. However, none of these heuristics has a full theoretical justification. In this work, we develop a novel PO framework that provides theoretical guidance to effectively sample dispreferred completions. To achieve this, we formulate PO as minimizing the negative log-likelihood (NLL) of a probability model and propose to estimate its normalization constant via a sampling strategy. As we will demonstrate, these estimative samples can act as dispreferred completions in PO. We then select contrastive divergence (CD) as the sampling strategy, and propose a novel MC-PO algorithm that applies the Monte Carlo (MC) kernel from CD to sample hard negatives w.r.t. the parameterized reward model. Finally, we propose the OnMC-PO algorithm, an extension of MC-PO to the online setting. On popular alignment benchmarks, MC-PO outperforms existing SOTA baselines, and OnMC-PO leads to further improvement.

Figures

Figures reproduced from arXiv: 2502.04567 by the authors.

**Figure 1.** Left: existing studies choose a dispreferred completion as the one that maximizes the gap with the preferred completion based on human (or AI) ranked scores. Right: we propose theoretical guidance to sample dispreferred completion(s) proportionally to the parameterized reward model. As the parameters evolve during training, the sampling of dispreferred completion changes. probability mass w.r.t. the probability mod… view at source ↗

**Figure 2.** Winrate evaluation of the optimized Llama-3.1-8B-SFT model using MC-PO, versus its Max, and Min sampling based variants. Five modified Nectar datasets are used for training. x negs represents that the training dataset contains x negative candidates for each input prompt. For example, the 3 negs dataset is constructed by removing rank-2 and rank-3 completions from the Nectar dataset. tion. MC-PO, however, samples a … view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

York's Cavity Formalism and Quantum Modified Thermodynamics of (2+1)D Black Holes
gr-qc 2025-06 reject novelty 4.0 of 10

The paper claims Barrow entropy corrections reshape BTZ black hole thermodynamics in a cavity, but the free energy analysis rests on a wrong extrinsic curvature term and an incorrect zero-crossing interpretation.

Reference graph

Works this paper leans on

39 extracted references · 9 canonical work pages · cited by 1 Pith paper

[1]

G., Guo, Z

Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp.\ 4447--4455. PMLR, 2024

2024
[2]

E., and Nocedal, J

Bottou, L., Curtis, F. E., and Nocedal, J. Optimization methods for large-scale machine learning. SIAM review, 60 0 (2): 0 223--311, 2018

2018
[3]

Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

1952
[4]

Noise contrastive alignment of language models with explicit rewards

Chen, H., He, G., Yuan, L., Cui, G., Su, H., and Zhu, J. Noise contrastive alignment of language models with explicit rewards. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 a . URL https://openreview.net/forum?id=KwRLDkyVOl

work page 2024
[5]

Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization

Chen, Z., Liu, F., Zhu, J., Du, W., and Qi, Y. Towards improved preference optimization pipeline: from data generation to budget-controlled regularization. arXiv preprint arXiv:2411.05875, 2024 b

work page Pith review arXiv 2024
[6]

Ultrafeedback: Boosting language models with high-quality feedback, 2024

Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with high-quality feedback, 2024. URL https://openreview.net/forum?id=pNkOx3IVWI

work page 2024
[7]

Anchored preference optimization and contrastive revisions: Addressing underspecification in alignment

D'Oosterlinck, K., Xu, W., Develder, C., Demeester, T., Singh, A., Potts, C., Kiela, D., and Mehri, S. Anchored preference optimization and contrastive revisions: Addressing underspecification in alignment. CoRR, 2024

work page 2024
[8]

The llama 3 herd of models

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

arXiv 2024

Show all 39 references

[9]

Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024

2024 arXiv
[10]

Kto: Model alignment as prospect theoretic optimization

Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024

2024 arXiv
[11]

Aligning language models with preferences through f-divergence minimization

Go, D., Korbak, T., Kruszewski, G., Rozen, J., Ryu, N., and Dymetman, M. Aligning language models with preferences through f-divergence minimization. In Proceedings of the 40th International Conference on Machine Learning, pp.\ 11546--11583, 2023

2023
[12]

Direct language model alignment from online ai feedback

Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024

2024 arXiv
[13]

and Hyv \"a rinen, A

Gutmann, M. and Hyv \"a rinen, A. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp.\ 297--304. JMLR Workshop and Conference Proc...

2010
[14]

Hinton, G. E. Training products of experts by minimizing contrastive divergence. Neural computation, 14 0 (8): 0 1771--1800, 2002

2002
[15]

Towards efficient and exact optimization of language model alignment

Ji, H., Lu, C., Niu, Y., Ke, P., Wang, H., Zhu, J., Tang, J., and Huang, M. Towards efficient and exact optimization of language model alignment. arXiv preprint arXiv:2402.00856, 2024

2024 arXiv
[16]

W., and On, K.-W

Jung, S., Han, G., Nam, D. W., and On, K.-W. Binary classifier optimization for large language model alignment. arXiv preprint arXiv:2404.04656, 2024

2024 arXiv
[17]

H., Gonzalez, J

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

2023
[18]

Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, S., et al. T " ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024

2024 arXiv
[19]

E., and Stoica, I

Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Zhu, B., Gonzalez, J. E., and Stoica, I. From live data to high-quality benchmarks: The arena-hard pipeline, 2024

2024
[20]

Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer

Liu, Z., Lu, M., Zhang, S., Liu, B., Guo, H., Yang, Y., Blanchet, J., and Wang, Z. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405.16436, 2024

2024 arXiv
[21]

Simpo: Simple preference optimization with a reference-free reward

Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024

2024 arXiv
[22]

Adding error bars to evals: A statistical approach to language model evaluations

Miller, E. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint arXiv:2411.00640, 2024

2024 arXiv
[23]

A., Lindsten, F., and Schön, T

Naesseth, C. A., Lindsten, F., and Schön, T. B. Elements of sequential monte carlo, 2024. URL https://arxiv.org/abs/1903.04797

2024 arXiv
[24]

On the connection between noise-contrastive estimation and contrastive divergence

Olmin, A., Lindqvist, J., Svensson, L., and Lindsten, F. On the connection between noise-contrastive estimation and contrastive divergence. In International Conference on Artificial Intelligence and Statistics, pp.\ 3016--3024. PMLR, 2024

2024
[25]

Training language models to follow instructions with human feedback

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

2022
[26]

Smaug: Fixing failure modes of preference optimisation with dpo-positive

Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228, 2024

2024 arXiv
[27]

and Schaal, S

Peters, J. and Schaal, S. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pp.\ 745--750, 2007

2007
[28]

Online dpo: Online direct preference optimization with fast-slow chasing

Qi, B., Li, P., Li, F., Gao, J., Zhang, K., and Zhou, B. Online dpo: Online direct preference optimization with fast-slow chasing. arXiv preprint arXiv:2406.05534, 2024

2024 arXiv
[29]

D., Ermon, S., and Finn, C

Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

2024
[30]

Direct nash optimization: Teaching language models to self-improve with general preferences

Rosset, C., Cheng, C.-A., Mitra, A., Santacroce, M., Awadallah, A., and Xie, T. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715, 2024

2024 arXiv
[31]

Proximal policy optimization algorithms

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

2017 arXiv
[32]

Skalse, J. M. V., Howe, N. H., Krasheninnikov, D., and Krueger, D. Defining and characterizing reward gaming. In Advances in Neural Information Processing Systems, 2022

2022
[33]

Z., Zheng, Z., Calandriello, D., Cao, Y., Tarassov, E., Munos, R., Pires, B

Tang, Y., Guo, D. Z., Zheng, Z., Calandriello, D., Cao, Y., Tarassov, E., Munos, R., Pires, B. \'A ., Valko, M., Cheng, Y., et al. Understanding the performance gap between online and offline alignment algorithms. arXiv preprint arXiv:2405.08448, 2024

2024 arXiv
[34]

Zephyr: Direct distillation of lm alignment

Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944, 2023

2023 arXiv
[35]

Self-play preference optimization for language model alignment

Wu, Y., Sun, Z., Yuan, H., Ji, K., Yang, Y., and Gu, Q. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675, 2024

2024 arXiv
[36]

Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Van Durme, B., Murray, K., and Kim, Y. J. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401.08417, 2024

2024 arXiv
[37]

Starling-7b: Improving llm helpfulness & harmlessness with rlaif, 2023

Zhu, B., Frick, E., Wu, T., Zhu, H., and Jiao, J. Starling-7b: Improving llm helpfulness & harmlessness with rlaif, 2023

2023
[38]

M., Stiennon, N., Wu, J., Brown, T

Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

1909 arXiv
[39]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

G., Guo, Z

Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp.\ 4447--4455. PMLR, 2024

2024

[2] [2]

E., and Nocedal, J

Bottou, L., Curtis, F. E., and Nocedal, J. Optimization methods for large-scale machine learning. SIAM review, 60 0 (2): 0 223--311, 2018

2018

[3] [3]

Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

1952

[4] [4]

Noise contrastive alignment of language models with explicit rewards

Chen, H., He, G., Yuan, L., Cui, G., Su, H., and Zhu, J. Noise contrastive alignment of language models with explicit rewards. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 a . URL https://openreview.net/forum?id=KwRLDkyVOl

work page 2024

[5] [5]

Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization

Chen, Z., Liu, F., Zhu, J., Du, W., and Qi, Y. Towards improved preference optimization pipeline: from data generation to budget-controlled regularization. arXiv preprint arXiv:2411.05875, 2024 b

work page Pith review arXiv 2024

[6] [6]

Ultrafeedback: Boosting language models with high-quality feedback, 2024

Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with high-quality feedback, 2024. URL https://openreview.net/forum?id=pNkOx3IVWI

work page 2024

[7] [7]

Anchored preference optimization and contrastive revisions: Addressing underspecification in alignment

D'Oosterlinck, K., Xu, W., Develder, C., Demeester, T., Singh, A., Potts, C., Kiela, D., and Mehri, S. Anchored preference optimization and contrastive revisions: Addressing underspecification in alignment. CoRR, 2024

work page 2024

[8] [8]

The llama 3 herd of models

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

arXiv 2024