Pith. sign in

REVIEW 4 major objections 6 minor 47 references

On Monotonicity in AI Alignment

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that comparison-based preference-learning methods, including DPO and generalized Bradley-Terry models, are locally pairwise monotone: an infinitesimal preference for a response can never shrink its score gap over the…

desk verdict Correct general theorem and useful taxonomy, but the flagship DPO guarantee is vacuous under the softmax shift symmetry the paper doesn't flag. read the letter →

arxiv 2506.08998 v1 pith:ODORORNX submitted 2025-06-10 math.ST cs.LGstat.MLstat.TH

Gilles Bareilles , Julien Fageot , Lê-Nguyên Hoang , Peva Blanchard , Wassim Bouaziz , Sébastien Rouault , El-Mahdi El-Mhamdi This is my paper
classification math.STcs.LGstat.MLstat.TH MSC 62J1562F07
keywords preferencelearningAIalignmentmonotonicityDirectOptimizationGeneralizedBradley-Terrycomparison-basedlossimplicitfunctiontheoremscoredifference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Comparison-based alignment fine-tunes models so that preferred responses score higher than rejected ones, yet training can visibly lower the model's score for the preferred response. This paper asks what monotonicity guarantee such methods actually provide. For a single loss form that subsumes Direct Preference Optimization (DPO), Generalized Preference Optimization (GPO), and Generalized Bradley-Terry (GBT) models, it proves that at any strict local minimum with a positive-definite Hessian, adding an infinitesimal comparison (or intensifying an existing one) never shrinks the score gap between chosen and rejected responses. It then maps which stronger forms of monotonicity, such as global, individual-score, individual-probability, and gradient-descent monotonicity, follow and which require unrealistic conditions. The result clarifies what alignment methods guarantee and where observed likelihood-displacement failures fall outside the theory.

What carries the argument

The central object is the comparison-based loss $\mathrm{LOSS}(\theta|D)=R(\theta)+\sum_{(x,y,z,c)\in D}\ell(s_{yz|x}(\theta),c)$, where $s_{yz|x}(\theta)=s_{y|x}(\theta)-s_{z|x}(\theta)$ is the score difference between the preferred and rejected responses. The proof machinery is the implicit function theorem applied to the first-order condition $\nabla_\theta\mathrm{LOSS}=0$: a small weight $\varepsilon$ on a new comparison shifts the local minimum to first order along $g'(0)=-\alpha\,[\nabla^2\mathrm{LOSS}(\theta^*|D)]^{-1}\nabla_\theta s_{yz|x}(\theta^*)$, with $\alpha=-\partial_s\ell(s_{yz|x}(\theta^*),\max C)>0$. The resulting identity $s_{yz|x}(\theta_\varepsilon)-s_{yz|x}(\theta^*)=\varepsilon\,\alpha\,\nabla s_{yz|x}(\theta^*)^\top[\nabla^2\mathrm{LOSS}(\theta^*|D)]^{-1}\nabla s_{yz|x}(\theta^*)+o(\varepsilon)$ is positive because the Hessian is positive definite, which carries all local pairwise monotonicity results. For individual-score monotonicity the same calculation is forced through a max-diagonal-dominance condition on $\nabla_\theta s_{|x}^\top[\nabla^2\mathrm{LOSS}]^{-1}\nabla_\theta s_{|x}$.

What would settle it

In a one-dimensional linear model with the DPO loss $\ell(s,1)=-\log\sigma(s)$ and no regularization, compute the exact minimizer $\theta^*$ for a dataset containing one comparison, then add a second identical comparison with infinitesimal weight $\varepsilon$; the theorem predicts the new minimizer satisfies $s(\theta_\varepsilon)>s(\theta^*)$. A single case where the inequality reverses for arbitrarily small $\varepsilon$ would refute Theorem 1. The same experiment with IPO's squared loss, which violates Assumption 2, should exhibit the failure the theorem excludes.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that the counterintuitive observation that training on a stated preference for y over z can lower the model's score for y does not contradict a weaker guarantee that does hold. For the loss (1) covering DPO, GPO, and GBT, the paper proves (Theorem 1) that under Assumptions 1 and 2, at a local minimum $\theta^*$ with positive-definite Hessian and $\nabla s_{zy|x}(\theta^*)\neq 0$, infinitesimally adding the unequivocal comparison $(x,y,z,\max C)$ yields local minima whose score difference $s_{yz|x}$ is at least as large as at $\theta^*$. The analogue for comparison intensification follows from Assumption 3 (Theorem 2), and under strong convexity the guarantee becomes global (Theorem 3). The paper further shows that generalized Bradley-Terry models are fully pairwise monotone, hence individual-probability monotone for softmax outputs, while individual-score monotonicity needs max-diagonal dominance, which is implausible for large response spaces.

Load-bearing premise

The guarantee collapses if the model is not at a strict local minimum with positive-definite Hessian, or if the loss's derivative with respect to the score gap at the maximal comparison is not negative; the paper itself notes that IPO and SLiC losses violate the latter condition.

Editorial extensions

If this is right

  • Any alignment method covered by loss (1) under Assumptions 1-2, including DPO and GBT with a maximal comparison, will not decrease the score difference between the preferred and rejected responses after an infinitesimal comparison addition or intensification at a strict local minimum.
  • Under Assumption 4, which combines convex per-comparison losses with strongly convex regularization, the local guarantee upgrades to a global one: any dataset change favoring y over z increases the score gap at the unique minimizer.
  • Individual-score monotonicity, meaning the preferred response's own score rising, is not guaranteed and requires max-diagonal dominance of a matrix akin to a Gram matrix, so reported drops in the chosen response's score are consistent with the theory.
  • Generalized Bradley-Terry models with scores as free parameters are fully pairwise monotone, which for softmax probabilities implies that the preferred response's probability rises.
  • For plain gradient descent with no regularization, pairwise gradient-descent monotonicity holds whenever the score-difference gradient is nonzero, and stronger inner-product conditions give individual-score and fully-pairwise gradient-descent monotonicity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct practical diagnostic follows: when an alignment run appears anti-monotone, first check whether the loss satisfies the sign condition $\partial_s\ell(s,\max C)<0$; the paper's own IPO example shows a popular loss can fail it and thus sit outside the theorem.
  • Because the guaranteed quantity is the score gap rather than individual scores, re-analyzing existing DPO training logs through score-difference trajectories should show monotone gaps even where individual probabilities fall; this is a testable reinterpretation of published failure modes.
  • The max-diagonal-dominance condition suggests individual-score monotonicity becomes effectively impossible as the response space grows, so one could test whether violations scale with vocabulary size or sampling temperature.
  • The global convexity result points to a possible nonconvex extension: replacing strong convexity with a Polyak-Lojasiewicz condition or layer-wise Hessian bounds might carry global pairwise monotonicity to deep networks, where positive-definite Hessians are rare.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a general comparison-based preference learning framework of the form LOSS(theta|D) = R(theta) + sum_c ell(s_{yz|x}(theta), c), which subsumes DPO, GPO, and Generalized Bradley-Terry models. It formalizes several notions of monotonicity (pairwise vs. individual, local vs. global, score vs. probability, minimum vs. gradient-descent) and studies which of these are guaranteed. The main positive result (Theorem 1) is that, under a strict local minimum with positive definite Hessian and a nonzero gradient of the score difference, adding an infinitesimal unequivocal comparison (x,y,z,maxC) locally increases s_{yz|x}. Theorem 2 proves an analogous result for comparison intensification under a mixed-partial sign condition. The paper also reports an empirical observation that during DPO fine-tuning the chosen response's score can decrease, motivating the theoretical analysis.

Significance. If the conditional theorems are taken at face value, this is a useful systematization of monotonicity guarantees in preference learning: it provides a common umbrella for DPO, GPO, and GBT, and it clearly separates pairwise monotonicity (which can be guaranteed locally) from individual-score and individual-probability monotonicity (which are shown to be much more demanding). The main proof via the implicit function theorem is standard and correct under its assumptions, and the paper's negative results on individual-score monotonicity, based on max-diagonal dominance, are informative. However, the flagship DPO application is undermined by a softmax gauge-invariance issue that makes the positive-definite-Hessian premise vacuous for standard DPO with no regularization, and Corollary 1 contains a sign error (minC instead of maxC). These issues affect the scope of the paper's central claim, but they are fixable within the manuscript's framework.

major comments (4)
  1. [Section 4.2, Corollary 1] Corollary 1 states monotonicity for the addition of (x,y,z,minC) with the gradient condition written as nabla s_{zy|x}(theta*) != 0. For binary comparisons C={-1,1}, minC=-1 corresponds to a preference for z over y, and for the DPO/BT loss one has d_s ell(s,minC) > 0, so the conclusion is false as stated. The correct statement should use maxC (the unequivocal comparison in favor of y) and the gradient of s_{yz|x}, matching Theorem 1 and Assumption 2. Please correct this statement and the associated proof references.
  2. [Section 3.1 and Section 4.2 (Theorem 1/Corollary 1)] For a softmax policy (Eq. (2)) with a bias vector or an output-embedding direction that adds a constant to all logits, pi_theta(y|x) is unchanged. In the DPO scoring (Eq. (3)), s_{y|x}(theta) is then unchanged, so with R=0 the entire loss LOSS(theta|D) is constant along that direction. Hence nabla^2 LOSS(theta*|D) has a zero eigenvalue at every parameter point, and the premise of Theorem 1 and Corollary 1 (positive-definite Hessian) is unsatisfiable for this flagship instance. The Limitations paragraph does not flag this structural obstruction. Please state the theorem modulo the logit-shift gauge, e.g., positive definiteness on the quotient of R^D by the symmetry directions, or restrict the DPO claim to parameterizations where the symmetry is absent.
  3. [Appendix B, proof of Theorem 2] The proof only analyzes the case where the perturbed comparison appears in D as (x,y,z) with c replaced by c+epsilon. When D contains the reversed comparison (x,z,y,c'), the push in Eq. (13) replaces c' by proj_C(c'-epsilon), and the first-order term in epsilon involves d_c ell(s_{zy|x}, c') rather than d_c d_s ell(s_{yz|x}, c); the display after Eq. (34) does not cover this case. Since Definition 2 applies to both orientations, the proof is incomplete as written. Please complete the sign analysis for reversed pairs or add an assumption restricting the dataset.
  4. [Section 5.2, Theorem 4 and Appendix D] For the decrease of s_{z|x}(theta_epsilon), the first-order coefficient is M_{zz} - M_{zy}, where M = nabla_theta s_{|x}(theta*)^T (nabla^2 LOSS(theta*|D))^{-1} nabla_theta s_{|x}(theta*). Max-diagonal dominance as defined (M_{ii} >= max_{j != i} M_{ij}) gives M_{zz} - M_{zy} >= 0, not the strict negativity needed for s_z to decrease; the proof's statement 'we deduce beta > 0' is therefore invalid as written. There is also a typo in the last displayed equation (s_zy where s_z is meant). Please state a strict version of the dominance condition or supply a correct argument.
minor comments (6)
  1. [Section 4.2 and 4.3 headings] The headings 'Pairwise monotonocity' should be 'Pairwise monotonicity'; this typo appears in multiple places.
  2. [Appendix B heading] The heading reads 'Proofs of pairwise monotonicity for unequivocal comparisons', but the appendix proves Theorem 2 for comparison intensification; the heading should be changed accordingly.
  3. [Figure 1 and Section 2 experiments] The figure reports score differences without error bars or confidence intervals, and the text notes the base-vs-instruct observation is 'far from robust'; please add uncertainty quantification or temper the wording.
  4. [Appendix A, last paragraph of proof of Theorem 1] The concluding sentence says the score difference between z and y increases for a comparison intensification in favor of z; it should say between y and z in favor of y.
  5. [Theorem 5] The fully-pairwise gradient-descent condition is stated as 'forall w in A - {z}'; Definition 7 requires all w != y, and the condition as stated is never satisfied for w=y. It should be 'forall w in A - {y}' (or 'forall w in A - {y,z}' if pairwise monotonicity is treated separately).
  6. [Section 1, outline] The outline says 'Section 2 reviews related work. Section 2 motivates...'; the second clause should refer to the experimental material in Section 2 or to a later section.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central monotonicity proof is a direct implicit-function theorem calculation and does not assume its conclusion.

full rationale

The central derivation chain is self-contained and non-circular. Theorem 1 (Appendix A) starts from the optimality condition ∇LOSS(θ*|D)=0 and applies the implicit function theorem to the perturbed loss LOSS(θ|Dε)=LOSS(θ|D)+εℓ(syz|x(θ),maxC). The first-order expansion syz|x(θε)-syz|x(θ*)=εα∇s^T H^{-1}∇s+o(ε), with α=-∂sℓ(syz|x(θ*),maxC)>0, follows directly from Assumption 2, and positivity uses only H≻0 and ∇s≠0; the conclusion is not used as an input. Assumption 2 is an explicit hypothesis, not a disguised form of the target conclusion. The paper's Proposition 1 verifies Assumption 2 for GBT via the cited cumulant-derivative property [12, Theorem 1], but even though [12] has overlapping authorship, it is a published, externally checkable mathematical lemma used only to instantiate a sufficient condition; Theorem 1's conditional statement remains true regardless of that lemma. No parameter is fitted and then called a prediction, no uniqueness claim is imported to forbid alternatives, and no known result is merely renamed. The vacuousness concern for Corollary 1 under softmax logit-shift symmetries (the positive-definite Hessian premise may fail for softmax policies with bias directions) is a soundness or scoping issue, not a circularity; the paper's Limitations candidly states that its theory does not capture the gradient-descent score changes shown in Figure 1. Thus no circular reduction is exhibited.

Assumptions & free parameters 0 free parameters · 8 assumptions · 0 invented entities

The theoretical claims are conditional on smoothness, sign conditions, and convexity assumptions; the paper introduces no fitted parameters. The only notable external dependency is the GBT cumulant property from [12], a published paper by two of the present authors, used as a lemma rather than as the target result. This keeps the circularity burden low but not zero.

assumptions (8)
  • domain assumption Assumption 1: each score function s_y|x(θ) is continuously differentiable in θ.
    Invoked for the implicit function theorem in Theorems 1 and 2 and for gradient computations in Theorem 5.
  • domain assumption Assumption 2: C has a maximum and ∂_s ℓ(s, maxC) < 0 for all s.
    This is the sign condition that makes the added comparison push the score gap upward; it holds for BT and GBT, but not for IPO and SLiC variants.
  • domain assumption Assumption 3: C is an interval, loss and regularization are C^2, and ∂_c∂_s ℓ(s,c) < 0.
    Needed for local pairwise monotonicity under comparison intensification (Theorem 2) and for individual-score monotonicity (Theorem 4).
  • domain assumption Assumption 4: the loss is convex in θ for each comparison and the regularization is strongly convex.
    Used in Theorem 3 to globalize the local monotonicity statements; typically holds for convex losses with linear score models.
  • standard math Strict local minimum with positive definite Hessian at θ*.
    Required for the implicit function theorem; restricts the result to non-degenerate minima, which are not guaranteed for neural network losses.
  • domain assumption Probabilities are softmax functions of scores (Eq. 2).
    Used to translate fully-pairwise score monotonicity into individual-probability monotonicity (Proposition 3).
  • standard math GBT cumulant derivative property from [12, Theorem 1]: Φ'_f is a strictly increasing odd bijection from R to (min C, max C).
    Used in Proposition 1 to guarantee Assumption 2 for GBT losses; this is a published prior result by two of the present authors.
  • domain assumption Max-diagonal dominance of the matrix ∇θs|x^T (∇²LOSS)^{-1} ∇θs|x (Theorem 4).
    A strong sufficient condition for local individual-score monotonicity; the authors note it is unlikely to hold for large output spaces.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Monotonicity in AI Alignment." pith.science (2026). https://pith.science/paper/ODORORNX

@misc{pith2026250608998,
  author       = {Pith},
  title        = {Pith review of: On Monotonicity in AI Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ODORORNX}},
  note         = {Machine review of arXiv:2506.08998}
}
abstract

Comparison-based preference learning has become central to the alignment of AI models with human preferences. However, these methods may behave counterintuitively. After empirically observing that, when accounting for a preference for response $y$ over $z$, the model may actually decrease the probability (and reward) of generating $y$ (an observation also made by others), this paper investigates the root causes of (non) monotonicity, for a general comparison-based preference learning framework that subsumes Direct Preference Optimization (DPO), Generalized Preference Optimization (GPO) and Generalized Bradley-Terry (GBT). Under mild assumptions, we prove that such methods still satisfy what we call local pairwise monotonicity. We also provide a bouquet of formalizations of monotonicity, and identify sufficient conditions for their guarantee, thereby providing a toolbox to evaluate how prone learning models are to monotonicity violations. These results clarify the limitations of current methods and provide guidance for developing more trustworthy preference learning algorithms.

Figures

Figures reproduced from arXiv: 2506.08998 by the authors.

Figure 1
Figure 1. For each model Llama, at each step, we report the difference of scores, before and after [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 36 canonical work pages

  1. [1]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024

    2024

  2. [2]

    Generalizing while preserving monotonicity in comparison-based preference learning models

    Anonymous. Generalizing while preserving monotonicity in comparison-based preference learning models. Technical report, Under submission, 2025

    work page 2025

  3. [3]

    The necessity of cardinal human feedback for language model alignment

    Anonymous. The necessity of cardinal human feedback for language model alignment. Technical report, Unknown, 2025

    work page 2025

  4. [4]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Rémi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, International Conference on Artificial Intelligence and Statistics, 2-4 May 2024, Palau de Con...

    work page 2024

  5. [5]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika, 39(3/4):324–345, 1952

    1952

  6. [6]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  7. [7]

    Zhang, Xinyi Chen, Qiuyi (Richard) Zhang, Rajesh Ranganath, and Kyunghyun Cho

    Angelica Chen, Sadhika Malladi, Lily H. Zhang, Xinyi Chen, Qiuyi (Richard) Zhang, Rajesh Ranganath, and Kyunghyun Cho. Preference learning algorithms do not learn preference rankings. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors,Advances in Neural Information Processing Systems...

    work page 2024

  8. [8]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Garnett, editors,Advances in Neural Information Processing Systems 30: Annual Conference on Neural ...

    2017

Show all 47 references
  1. [9]

    Em algorithms for generalized bradley-terry models

    Villo Csiszár. Em algorithms for generalized bradley-terry models. InAnnales Universitatis Scientiarum Budapestinensis de Rolando Eötvös Nominatae (Sectio Computatorica), volume 36, pages 143–157, 2012

    2012

  2. [10]

    Ultrafeedback: Boosting language models with scaled ai feedback, 2024

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with scaled ai feedback, 2024. 10

    2024

  3. [11]

    Safesora: Towards safety alignment of text2video generation via a human preference dataset

    Juntao Dai, Tianle Chen, Xuyao Wang, Ziran Yang, Taiye Chen, Jiaming Ji, and Yaodong Yang. Safesora: Towards safety alignment of text2video generation via a human preference dataset. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomc...

    2024

  4. [12]

    Generalized Bradley-Terry Models for Score Estimation from Paired Comparisons.Proceedings of the AAAI Conference on Artificial Intelligence, 38(18):20379–20386, March 2024

    Julien Fageot, Sadegh Farhadkhani, Lê-Nguyên Hoang, and Oscar Villemaud. Generalized Bradley-Terry Models for Score Estimation from Paired Comparisons.Proceedings of the AAAI Conference on Artificial Intelligence, 38(18):20379–20386, March 2024

    2024

  5. [13]

    Peter Campbell, Michael F

    Yuan Guo, Peng Tian, Jayashree Kalpathy-Cramer, Susan Ostmo, J. Peter Campbell, Michael F. Chiang, Deniz Erdogmus, Jennifer G. Dy, and Stratis Ioannidis. Experimental design under the bradley-terry model. In Jérôme Lang, editor,Proceedings of the Twenty-Seventh International J...

    2018

  6. [14]

    Incomplete contracting and ai alignment

    Dylan Hadfield-Menell and Gillian K Hadfield. Incomplete contracting and ai alignment. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 417–422, 2019

    2019

  7. [15]

    Meta faces ghana lawsuits over impact of extreme content on moderators.The Guardian, 2025

    Rachel Hall and Claire Wilmot. Meta faces ghana lawsuits over impact of extreme content on moderators.The Guardian, 2025

    2025

  8. [16]

    Cleaning up chatgpt takes heavy toll on human workers

    Karen Hao and Deepa Seetharaman. Cleaning up chatgpt takes heavy toll on human workers. Wall Street Journal, 24, 2023

    2023

  9. [17]

    Bradley Knox, and Dorsa Sadigh

    Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without RL.CoRR, abs/2310.13639, 2023

    2023 arXiv

  10. [18]

    Towards robust end-to-end alignment

    Lê Nguyên Hoang. Towards robust end-to-end alignment. In Huáscar Espinoza, Seán Ó hÉigeartaigh, Xiaowei Huang, José Hernández-Orallo, and Mauricio Castillo-Effen, editors, Workshop on Artificial Intelligence Safety 2019 co-located with the Thirty-Third AAAI Confer- ence on Art...

    2019

  11. [19]

    Tournesol: A quest for a large, secure and trustworthy database of reliable human judgments.arXiv preprint arXiv:2107.07334, 2021

    Lê-Nguyên Hoang, Louis Faucon, Aidan Jungo, Sergei V olodin, Dalia Papuc, Orfeas Liossatos, Ben Crulis, Mariame Tighanimine, Isabela Constantin, Anastasiia Kucherenko, et al. Tournesol: A quest for a large, secure and trustworthy database of reliable human judgments.arXiv prep...

    2021 arXiv

  12. [20]

    Africa’s content moderators want compensation for job trauma.Deutsche Welle, 2025

    Stephanie Höppner. Africa’s content moderators want compensation for job trauma.Deutsche Welle, 2025

    2025

  13. [21]

    A user study of perceived carbon footprint.CoRR, abs/1911.11658, 2019

    Victor Kristof, Valentin Quelquejay-Leclère, Robin Zbinden, Lucas Maystre, Matthias Gross- glauser, and Patrick Thiran. A user study of perceived carbon footprint.CoRR, abs/1911.11658, 2019

    1911 arXiv

  14. [22]

    L, Arun P

    Adarsh N. L, Arun P. V ., and Aravindh N. L. Enhancing image caption generation using reinforcement learning with human feedback.CoRR, abs/2403.06735, 2024

    2024 arXiv

  15. [23]

    Procaccia

    Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, and Ariel D. Procaccia. Webuildai: Participatory framework for algorithmic governance.Proc. ACM Hum. Comput. Interact., 3(CSCW):181:...

    2019

  16. [24]

    Collins, Yiwen Luo, Yang Li, Kai J

    Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, Junjie Ke, Krishnamurthy Dj Dvijotham, Katherine M. Collins, Yiwen Luo, Yang Li, Kai J. Kohlhoff, Deepak Ramachandran, and Vidhya Navalpakka...

    2024

  17. [25]

    Prefpaint: Aligning image inpainting diffusion model with human preference

    Kendong Liu, Zhiyu Zhu, Chuanhao Li, Hui Liu, Huanqiang Zeng, and Junhui Hou. Prefpaint: Aligning image inpainting diffusion model with human preference. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors...

    2024

  18. [26]

    Blanchet, and Zhaoran Wang

    Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose H. Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in RLHF: your SFT loss is implicitly an adversarial regularizer. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, ...

    2024

  19. [27]

    Chan, Jakob N

    Chris Lu, Samuel Holt, Claudio Fanconi, Alex J. Chan, Jakob N. Foerster, Mihaela van der Schaar, and Robert T. Lange. Discovering preference optimization algorithms with and for large language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paq...

    2024

  20. [28]

    Wiley New York, 1959

    R Duncan Luce et al.Individual choice behavior, volume 4. Wiley New York, 1959

    1959

  21. [29]

    Simpo: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors,Advances in Neural Information Processing Syst...

    2024

  22. [30]

    Menke and Tony R

    Joshua E. Menke and Tony R. Martinez. A bradley-terry artificial neural network model for individual ratings in group competitions.Neural Comput. Appl., 17(2):175–186, 2008

    2008

  23. [31]

    Gaikwad, Edmond Awad, Sohan Dsouza, Iyad Rahwan, Pradeep Ravikumar, and Ariel D

    Ritesh Noothigattu, Snehalkumar (Neil) S. Gaikwad, Edmond Awad, Sohan Dsouza, Iyad Rahwan, Pradeep Ravikumar, and Ariel D. Procaccia. A voting-based system for ethical decision making. In Sheila A. McIlraith and Kilian Q. Weinberger, editors,Proceedings of the Thirty-Second AA...

    2018

  24. [32]

    Smaug: Fixing failure modes of preference optimisation with dpo-positive.CoRR, abs/2402.13228, 2024

    Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive.CoRR, abs/2402.13228, 2024

    2024 arXiv

  25. [33]

    Iterative reasoning preference optimization

    Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances i...

    2024

  26. [34]

    Openai used kenyan workers on less than $2 per hour to make chatgpt less toxic

    Billy Perrigo. Openai used kenyan workers on less than $2 per hour to make chatgpt less toxic. Time Magazine, 18:2023, 2023

    2023

  27. [35]

    The analysis of permutations.Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193–202, 1975

    Robin L Plackett. The analysis of permutations.Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193–202, 1975

    1975

  28. [36]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Ne...

    2023

  29. [37]

    Unintentional unalignment: Likelihood displacement in direct preference optimization.CoRR, abs/2410.08847, 2024

    Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization.CoRR, abs/2410.08847, 2024

    2024 arXiv

  30. [38]

    Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin...

    2020

  31. [39]

    Generalized preference optimization: A unified approach to offline alignment

    Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. InForty-first International C...

    2024

  32. [40]

    A law of comparative judgment.Psychological Review, 34(4):273–286, 1927

    Louis Leon Thurstone. A law of comparative judgment.Psychological Review, 34(4):273–286, 1927

    1927

  33. [41]

    torchtune: Pytorch’s finetuning library, April 2024

    torchtune maintainers and contributors. torchtune: Pytorch’s finetuning library, April 2024

    2024

  34. [42]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  35. [43]

    Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G. Honavar. Cal-dpo: Calibrated direct preference optimization for language model alignment. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors...

    2024

  36. [44]

    Die berechnung der turnier-ergebnisse als ein maximumproblem der wahrschein- lichkeitsrechnung.Mathematische Zeitschrift, 29(1):436–460, 1929

    Ernst Zermelo. Die berechnung der turnier-ergebnisse als ein maximumproblem der wahrschein- lichkeitsrechnung.Mathematische Zeitschrift, 29(1):436–460, 1929

    1929

  37. [45]

    Speechalign: Aligning speech generation to human preferences

    Dong Zhang, Zhaowei Li, Shimin Li, Xin Zhang, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechalign: Aligning speech generation to human preferences. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors,Adv...

    2024

  38. [46]

    Deep learning-based learning to rank with ties for image re-ranking

    Piplong Zhao, Ou Wu, Liyuan Guo, Weiming Hu, and Jinfeng Yang. Deep learning-based learning to rank with ties for image re-ranking. In2016 IEEE International Conference on Digital Signal Processing (DSP), pages 452–456. IEEE, 2016

    2016

  39. [47]

    Slic- hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic- hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023. 13 Supplemental material A Proofs of pairwise monotonicity for unequivocal comparisons Proo...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.