Pith. sign in

REVIEW 4 minor 2 cited by

Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?

T0 review · 0 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper establishes that Nash Learning from Human Feedback attains the minimax-optimal distortion of $(1/2+o(1))\beta$, while RLHF and DPO are provably worse.

desk verdict A rigorous and novel distortion framework for alignment with heterogeneous preferences; the minimax optimality of NLHF over RLHF/DPO is real within the stated model, and the paper deserves serious peer review. read the letter →

arxiv 2505.23749 v1 pith:NOGWMIK3 submitted 2025-05-29 cs.LG cs.GT

Paul Gölz , Nika Haghtalab , Kunhe Yang This is my paper
classification cs.LGcs.GT MSC 68T0591B1491A1062F07
keywords distortionAIalignmentpreferenceoptimizationBradley-TerrymodelsocialchoicetheoryNashlearningfromhumanfeedbackreinforcementpluralistic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Aligning a model to pairwise comparisons does not automatically maximize the average utility of the users who made those comparisons. This paper introduces a worst-case 'distortion' ratio—the optimal average utility divided by the average utility of the learned policy—and shows that Nash Learning from Human Feedback (NLHF) attains the minimax-optimal distortion of $(1/2+o(1))\beta$ for Bradley–Terry temperature $\beta$. The same analysis shows RLHF and DPO are worse, sometimes exponentially worse, when a KL constraint or correlated comparison sampling is present. If correct, the result gives a principled reason to prefer Nash-style, game-theoretic alignment over single-reward-model fitting in heterogeneous populations.

What carries the argument

The load-bearing object is a linearization lemma for expected Bradley–Terry win rates. For $L=\sigma'(0)=1/4$ and $\ell_\beta=(\sigma(\beta)-1/2)/\beta$, every pair $x,y$ satisfies $$\$\beta$(\ell_\$\beta$ \mathrm{AvgUtil}(x)-L\,\mathrm{AvgUtil}(y)) \le p(x\succ y)-\tfrac12 \le \$\beta$(L\,\mathrm{AvgUtil}(x)-\ell_\$\beta$ \mathrm{AvgUtil}(y)).$$ This sandwiches the nonlinear sigmoid between affine functions of average utilities, converting ordinal comparison data into linear constraints on welfare. For NLHF, the symmetric constrained zero-sum game has value zero, so comparing the NLHF policy against the utility-maximizing benchmark $\pi^\star$ and applying the linearization yields $\mathrm{AvgUtil}(\pi^\star)/\mathrm{AvgUtil}(\pi_{\mathrm{NLHF}})\le L/\ell_\beta$, which is exactly the claimed distortion bound.

What would settle it

Search for a counterexample to the claimed distortion bound: enumerate small populations of Bradley-Terry users with utilities in $[0,1]$, compute the exact expected win rates, solve the NLHF game, and compare the worst-case ratio $\max_\pi \mathrm{AvgUtil}(\pi)/\mathrm{AvgUtil}(\pi_{\mathrm{NLHF}})$ with the bound $\frac{\beta}{2}\frac{1+e^{-\beta}}{1-e^{-\beta}}$; any instance with a larger ratio would refute the theorem. Since the theorem is proven under the single-state and expressive-policy assumptions, a second falsifier is empirical: train a policy with a restricted architecture on a multi-state preference dataset and check whether the observed average-utility ratio exceeds the theorem's bound.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a complete comparison of alignment methods under a new worst-case criterion. For any instance $D$, any number of alternatives $m$, any data distribution $\mu$, any Bradley–Terry temperature $\beta$, any reference policy $\pi_{\mathrm{ref}}$, and any KL budget $\tau$, the distortion of NLHF is at most $$\frac{\$\beta$}{2}\cdot\frac{1+$e^{{-\beta}}$}{1-$e^{{-\beta}}$} = \left(\frac12+o(1)\right)\$\beta$,$$ and Theorem 3 shows no method can guarantee better. RLHF (and DPO, which is equivalent in this model) fails to match the bound: its distortion is at least $(1-o(1))\beta$ without KL constraints, $e^{\Omega(\beta)}$ with KL constraints, and unbounded when comparison pairs are sampled from a correlated distribution rather than i.i.d. The paper reads this as evidence that reward-model fitting optimizes for a 'mythical user' rather than for the population's average utility, whereas the Nash-equilibrium approach hedges against all feasible policies and therefore tracks average utility.

Load-bearing premise

The load-bearing simplification is that alignment is analyzed one state at a time with a policy class expressive enough to realize the optimal distribution at each state; the distortion guarantees need not transfer to deployment where generalization across states or limited policy expressivity matters.

Editorial extensions

If this is right

  • NLHF's distortion guarantee is the best possible among all alignment methods in the model, so no comparison-based method can guarantee a better worst-case ratio of average utility.
  • RLHF and DPO can be worse than NLHF by a constant factor even without KL constraints, exponentially worse with a KL budget, and unbounded when comparison pairs are sampled from a correlated distribution.
  • Because DPO is equivalent to RLHF in this analysis, the negative distortion results for RLHF apply to DPO as well.
  • The equivalence between KL-regularized and KL-constrained NLHF means the optimal distortion guarantee carries over to the regularized objective used in practice.
  • Single-model Bradley-Terry leaderboards inherit Borda's distortion, so such rankings may misreport average usability across a heterogeneous user base.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct consequence of the lower bound is that pure pairwise comparison data has an information bottleneck: average utility cannot be recovered exactly, and the gap grows linearly with the Bradley-Terry temperature; adding even a small amount of cardinal signal, such as a few utility ratings, is a natural testable way to break the bound.
  • The single-state analysis suggests the guarantees are per-state; a multi-state extension that tracks distortion after restricting the policy class or after preference generalization would reveal whether the worst-case gaps compound in deployment.
  • The exponential lower bound for RLHF exploits the mismatch between the comparison-pair distribution and the reference policy; if $\mu=\pi_{\mathrm{ref}}$, the bound does not apply, and characterizing RLHF's distortion under that matching assumption is an open problem that could change the practical verdict.
  • Since the algorithm-independent lower bound holds for any method that respects the Condorcet-loser criterion, the optimal distortion is not unique to NLHF; any rule satisfying that axiom faces the same ceiling, which suggests the framework separates families of alignment methods rather than singling out one algorithm.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 4 minor

Summary. The paper defines a distortion measure for preference-learning alignment methods under heterogeneous Bradley-Terry users, treats the social-choice setting as the unconstrained limit, and compares RLHF/DPO (Borda) with NLHF (Maximal Lotteries). Its central results are: a voting-rule-independent lower bound of (β/2)(1+e^{-β})/(1-e^{-β}); an upper bound of the same value for NLHF, making it minimax optimal; Borda/RLHF distortion between (1-o(1))β and O(β^2) in the unconstrained case, e^{Ω(β)} under KL constraints, and unbounded under correlated pair sampling. The paper also provides finite-sample concentration bounds and an equivalence between regularized and constrained RLHF/NLHF.

Significance. If correct, this is a significant contribution: it gives a principled worst-case metric for pluralistic alignment, shows NLHF is provably minimax optimal while RLHF/DPO are not, and connects alignment to social-choice distortion with non-trivial positive results enabled by Bradley-Terry noise. The proofs are detailed and largely self-contained; the linearization lemma and the lower-bound constructions are carefully checked, and the paper is honest about its scope: the analysis is per-state (Section 2), the d≥2 lower bound relies on a Condorcet-loser axiom, and the finite-sample constants are unspecified. These limitations affect external applicability but not the internal validity of the main theorems. The manuscript also presents parameter-free bounds, with no parameters fitted to data, and clearly identifies open problems such as RLHF distortion under μ=πref.

minor comments (4)
  1. [Figure 3 caption] The caption says that 'NLHF has a worse distortion for every value of β>0', which inverts the comparison made in the text and in the plotted curves; the intended statement is presumably that RLHF/Borda has a worse (larger) distortion guarantee, whereas NLHF/Maximal Lotteries attains the smaller optimal value.
  2. [Section D.2, Theorem 12] The sentence 'which we used to plot Section D.2' should refer to the figure or the displayed formula rather than to the appendix section; as written it is confusing.
  3. [Section E.1, Theorem 7] The finite-sample bound is stated with an O() that hides a threshold on n: the proof chooses δ=Θ(1/√n), while Lemma 10 requires δ≥m^2 exp(-nd μmin^2/8), so the displayed bound is only claimed for sufficiently large n. Please state this threshold dependence explicitly in the theorem statement or proof sketch.
  4. [Section 2, footnote 8] The uniqueness assertion for the MLE requires that every alternative wins at least one comparison against every other alternative, which may fail in finite samples; since the later arguments only need the limiting Borda equivalence, please clarify that finite-sample statements are made conditional on this event or handle non-uniqueness explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity identified; the distortion bounds are derived from the BT model via the sigmoid linearization lemma and zero-sum game value, with no fitted parameter or self-citation serving as premise.

full rationale

The central derivation chain is self-contained. Theorem 7's NLHF upper bound follows from the definition of the NLHF policy as a max-min strategy in a symmetric zero-sum game, the fact that such games have value 0, and Lemma 1, which linearly bounds the sigmoid expected win-rate by constants L=1/4 and ℓβ=(σ(β)-1/2)/β. The utility-maximizing benchmark π* is inserted as an arbitrary feasible policy after the min operator, so its appearance is an instance of the definition of distortion, not a fitted input. No parameter is estimated from a subset and then 'predicted' on a related quantity; the finite-sample bounds use standard concentration separately. The matching lower bound (Theorem 3) is algorithm-independent: it constructs a distribution whose expected win-rates are all 1/2 (for d=1), applies the pigeonhole principle, and for d≥2 explicitly invokes the probabilistic Condorcet loser criterion rather than assuming the conclusion. External citations, such as the Borda–MLE equivalence and non-identifiability of mixtures, are either proved in the appendix or used as motivating context, not as substitutes for the derivation. The KL-constrained/regularized equivalence (Proposition 13) is proved by convex duality and KKT conditions. The paper explicitly states its single-state abstraction and the d≥2 lower-bound caveat; these are scope limitations affecting external applicability, not hidden circular steps.

Assumptions & free parameters 0 free parameters · 8 assumptions · 0 invented entities

The paper introduces no free parameters fitted to data; the model inputs (β, μ, ν, τ, πref) are given. The axioms listed are the key modeling assumptions and standard mathematical tools on which the central claims rest. The single-state assumption and the Bradley-Terry model are the most domain-specific premises. No invented entities (particles, forces, dimensions) appear.

assumptions (8)
  • domain assumption Users' pairwise comparisons follow a Bradley-Terry model: Pr[x≻y] = σ(β(u(x)-u(y))).
    Section 2 assumes this generative model for all preference data; all distortion bounds are relative to this model.
  • domain assumption Utilities are bounded in [0,1] for each user and alternative.
    Section 2 states 0≤u(x)≤1; this is needed to keep distortion finite and to derive the linearization lemma.
  • domain assumption Comparison pairs are sampled i.i.d. from a fixed distribution μ (or ν in extensions), independently across users.
    Section 2 assumes this sampling model; Theorem 9 relaxes it to a distribution over pairs for the RLHF lower bound.
  • domain assumption Alignment is analyzed per single state; the policy class can adopt the optimal action distribution at each state independently.
    Section 2 states 'we focus our analysis on a single state x... abstracts from the generalization problem.' The KL-constrained bounds depend on this abstraction.
  • domain assumption Borda scores order the MLE reward estimates (Borda-MLE equivalence).
    Used in Sections 3.1 and E.2 to reduce RLHF analysis to Borda analysis; cited from Siththaranjan et al. 2023 and Procaccia et al. 2025.
  • standard math Existence of Nash equilibrium in symmetric finite zero-sum games with convex compact strategy sets.
    Used in Theorem 7 proof (Appendix E.1) to set the value of the win-rate game to 0.
  • standard math Slater's condition holds for the KL-constrained optimization, so KKT conditions characterize optima.
    Used in Appendix E.4 to prove equivalence between regularized and constrained NLHF and RLHF.
  • standard math Concentration inequalities (Bernstein, Chernoff) hold for the finite-sample win-rate estimates.
    Used in Appendix C to derive finite-sample distortion guarantees.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?." pith.science (2026). https://pith.science/paper/NOGWMIK3

@misc{pith2026250523749,
  author       = {Pith},
  title        = {Pith review of: Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NOGWMIK3}},
  note         = {Machine review of arXiv:2505.23749}
}
abstract

After pre-training, large language models are aligned with human preferences based on pairwise comparisons. State-of-the-art alignment methods (such as PPO-based RLHF and DPO) are built on the assumption of aligning with a single preference model, despite being deployed in settings where users have diverse preferences. As a result, it is not even clear that these alignment methods produce models that satisfy users on average -- a minimal requirement for pluralistic alignment. Drawing on social choice theory and modeling users' comparisons through individual Bradley-Terry (BT) models, we introduce an alignment method's distortion: the worst-case ratio between the optimal achievable average utility, and the average utility of the learned policy. The notion of distortion helps draw sharp distinctions between alignment methods: Nash Learning from Human Feedback achieves the minimax optimal distortion of $(\frac{1}{2} + o(1)) \cdot \beta$ (for the BT temperature $\beta$), robustly across utility distributions, distributions of comparison pairs, and permissible KL divergences from the reference policy. RLHF and DPO, by contrast, suffer $\geq (1 - o(1)) \cdot \beta$ distortion already without a KL constraint, and $e^{\Omega(\beta)}$ or even unbounded distortion in the full setting, depending on how comparison pairs are sampled.

Figures

Figures reproduced from arXiv: 2505.23749 by the authors.

Figure 1
Figure 1. The typical RLHF pipeline. The preference optimization process begins by collecting comparison data from users with heterogeneous utilities. A single Bradley-Terry model is then fit to this data via Maximum Likelihood Estimation (MLE), producing a single reward model that represents a “mythical user” whose utility best explains the observed comparisons. This reward model is used to fine-tune the pretrained policy. W… view at source ↗
Figure 2
Figure 2. Bounds on probability of preferring x over y, β = 5. For many samples, this estimator concentrates around its mean and allows to select the optimal alternative. The argument would extend if some observed events from a user had a probability that is affine in the user’s utilities. Alas, we are not so lucky: the sigmoid function in the probability of the event “i ranks x over y” is nonlinear, and we show in Theorem 3 … view at source ↗
Figure 3
Figure 3. Comparison of the distortion achieved by NLHF/Maximum Lotteries and the lower bound [PITH_FULL_IMAGE:figures/full_fig_p025_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Utilities for first 14 alternatives in the sequences constructed in Lemma [PITH_FULL_IMAGE:figures/full_fig_p035_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Internal Pluralism and the Limits of Pairwise Comparisons

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Under internal pluralism, forced local pairwise comparisons erase inseparable priorities and distort conflicted answers, while allowing indecision reports can sharply reduce queries needed to learn preference weights.

  2. Power and Limitations of Aggregation in Compound AI Systems

    cs.AI 2026-02 conditional novelty 7.0 of 10

    In a principal-agent model of compound AI, aggregation expands the set of outputs a designer can elicit exactly when one of three mechanisms — feasibility expansion, support expansion, or binding set contraction — hol...

Reference graph

Works this paper leans on

59 extracted references · 36 canonical work pages · cited by 2 Pith papers

  1. [1]

    URL https://incidentdatabase.ai/

    AI Incident Database . URL https://incidentdatabase.ai/

    work page

  2. [2]

    Statistical methods for ranking data, volume 1341

    Mayer Alvo and LH Philip. Statistical methods for ranking data, volume 1341. Springer, 2014

    2014

  3. [3]

    Approximating optimal social choice under metric preferences

    Elliot Anshelevich, Onkar Bhardwaj, Edith Elkind, John Postl, and Piotr Skowron. Approximating optimal social choice under metric preferences. Artificial Intelligence, 264: 0 27--51, 2018

    work page 2018

  4. [4]

    Voudouris

    Elliot Anshelevich, Aris Filos-Ratsikas , Nisarg Shah, and Alexandros A. Voudouris. Distortion in social choice problems: The first 15 years and beyond. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pages 4294--4301, 2021

    work page 2021

  5. [5]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447--4455. PMLR, 2024

    work page 2024

  6. [6]

    A statistical decision-theoretic framework for social choice

    Hossein Azari Soufiani, David C Parkes, and Lirong Xia. A statistical decision-theoretic framework for social choice. Advances in Neural Information Processing Systems, 27, 2014

    work page 2014

  7. [7]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022 a

    arXiv 2022

  8. [8]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022 b

    arXiv 2022

Show all 59 references
  1. [9]

    Procaccia, and Nisarg Shah

    Gerdus Benade, Swaprava Nath, Ariel D. Procaccia, and Nisarg Shah. Preference elicitation for participatory budgeting. Management Science, 67 0 (5): 0 2813--2827, 2021

    2021

  2. [10]

    Procaccia, and Or Sheffet

    Craig Boutilier, Ioannis Caragiannis, Simi Haber, Tyler Lu, Ariel D. Procaccia, and Or Sheffet. Optimal social choice functions: A utilitarian view. In Proceedings of the 13th ACM Conference on Electronic Commerce , pages 197--214. ACM, 2012

    2012

  3. [11]

    Human alignment of large language models through online preference optimisation

    Daniele Calandriello, Zhaohan Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al. Human alignment of large language models through online preference optimisation. In Proceedings of ...

    2024

  4. [12]

    Procaccia

    Ioannis Caragiannis and Ariel D. Procaccia. Voting almost maximizes social welfare despite limited communication. Artificial Intelligence, 175 0 (9-10): 0 1655--1671, 2011

    2011

  5. [13]

    Maxmin- RLHF : Alignment with diverse human preferences

    Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. Maxmin- RLHF : Alignment with diverse human preferences. arXiv preprint arXiv:2402.08925, 2024

    2024 arXiv

  6. [14]

    Breaking the Metric Voting Distortion Barrier

    Moses Charikar, Prasanna Ramakrishnan, Kangning Wang, and Hongxun Wu. Breaking the Metric Voting Distortion Barrier . Journal of the ACM, 71 0 (6): 0 1--33, 2024

    2024

  7. [15]

    Pal: Pluralistic alignment framework for learning from heterogeneous preferences

    Daiwei Chen, Yi Chen, Aniket Rege, and Ramya Korlakai Vinayak. Pal: Pluralistic alignment framework for learning from heterogeneous preferences. arXiv preprint arXiv:2406.08469, 2024

    2024 arXiv

  8. [16]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference...

    2024

  9. [17]

    Direct preference optimization with unobserved preference heterogeneity

    Keertana Chidambaram, Karthik Vinay Seetharaman, and Vasilis Syrgkanis. Direct preference optimization with unobserved preference heterogeneity. arXiv preprint arXiv:2405.15065, 2024

    2024

  10. [18]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  11. [19]

    Common voting rules as maximum likelihood estimators

    Vincent Conitzer and Tuomas Sandholm. Common voting rules as maximum likelihood estimators. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence, pages 145--152, 2005

    2005

  12. [20]

    Position: social choice should guide ai alignment in dealing with diverse human feedback

    Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H Holliday, Bob M Jacobs, Nathan Lambert, Milan Moss \'e , Eric Pacuit, Stuart Russell, Hailey Schoelkopf, et al. Position: social choice should guide ai alignment in dealing with diverse human feedback. In Proceedings o...

    2024

  13. [21]

    Mapping social choice theory to RLHF

    Jessica Dai and Eve Fleisig. Mapping social choice theory to RLHF . In ICLR 2024 Workshop on Reliable and Responsible Foundation Models, 2024

    2024

  14. [22]

    Metric distortion with elicited pairwise comparisons

    Soroush Ebadian, Daniel Halpern, and Evi Micha. Metric distortion with elicited pairwise comparisons. In Proceedings of the 33rd International Joint Conference on Artificial Intelligence ( IJCAI ) , pages 2791--2798, 2024 a

    2024

  15. [23]

    Optimized Distortion and Proportional Fairness in Voting

    Soroush Ebadian, Anson Kahng, Dominik Peters, and Nisarg Shah. Optimized Distortion and Proportional Fairness in Voting . ACM Transactions on Economics and Computation, 12 0 (1): 0 1--39, 2024 b

    2024

  16. [24]

    KTO : Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO : Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024

    2024 arXiv

  17. [25]

    Fishburn

    Peter C. Fishburn. Probabilistic social choice based on simple voting comparisons. The Review of Economic Studies, 51 0 (4): 0 683--692, 1984

    1984

  18. [26]

    Distortion Under Public-Spirited Voting

    Bailey Flanigan, Ariel D Procaccia, and Sven Wang. Distortion Under Public-Spirited Voting . In Proceedings of the 24th ACM Conference on Economics and Computation , EC '23, page 700. Association for Computing Machinery, 2023

    2023

  19. [27]

    Axioms for AI alignment from human feedback

    Luise Ge, Daniel Halpern, Evi Micha, Ariel D Procaccia, Itai Shapira, Yevgeniy Vorobeychik, and Junlin Wu. Axioms for AI alignment from human feedback. arXiv preprint arXiv:2405.14758, 2024

    2024 arXiv

  20. [28]

    Resolving the optimal metric distortion conjecture

    Vasilis Gkatzelis, Daniel Halpern, and Nisarg Shah. Resolving the optimal metric distortion conjecture. In 2020 IEEE 61st Annual Symposium on Foundations of Computer Science ( FOCS ) , pages 1427--1438. IEEE, 2020

    2020

  21. [29]

    Metric distortion under probabilistic voting

    Mohak Goyal and Sahasrajit Sarmasarkar. Metric distortion under probabilistic voting. arXiv preprint arXiv:2405.14223v4, 2025

    2025 arXiv

  22. [30]

    Correcting the mythos of KL -regularization: Direct alignment without overoptimization via chi-squared preference optimization

    Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J Foster. Correcting the mythos of KL -regularization: Direct alignment without overoptimization via chi-squared preference optimization. arXiv preprint arXiv:2407.13399, 2024

    2024 arXiv

  23. [31]

    Plurality Veto : A Simple Voting Rule Achieving Optimal Metric Distortion

    Fatih Erdem Kizilkaya and David Kempe. Plurality Veto : A Simple Voting Rule Achieving Optimal Metric Distortion . In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence , pages 349--355. International Joint Conferences on Artificial Intel...

    2022

  24. [32]

    Provably mitigating overoptimization in RLHF : Your SFT loss is implicitly an adversarial regularizer

    Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in RLHF : Your SFT loss is implicitly an adversarial regularizer. In The Thirty-eighth Annual Conference on Neural Information Process...

    2024

  25. [33]

    Jackpot! alignment as a maximal lottery

    Roberto-Rafael Maura-Rivero, Marc Lanctot, Francesco Visin, and Kate Larson. Jackpot! alignment as a maximal lottery. arXiv preprint arXiv:2501.19266, 2025

    2025 arXiv

  26. [34]

    SimPO : Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO : Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37: 0 124198--124235, 2024

    2024

  27. [35]

    Ai alignment and social choice: Fundamental limitations and policy implications

    Abhilash Mishra. Ai alignment and social choice: Fundamental limitations and policy implications. arXiv preprint arXiv:2310.16048, 2023

    2023 arXiv

  28. [36]

    Nash learning from human feedback

    Remi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, C \^o me Fiegel, et al. Nash learning from human feedback. In International Conference on Machine Learning, pages 36743--367...

    2024

  29. [37]

    Axioms for learning from pairwise comparisons

    Ritesh Noothigattu, Dominik Peters, and Ariel D Procaccia. Axioms for learning from pairwise comparisons. Advances in Neural Information Processing Systems, 33: 0 17745--17754, 2020

    2020

  30. [38]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 0 2...

    2022

  31. [39]

    RLHF from heterogeneous feedback via personalization and preference aggregation

    Chanwoo Park, Mingyang Liu, Dingwen Kong, Kaiqing Zhang, and Asuman Ozdaglar. RLHF from heterogeneous feedback via personalization and preference aggregation. arXiv preprint arXiv:2405.00254, 2024

    2024 arXiv

  32. [40]

    Personalizing reinforcement learning from human feedback with variational preference learning

    Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  33. [41]

    Procaccia and Jeffrey S

    Ariel D. Procaccia and Jeffrey S. Rosenschein. The distortion of cardinal preferences in voting. In International Workshop on Cooperative Information Agents , pages 317--331. Springer, 2006

    2006

  34. [42]

    Clone-robust ai alignment

    Ariel D Procaccia, Benjamin Schiffer, and Shirley Zhang. Clone-robust ai alignment. arXiv preprint arXiv:2501.09254, 2025

    2025 arXiv

  35. [43]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36: 0 53728--53741, 2023

    2023

  36. [44]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  37. [45]

    Direct alignment with heterogeneous preferences

    Ali Shirali, Arash Nasr-Esfahany, Abdullah Alomar, Parsa Mirtaheri, Rediet Abebe, and Ariel Procaccia. Direct alignment with heterogeneous preferences. arXiv preprint arXiv:2502.16320, 2025

    2025 arXiv

  38. [46]

    Distributional preference learning: Understanding and accounting for hidden context in RLHF

    Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell. Distributional preference learning: Understanding and accounting for hidden context in RLHF . arXiv preprint arXiv:2312.08358, 2023

    2023 arXiv

  39. [47]

    Position: a roadmap to pluralistic alignment

    Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. Position: a roadmap to pluralistic alignment. In Proceedings of the 41st International Conference on Machin...

    2024

  40. [48]

    A minimaximalist approach to reinforcement learning from human feedback

    Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.04056, 2024

    2024 arXiv

  41. [49]

    Learning populations of preferences via pairwise comparison queries

    Gokcan Tatli, Yi Chen, and Ramya Korlakai Vinayak. Learning populations of preferences via pairwise comparison queries. In International Conference on Artificial Intelligence and Statistics, pages 1720--1728. PMLR, 2024

    2024

  42. [50]

    Is RLHF more difficult than standard RL ? a theoretical perspective

    Yuanhao Wang, Qinghua Liu, and Chi Jin. Is RLHF more difficult than standard RL ? a theoretical perspective. Advances in Neural Information Processing Systems, 36: 0 76006--76032, 2023

    2023

  43. [51]

    Metric learning from limited pairwise preference comparisons

    Zhi Wang, Geelon So, and Ramya Korlakai Vinayak. Metric learning from limited pairwise preference comparisons. In Uncertainty in Artificial Intelligence, pages 3571--3602. PMLR, 2024

    2024

  44. [52]

    Self-play preference optimization for language model alignment

    Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675, 2024

    2024 arXiv

  45. [53]

    Bayesian estimators as voting rules

    Lirong Xia. Bayesian estimators as voting rules. In Uncertainty in artificial intelligence, 2018

    2018

  46. [54]

    Learning and decision-making from rank data

    Lirong Xia. Learning and decision-making from rank data. Morgan & Claypool Publishers, 2019

    2019

  47. [55]

    On the identifiability of mixtures of ranking models

    Xiaomin Zhang, Xucheng Zhang, Po-Ling Loh, and Yingyu Liang. On the identifiability of mixtures of ranking models. arXiv preprint arXiv:2201.13132, 2022

    2022 arXiv

  48. [56]

    Learning mixtures of plackett-luce models from structured partial orders

    Zhibing Zhao and Lirong Xia. Learning mixtures of plackett-luce models from structured partial orders. Advances in Neural Information Processing Systems, 32, 2019

    2019

  49. [57]

    Learning mixtures of plackett-luce models

    Zhibing Zhao, Peter Piech, and Lirong Xia. Learning mixtures of plackett-luce models. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2...

    2016

  50. [58]

    Provable multi-party reinforcement learning with diverse human feedback

    Huiying Zhong, Zhun Deng, Weijie J Su, Zhiwei Steven Wu, and Linjun Zhang. Provable multi-party reinforcement learning with diverse human feedback. arXiv preprint arXiv:2403.05006, 2024

    2024 arXiv

  51. [59]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.