Pith. sign in

REVIEW 5 major objections 5 minor 57 references

Beyond the Binary: Capturing Diverse Preferences With Reward Regularization

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reward models trained on one annotator's binary choice systematically miss cases where users would disagree, and a margin term built from synthetic preference judgments recovers much of that lost alignment.

desk verdict A promising margin-regularization idea and a useful taxonomy, but the main experiment compares a retrained model against a released checkpoint, so the headline improvement is confounded. read the letter →

arxiv 2412.03822 v1 pith:EFRLLTVK submitted 2024-12-05 cs.CL cs.AI

Vishakh Padmakumar , Chuanyang Jin , Hannah Rose Kirk , He He This is my paper · ORCID
classification cs.CLcs.AI
keywords rewardmodelingpreferencetuningsubjectivityannotatordisagreementmarginregularizationsyntheticannotationsRLHFpluralisticalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that standard reward modeling, which learns from a single annotator's binary pick between two outputs, is poorly calibrated for real deployment where users hold diverse preferences. It classifies subjectivity along two axes: prompts that admit multiple correct answers, and response pairs that are paraphrase-level indistinguishable. On a 150-example test set, the authors show that a DeBERTa-v3 reward model correlates weakly with aggregate user preferences on these subjective cases, especially out of domain. They then add a margin term to the training loss, computed from ten synthetic preference judgments per example generated by an LLM, that shrinks when synthetic annotators disagree. The regularized model improves correlation with aggregate preferences on multiple-correct and out-of-domain subsets without degrading single-correct performance.

What carries the argument

The key object is the margin term $m_{x,y_0,y_1} \in [0,1]$ inserted inside the sigmoid of the reward-model loss. It is computed by sampling $n$ synthetic binary judgments $j'_i$ from an LLM for the same prompt-response pair and taking $\left|\sum_i j'_i - n/2\right| / (n/2)$: the margin is one when all synthetic annotators agree and zero when they split evenly. Inserting it as $\log\sigma(r_\theta(x,y_j) - r_\theta(x,y_{1-j}) - m)$ forces the model to assign a large score gap only when the (synthetic) population is unanimous, and to keep predictions near the decision boundary when preference is contested. The taxonomy of subjectivity, plurality of correct answers and indistinguishability of responses, is the accompanying device used to locate where the baseline fails and where the regularization helps.

What would settle it

On the same 150 examples, collect true preference votes from a large panel of independent annotators (not the 10-people projection protocol), compute real disagreement margins, and correlate them with Llama-3's synthetic margins. Low or zero correlation would falsify the claim that synthetic judgments estimate aggregate user disagreement.

Watch

Extended reading notes

Core claim

The central claim is that the binary preference signal used to train reward models systematically misrepresents aggregate human preferences wherever legitimate disagreement exists, and that a cheap, plug-in regularization can partially correct this. The authors define an example as subjective when the prompt admits multiple correct answers or the two candidate responses are near-paraphrases, and they measure a reward model's predictions against "If you asked 10 people" estimates of population preference. Baseline reward-model predictions correlate much more weakly with these aggregate preferences on multiple-correct (Pearson 0.572) and out-of-domain (0.377) examples than on single-correct ones (0.802). Incorporating a disagreement-scaled margin term into the logistic preference loss, built from synthetic judgments sampled from Llama-3 70B, raises multiple-correct correlation to 0.626 and out-of-domain correlation to 0.488, while lowering L1 error and leaving single-correct performance unchanged. The implication the paper presses is that disagreement should be treated as information about the preference distribution, not as label noise to be averaged away.

Load-bearing premise

The load-bearing premise is that an LLM's synthetic preference disagreements, and the "If you asked 10 people" estimates used to evaluate them, both reflect how real users actually disagree; if either is wrong, the margin regularizes toward the wrong target.

Editorial extensions

If this is right

  • Existing binary preference datasets can be upgraded without re-annotation: sample synthetic judgments once, store a per-example margin, and retrain the reward model with the modified loss.
  • Regularized reward models improve aggregate-preference correlation on multiple-correct-answer prompts (Pearson 0.572 to 0.626) and on out-of-domain data (0.377 to 0.488), with L1 error down across in-domain, out-of-domain, and indistinguishable-response subsets.
  • Single-correct-answer and distinguishable-response performance is essentially unchanged, so the intervention targets the subjectivity failure mode rather than trading off overall quality.
  • The framework can accommodate aggregation rules other than the utilitarian majority vote used in the experiments, since the margin is a design choice rather than a fixed quantity.
  • For developers, the method offers a stopgap that approximates multi-annotator preference data without the cost of recollecting human judgments at scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if synthetic disagreement tracks human disagreement, the same margin trick should transfer to direct preference-optimization losses (e.g., DPO-style objectives), not only to learned reward models; the paper only tests DeBERTa-v3.
  • Editorial inference: the "If you asked 10 people" annotation protocol could itself become a cheap, validated way to collect preference distributions at scale; a calibration study comparing it to true independent votes would settle this.
  • Editorial inference: the taxonomy suggests a data-selection heuristic, deliberately oversampling subjective examples during reward training might improve calibration on exactly the inputs where deployed models currently appear arbitrary.
  • Editorial inference: a risk not resolved here is that LLM-generated disagreement may reflect the model's own priors rather than user diversity; the paper flags this concern in its social impacts statement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper argues that reward models trained on binary preference judgments misrepresent aggregate user preferences for subjective items. It proposes a two-dimensional taxonomy of subjectivity (plurality of correct answers and indistinguishability of candidate responses), analyzes reward-model performance on 150 examples spanning in- and out-of-domain datasets, and introduces a margin-based regularization term estimated from synthetic pairwise judgments generated by Llama-3 70B. The regularized model is reported to improve Pearson correlation and L1 agreement with human '10 people' population-preference estimates, especially on multiple-correct-answer and out-of-domain examples.

Significance. If the reported improvement is causal and replicable, the margin-regularization method is a cheap, drop-in modification to standard reward modeling and the taxonomy offers a useful decomposition of subjectivity in preference data. The authors are appropriately cautious in the Social Impacts Statement about the limits of synthetic annotations. The evaluation is conducted against external human aggregate-preference estimates, which mitigates circularity. However, the evidence base is thin (150 examples, per-cell N as low as 10, no significance tests) and the main comparison is confounded, so the central claim is currently under-supported.

major comments (5)
  1. [§3.3, Tables 1-2] The reported Δ gains do not isolate the margin term: the baseline is the released OpenAssistant Deberta-V3 checkpoint (footnote 3), while the regularized model is retrained from the same pretrained weights with a learning-rate sweep (footnote 6), so differences in data ordering, optimization hyperparameters, and checkpoint selection are confounded with the intervention; a matched no-margin baseline trained under identical conditions is required to support the causal claim.
  2. [§3.2] The test set construction does not state that the 25 examples per dataset were excluded from the training data used for the four in-distribution datasets, so the ID correlations in Tables 1-2 may include memorization effects; the paper should explicitly describe the train/test split and, if necessary, use a held-out subset.
  3. [§3.2, Tables 1-2] The study has insufficient statistical power for its claims: 150 test examples with per-cell N as low as 10 (Single Correct OOD), no confidence intervals or significance tests, and several reported gains (e.g., Δ=0.015 for All ID, Δ=0.029 for UltraFeedback, Δ=0.021 for Indistinguishable) are within the range of plausible sampling noise; the authors should report bootstrap intervals or per-item significance tests.
  4. [Abstract, §3.2, Tables 1-2] The claim that reward models correlate weakly with user preferences along both taxonomy dimensions is contradicted by the baseline results for the Indistinguishable axis: Table 1 shows Pearson correlation 0.640 for Distinguishable vs 0.631 for Indistinguishable, and Table 2 shows L1 0.244 vs 0.176, i.e., the baseline is not worse on indistinguishable pairs; the 'break it' phase only supports the Multiple Correct Answers dimension, and the paper should either reframe the claim or add stronger evidence for the second axis.
  5. [§3.3] The margin term is entirely derived from Llama-3 70B synthetic judgments, yet the paper never validates that Llama-3's disagreement distribution matches the human annotators' '10 people' estimates; a direct comparison between the synthetic margin and human disagreement on the test set (e.g., correlation between m and the spread of human judgments) would test the core premise, which the Social Impacts Statement itself acknowledges has 'mixed empirical validation'.
minor comments (5)
  1. [§2, Eq. (2)] The phrase 'is and indicator function' contains a typo; it should be 'is an indicator function'.
  2. [§3.2] The sentence 'Empirical evidence suggests that this approach yields more reliable results...' lacks a citation; please add a reference.
  3. [Appendix B, Task 3] In the 'User Preference Estimation' scale, the last option is listed as '10 A - 0 B' instead of '0 A - 10 B'; this appears to be a copy-paste error.
  4. [§3, opening sentence] The phrase 'we sent a categorization' should be 'we present a categorization'.
  5. [Tables 1-2] Please clarify whether Pearson correlations and L1 losses are computed on the normalized 0-1 preference fractions or on raw reward scores, and state this in the captions.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the synthetic margin is an independent proxy, not a refit of the human evaluation labels.

full rationale

The paper's core derivation chain is: define aggregate user preference p* (Eq. 2), estimate it via the 'If you asked 10 people' annotator protocol, compute a margin m from Llama-3 synthetic judgments (Eq. 3), and train with m in the loss (Eq. 4). The margin is constructed entirely from Llama-3 outputs and is never fitted to the human test labels; the test labels are collected separately for evaluation. Thus the reported improvement in Pearson correlation and L1 loss against human aggregate preferences is an externally falsifiable empirical claim, not a consequence of the definitions. The margin idea is credited to Touvron et al. (Llama 2), an external source. Self-citations (PRISM [27], [10], [28]) appear as related work or as one OOD test set; none is load-bearing for the central result. Two concerns are raised but they are validity risks, not circularity: (i) the baseline RM is a released OpenAssistant checkpoint rather than a matched retraining without the margin, so Tables 1-2 may confound the margin with training setup; (ii) the Social Impacts Statement itself flags that 'Our assumption that LLMs can better approximate preference heterogeneity than individual annotators has mixed empirical validation,' which bears on whether the synthetic margin is a valid proxy. These concerns do not make Eq. (4) reduce to Eq. (2) or rename a fitted parameter as a prediction, so the circularity score remains low.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two unvalidated assumptions: that Llama-3 synthetic judgments capture human disagreement (flagged by the authors), and that the self-report population-preference protocol measures what it claims. Three hand-chosen hyperparameters (n=10, top-p=0.9, learning rate) affect the results. No new physical or conceptual entities are introduced.

free parameters (3)
  • n (number of synthetic judgments per example) = 10
    Chosen by hand in Section 3.3. The margin in Eq. (3) is computed from these 10 samples, so n sets the granularity of the disagreement estimate.
  • top-p (nucleus sampling parameter) = 0.9
    Chosen to 'mimic the variability of a group of annotators' (Section 3.3). No sensitivity analysis is provided.
  • Learning rate = not reported
    Selected via hyperparameter sweep from 1e-04 to 1e-06 on validation data (footnote 6). The paper does not report the chosen value, making exact replication harder.
assumptions (4)
  • ad hoc to paper LLM synthetic judgments approximate the preference heterogeneity of a human population.
    The entire margin signal is derived from Llama-3 70B judgments. Section 4 (Social Impacts) flags this assumption as having 'mixed empirical validation' [46-49]. If false, the method injects model bias instead of correcting for human subjectivity.
  • domain assumption The 'If you asked 10 people' annotator protocol yields a valid estimate of aggregate preference p*.
    Section 3.2 and Appendix B use this self-report method as ground truth. The paper asserts empirical evidence supports it but provides no citation or validation.
  • domain assumption Test examples from the four in-domain datasets are disjoint from the reward model's training data.
    Section 3.2 says test examples are 'randomly selected' from datasets used to train the RM, but does not state that they were held out. Overlap would inflate reported ID correlations.
  • standard math Standard statistical tools (sigmoid loss, Pearson correlation, L1 loss) are appropriate for the analysis.
    These are conventional and unproblematic, though the lack of error bars is a separate issue.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond the Binary: Capturing Diverse Preferences With Reward Regularization." pith.science (2026). https://pith.science/paper/EFRLLTVK

@misc{pith2026241203822,
  author       = {Pith},
  title        = {Pith review of: Beyond the Binary: Capturing Diverse Preferences With Reward Regularization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EFRLLTVK}},
  note         = {Machine review of arXiv:2412.03822}
}
read the original abstract

Large language models (LLMs) are increasingly deployed via public-facing interfaces to interact with millions of users, each with diverse preferences. Despite this, preference tuning of LLMs predominantly relies on reward models trained using binary judgments where annotators select the preferred choice out of pairs of model outputs. In this work, we argue that this reliance on binary choices does not capture the broader, aggregate preferences of the target user in real-world tasks. We propose a taxonomy that identifies two dimensions of subjectivity where different users disagree on the preferred output-namely, the Plurality of Responses to Prompts, where prompts allow for multiple correct answers, and the Indistinguishability of Responses, where candidate outputs are paraphrases of each other. We show that reward models correlate weakly with user preferences in these cases. As a first step to address this issue, we introduce a simple yet effective method that augments existing binary preference datasets with synthetic preference judgments to estimate potential user disagreement. Incorporating these via a margin term as a form of regularization during model training yields predictions that better align with the aggregate user preferences.

Figures

Figures reproduced from arXiv: 2412.03822 by the authors.

Figure 1
Figure 1. Two dimensions of subjectivity along which reward model predictions correlate poorly [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 15 canonical work pages

  1. [1]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  2. [2]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  3. [3]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    arXiv 2023

  4. [4]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  5. [5]

    How far are we from agi

    Tao Feng, Chuanyang Jin, Jingyu Liu, Kunlun Zhu, Haoqin Tu, Zirui Cheng, Guanyu Lin, and Jiaxuan You. How far are we from agi. arXiv preprint arXiv:2405.10313, 2024

    arXiv 2024

  6. [6]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020

    2020

  7. [7]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  8. [8]

    Simpo: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024

    arXiv 2024

Show all 57 references
  1. [9]

    The History and Risks of Reinforce- ment Learning and Human Feedback, November 2023

    Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick. The History and Risks of Reinforce- ment Learning and Human Feedback, November 2023. URL http://arxiv.org/abs/ 2310.13595. arXiv:2310.13595 [cs]

    2023 arXiv

  2. [10]

    The past, present and better future of feedback learning in large language models for subjective human preferences and values

    Hannah Kirk, Andrew Bean, Bertie Vidgen, Paul Röttger, and Scott Hale. The past, present and better future of feedback learning in large language models for subjective human preferences and values. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

    2023

  3. [11]

    Artificial Intelligence, Values and Alignment

    Iason Gabriel. Artificial Intelligence, Values and Alignment. Minds and Machines, 30(3): 411–437, September 2020. ISSN 0924-6495, 1572-8641. doi: 10.1007/s11023-020-09539-2

    2020 doi

  4. [12]

    Artificial Intelligence, Humanistic Ethics

    John Tasioulas. Artificial Intelligence, Humanistic Ethics. Daedalus, 151(2):232–243, May

  5. [13]

    Beyond preferences in ai alignment

    Tan Zhi-Xuan, Micah Carroll, Matija Franklin, and Hal Ashton. Beyond preferences in ai alignment. arXiv preprint arXiv:2408.16984, 2024

    2024 arXiv

  6. [14]

    Meta Community Forum: Re- sults Analysis

    Samuel Chang, Estelle Ciesla, Michael Finch, James Fishkin, Lodewijk Gelauff, Ashish Goel, Ricky Hernandez Marquez, Shoaib Mohammed, and Alice Siu. Meta Community Forum: Re- sults Analysis. Technical report, Deliberative Democracy Lab, Stanford University, April 2024. URL http...

    2024

  7. [15]

    STELA: a community-centred approach to norm elicitation for AI alignment

    Stevie Bergman, Nahema Marchal, John Mellor, Shakir Mohamed, Iason Gabriel, and William Isaac. STELA: a community-centred approach to norm elicitation for AI alignment. Scientific Reports, 14(1):6616, March 2024. ISSN 2045-2322. doi: 10.1038/s41598-024-56648-4. URL https://www...

    2024 doi

  8. [16]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    2022 arXiv

  9. [17]

    Improving alignment of dialogue agents via targeted human judgements

    Amelia Glaese, Nat McAleese, Maja Tr˛ ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022

    2022 arXiv

  10. [18]

    Collective constitutional ai: Aligning a language model with public input

    Saffron Huang, Divya Siddarth, Liane Lovitt, Thomas I Liao, Esin Durmus, Alex Tamkin, and Deep Ganguli. Collective constitutional ai: Aligning a language model with public input. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1395–1417, 2024

    2024

  11. [19]

    Introducing Meta Llama 3: The most capable openly available LLM to date, April

    MetaAI. Introducing Meta Llama 3: The most capable openly available LLM to date, April

  12. [20]

    Mmtom-qa: Multimodal theory of mind question answering

    Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua B Tenenbaum, and Tianmin Shu. Mmtom-qa: Multimodal theory of mind question answering. arXiv preprint arXiv:2401.08743, 2024

    2024 arXiv

  13. [21]

    ChatGPT’s weekly users have doubled in less than a year

    Emma Roth. ChatGPT’s weekly users have doubled in less than a year. The Verge, August 2024. URL https://www.theverge.com/2024/8/29/24231685/ openai-chatgpt-200-million-weekly-users

    2024

  14. [22]

    Lamda: Language models for dialog applications

    Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022

    2022 arXiv

  15. [23]

    Taylor, Mark Diaz, Christopher M

    Lora Aroyo, Alex S. Taylor, Mark Diaz, Christopher M. Homan, Alicia Parrish, Greg Serapio- Garcia, Vinodkumar Prabhakaran, and Ding Wang. DICES Dataset: Diversity in Conversational AI Evaluation for Safety, June 2023. URL http://arxiv.org/abs/2306.11247. arXiv:2306.11247 [cs]

    2023 arXiv

  16. [24]

    Pretraining language models with human preferences

    Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, pages 17506–17533. PMLR, 2023

    2023

  17. [25]

    alignment

    Hannah Kirk, Bertie Vidgen, Paul Rottger, and Scott Hale. The empty signifier problem: Towards clearer paradigms for operationalising" alignment”in large language models. In Socially Responsible Language Modelling Research, 2023

    2023

  18. [26]

    Dreier, Emily K

    Suchin Gururangan, Dallas Card, Sarah K. Dreier, Emily K. Gade, Leroy Z. Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A. Smith. Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection, January 2022. URL http://arxiv.org/ abs/2201.10474. arXiv...

    2022 arXiv

  19. [27]

    Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. The PRISM Alignment Project: What Participatory, Representative and Individualised Human Fee...

    2024 arXiv

  20. [28]

    The cultural psychol- ogy of large language models: Is chatgpt a holistic or analytic thinker? arXiv preprint arXiv:2308.14242, 2023

    Chuanyang Jin, Songyang Zhang, Tianmin Shu, and Zhihan Cui. The cultural psychol- ogy of large language models: Is chatgpt a holistic or analytic thinker? arXiv preprint arXiv:2308.14242, 2023

    2023 arXiv

  21. [29]

    Diverging preferences: When do annotators disagree and do models know? arXiv preprint arXiv:2410.14632, 2024

    Michael JQ Zhang, Zhilin Wang, Jena D Hwang, Yi Dong, Olivier Delalleau, Yejin Choi, Eunsol Choi, Xiang Ren, and Valentina Pyatkin. Diverging preferences: When do annotators disagree and do models know? arXiv preprint arXiv:2410.14632, 2024. 8

  22. [30]

    Personalized soups: Per- sonalized large language model alignment via post-hoc parameter merging

    Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Per- sonalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310.11564, 2023

    arXiv

  23. [31]

    Lipton, and Liu Leqi

    Xinyu Li, Zachary C. Lipton, and Liu Leqi. Personalized Language Modeling from Person- alized Human Feedback, February 2024. URL http://arxiv.org/abs/2402.05133. arXiv:2402.05133 [cs]

    2024 arXiv

  24. [32]

    Personal- izing reinforcement learning from human feedback with variational preference learning

    Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personal- izing reinforcement learning from human feedback with variational preference learning. arXiv preprint arXiv:2408.10075, 2024

    2024 arXiv

  25. [33]

    Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards.Advances in Neural Information Processing Systems, 36, 2024

    Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards.Advances in Neural Information Processing System...

    2024

  26. [34]

    Fine-tuning language models to find agreement among humans with diverse preferences

    Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell- Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neura...

    2022

  27. [35]

    Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences

    Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.08925, 2024

    2024 arXiv

  28. [36]

    Holliday, Bob M

    Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H. Holliday, Bob M. Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, Emanuel Tewolde, and William S. Zwicker. Social Choice for AI Alignment: Dealing with Diverse Human Feedback, April...

    2024 arXiv

  29. [37]

    Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF, December 2023

    Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell. Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF, December 2023. URL http://arxiv.org/abs/2312.08358. arXiv:2312.08358 [cs, stat]

    2023 arXiv

  30. [38]

    Aligning Crowd Feedback via Distributional Preference Reward Modeling, February 2024

    Dexun Li, Cong Zhang, Kuicai Dong, Derrick Goh Xin Deik, Ruiming Tang, and Yong Liu. Aligning Crowd Feedback via Distributional Preference Reward Modeling, February 2024. URL http://arxiv.org/abs/2402.09764. arXiv:2402.09764 [cs]

    2024 arXiv

  31. [39]

    Uncertainty- aware reward model: Teaching reward models to know what is unknown

    Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, and Junge Zhang. Uncertainty- aware reward model: Teaching reward models to know what is unknown. arXiv preprint arXiv:2410.00847, 2024

    2024 arXiv

  32. [40]

    Webgpt: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021

    2021 arXiv

  33. [41]

    synthetic-instruct-gptj-pairwise (revision cc92d8d),

    Alex Havrilla. synthetic-instruct-gptj-pairwise (revision cc92d8d),

  34. [42]

    Ultrafeedback: Boosting language models with high-quality feedback, 2023

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023

    2023

  35. [43]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  36. [44]

    Transformers: State- of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in n...

    2020

  37. [45]

    Holistic evaluation of language models

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022

    2022 arXiv

  38. [46]

    Argyle, Ethan C

    Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua Gubler, Christopher Rytting, and David Wingate. Out of One, Many: Using Language Models to Simulate Human Samples, September

  39. [47]

    Are large language models good annotators? In Proceedings on, pages 38–48

    Jay Mohta, Kenan Ak, Yan Xu, and Mingwei Shen. Are large language models good annotators? In Proceedings on, pages 38–48. PMLR, 2023

    2023

  40. [48]

    Automated social science: Language models as scientist and subjects

    Benjamin S Manning, Kehang Zhu, and John J Horton. Automated social science: Language models as scientist and subjects. Technical report, National Bureau of Economic Research, 2024

    2024

  41. [49]

    Large language models can- not replace human participants because they cannot portray identity groups

    Angelina Wang, Jamie Morgenstern, and John P Dickerson. Large language models can- not replace human participants because they cannot portray identity groups. arXiv preprint arXiv:2402.01908, 2024

    2024 arXiv

  42. [50]

    arXiv:2209.06899 [cs]

    URL http://arxiv.org/abs/2209.06899. arXiv:2209.06899 [cs]

    arXiv

  43. [51]

    Stevie Bergman, Jennifer Chien, Mark Díaz, Seliem El-Sayed, Jaylen Pittman, Shakir Mohamed, and Kevin R

    William Agnew, A. Stevie Bergman, Jennifer Chien, Mark Díaz, Seliem El-Sayed, Jaylen Pittman, Shakir Mohamed, and Kevin R. McKee. The illusion of artificial inclusion, February

  44. [52]

    A Roadmap to Pluralistic Alignment, February 2024

    Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Al- thoff, and Yejin Choi. A Roadmap to Pluralistic Alignment, February 2024. URL http: //arxiv.org/abs/2402.0...

    2024 arXiv

  45. [54]

    Whose opinions do language models reflect? In International Conference on Machine Learning, pages 29971–30004

    Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International Conference on Machine Learning, pages 29971–30004. PMLR, 2023

    2023

  46. [56]

    arXiv:2401.08572 [cs]

    URL http://arxiv.org/abs/2401.08572. arXiv:2401.08572 [cs]

    arXiv

  47. [2022]

    doi: 10.1162/daed_a_01912

    ISSN 0011-5266. doi: 10.1162/daed_a_01912. URL https://doi.org/10.1162/ daed_a_01912

    doi

  48. [2023]

    URL https://huggingface.co/datasets/Dahoas/ synthetic-instruct-gptj-pairwise

  49. [2024]

    URL https://ai.meta.com/blog/meta-llama-3/

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.