arxiv: 2604.25895 · v1 · submitted 2026-04-28 · 💻 cs.CY · cs.AI· cs.CL

Recognition: unknown

Three Models of RLHF Annotation: Extension, Evidence, and Authority

Steve Coyne

Pith reviewed 2026-05-07 14:23 UTC · model grok-4.3

classification 💻 cs.CY cs.AIcs.CL

keywords RLHFhuman annotationpreference alignmentAI alignmentannotation modelsreinforcement learningvalue alignmenthuman feedback

0 comments

The pith

RLHF pipeline designers should decompose annotations into three separable dimensions and tailor each pipeline to the model most appropriate for that dimension rather than seeking a single unified pipeline.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper distinguishes three conceptual models for the role of human annotators in RLHF. In the extension model, annotators extend the system designers' own judgments about desired outputs. In the evidence model, annotators supply independent facts about moral, social, or other matters. In the authority model, annotators exercise independent authority as population representatives to determine outputs. These models carry different requirements for soliciting, validating, and aggregating annotations. Conflating the models produces identifiable failure modes, while decomposing annotations by dimension allows pipelines to match the right model to each part of the task.

Core claim

The normative role of annotators' judgments in preference-based alignment methods like RLHF can be understood through three distinct models: extension of designers' judgments, provision of independent evidence on facts, or exercise of independent authority as population representatives. Landmark papers in the RLHF literature implicitly rely on one or more of these models. Unintentional or intentional conflation of the models creates specific failure modes in how annotations are collected and used. Normative criteria can guide selection among the models, and the central recommendation is to decompose annotation tasks into separable dimensions so that each can use the pipeline best suited to a

What carries the argument

The three conceptual models of annotator roles—extension, evidence, and authority—which each imply distinct procedures for soliciting, validating, and aggregating annotations in RLHF pipelines.

If this is right

Solicitation, validation, and aggregation methods must differ depending on whether the model is extension, evidence, or authority.
Failure modes arise when a pipeline designed for one model is applied to annotations that fit another.
Normative criteria for choosing among the models can be derived from the specific goals of each annotation dimension.
Decomposition allows separate pipelines to be optimized without forcing a single approach across all judgments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same decomposition approach could apply to other forms of human feedback beyond RLHF, such as in direct preference optimization or constitutional AI.
Modular pipelines might reduce unintended value imposition by designers when authority dimensions are handled separately.
Implementation tests could compare error rates or consistency metrics between unified and decomposed annotation processes on the same tasks.

Load-bearing premise

The three models are meaningfully distinct, conflating them produces identifiable failure modes, and annotations can be decomposed into dimensions without losing essential information.

What would settle it

An empirical comparison showing that a single unified RLHF annotation pipeline achieves equivalent alignment outcomes to decomposed pipelines without exhibiting the distinct failure modes predicted by model conflation.

read the original abstract

Preference-based alignment methods, most prominently Reinforcement Learning with Human Feedback (RLHF), use the judgments of human annotators to shape large language model behaviour. However, the normative role of these judgments is rarely made explicit. I distinguish three conceptual models of that role. The first is extension: annotators extend the system designers' own judgments about what outputs should be. The second is evidence: annotators provide independent evidence about some facts, whether moral, social or otherwise. The third is authority: annotators have some independent authority (as representatives of the broader population) to determine system outputs. I argue that these models have implications for how RLHF pipelines should solicit, validate and aggregate annotations. I survey landmark papers in the literature on RLHF and related methods to illustrate how they implicitly draw on these models, describe failure modes that come from unintentionally or intentionally conflating them, and offer normative criteria for choosing among them. My central recommendation is that RLHF pipeline designers should decompose annotation into separable dimensions and tailor each pipeline to the model most appropriate for that dimension, rather than seeking a single unified pipeline.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper gives a useful new taxonomy for annotator roles in RLHF and maps it to pipeline choices, but the decomposition recommendation lacks operational criteria and examples.

read the letter

The main takeaway is a clean distinction among three models of what annotators are doing: extending the designers' own judgments, providing evidence on moral or social facts, or exercising independent authority as population representatives. The paper shows how landmark RLHF papers implicitly rely on one or more of these without saying so, and it flags failure modes that follow from treating them as interchangeable. That framing is new and helps make the normative stakes explicit when choosing how to collect, validate, and combine preferences. The survey of existing work is fair and points to concrete places where conflation shows up. The central recommendation—that designers should break annotation tasks into separable dimensions and match each to the right model rather than forcing one pipeline—follows logically from the distinctions. The soft spot is that the argument does not supply workable criteria for deciding which model dominates a given annotation, nor does it include even one worked example of decomposition in a real task like helpfulness or safety labeling. In practice many preferences blend extension, evidence, and authority at once, so clean separation may require arbitrary choices or lose signal. The paper is aimed at people who design or critique RLHF pipelines and at alignment researchers who want clearer language for value incorporation. It is worth a serious referee because the taxonomy is coherent and the topic matters, even though the normative claim will need more concrete guidance to land.

Referee Report

2 major / 1 minor

Summary. The paper distinguishes three conceptual models of the normative role of human annotators in RLHF: extension (annotators extend designers' own judgments), evidence (annotators supply independent evidence on moral, social or other facts), and authority (annotators exercise representative authority over outputs). It surveys landmark RLHF papers to show implicit reliance on these models, identifies failure modes arising from their conflation, and recommends that pipeline designers decompose annotations into separable dimensions and tailor solicitation, validation and aggregation procedures to the dominant model for each dimension rather than pursuing a single unified pipeline.

Significance. If the distinctions are robust and the failure modes are as described, the framework would be significant for AI alignment research by making explicit the normative assumptions that are usually left implicit in preference data collection. The literature survey grounds the concepts in concrete examples from existing work, and the emphasis on explicit model choice could improve the defensibility and consistency of RLHF systems. The paper's main strength is its provision of a structured normative lens rather than any empirical result or derivation.

major comments (2)

[§2] §2 (definitions of the three models): the separability assumption required for the decomposition recommendation is not established; the paper does not supply operational criteria for determining whether a given annotation (e.g., a safety or helpfulness judgment) is governed primarily by extension, evidence, or authority when these roles frequently co-occur in a single preference.
[the section on normative criteria and recommendations] The section presenting the central recommendation: while failure modes from conflation are illustrated via the survey, no procedure is given for partitioning real annotation tasks into the three dimensions without arbitrary choices or loss of blended information, leaving the practical advice under-specified relative to the strength of the normative claim.

minor comments (1)

[abstract] The abstract introduces 'normative criteria for choosing among them' without indicating what form those criteria take; a one-sentence preview would improve readability for readers encountering the framework for the first time.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments, which help clarify the scope and applicability of our framework. We address the major comments point by point below, indicating where revisions will be made to the manuscript.

read point-by-point responses

Referee: [§2] §2 (definitions of the three models): the separability assumption required for the decomposition recommendation is not established; the paper does not supply operational criteria for determining whether a given annotation (e.g., a safety or helpfulness judgment) is governed primarily by extension, evidence, or authority when these roles frequently co-occur in a single preference.

Authors: We agree that greater specificity on application would strengthen the paper. The models are analytically distinct because each carries different implications for validity and aggregation: extension requires fidelity to designer intent, evidence requires correspondence to external facts, and authority requires representativeness of a population. When roles co-occur, the framework recommends decomposing the annotation into sub-dimensions where one model predominates rather than applying a single model to the whole. We will revise §2 to include guiding questions for classification (e.g., 'Does the judgment appeal primarily to designer-specified values, observable facts, or collective preferences?') and brief examples drawn from safety and helpfulness tasks. This supplies operational heuristics while acknowledging that some interpretive judgment remains. revision: yes
Referee: The section presenting the central recommendation: while failure modes from conflation are illustrated via the survey, no procedure is given for partitioning real annotation tasks into the three dimensions without arbitrary choices or loss of blended information, leaving the practical advice under-specified relative to the strength of the normative claim.

Authors: The referee is correct that the recommendation is normative and does not include a complete step-by-step partitioning algorithm. The paper's survey demonstrates concrete failure modes from conflation, but the advice on decomposition is intentionally flexible to accommodate varied RLHF contexts. To reduce under-specification, we will expand the recommendations section with a worked example of a safety annotation decomposed into extension (designer-defined harm thresholds), evidence (empirical harm data), and authority (representative public values) components, including how to combine outputs via weighted aggregation. This illustrates handling of blended information without claiming to eliminate all arbitrariness. The revision will be incorporated. revision: yes

Circularity Check

0 steps flagged

No circularity: purely conceptual distinctions with no derivations or self-referential reductions

full rationale

The paper advances a tripartite conceptual taxonomy of RLHF annotation roles (extension, evidence, authority) and derives normative recommendations from logical distinctions and a literature survey. No equations, fitted parameters, or predictive derivations appear in the provided text or abstract. The central claim—that pipelines should decompose annotations by model—rests on explicit failure-mode illustrations from external papers rather than any self-definition, self-citation chain, or renaming of prior results by the same author. The argument is therefore self-contained against external benchmarks and contains no load-bearing step that reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The work rests on standard domain assumptions in AI alignment without introducing fitted parameters or new entities; the framework itself is an ad-hoc conceptual distinction.

axioms (2)

domain assumption Human annotators' judgments play a normative role in shaping LLM outputs via RLHF
This is the foundational premise that makes the three models relevant.
ad hoc to paper The extension, evidence, and authority models are distinct and have different practical implications for annotation pipelines
The paper's recommendation depends on this distinction being both meaningful and actionable in design choices.

pith-pipeline@v0.9.0 · 5486 in / 1409 out tokens · 144780 ms · 2026-05-07T14:23:53.905650+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

56 extracted references · 16 canonical work pages · 6 internal anchors

[1]

Arash Abizadeh. 2021. Counter-majoritarian democracy: Persistent minorities, federalism, and the power of numbers.American Political Science Review115, 3 (2021), 742–756

2021
[2]

Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation.AI magazine36, 1 (2015), 15–24. 15 FAccT ’26, June 25–28, 2026, Montreal, QC, Canada Coyne

2015
[3]

Ron Artstein. 2017. Inter-annotator agreement. InHandbook of linguistic annotation. Springer, 297–313

2017
[4]

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for alignment.arXiv preprint arXiv:2112.00861(2021)

work page internal anchor Pith review arXiv 2021
[5]

Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. InInternational Conference on Artificial Intelligence and Statistics. PMLR, 4447–4455

2024
[6]

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862(2022)

work page internal anchor Pith review arXiv 2022
[7]

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073(2022)

work page internal anchor Pith review arXiv 2022
[8]

Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Allocative versus representational harms in machine learning. In9th Annual conference of the special interest group for computing, information and society, Vol. 1. New York, NY

2017
[9]

Maarten Buyl, Hadi Khalaf, Claudio Mayrink Verdun, Lucas Monteiro Paes, Caio Cesar Vieira Machado, and Flavio du Pin Calmon. 2025. Ai alignment at your discretion. InProceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency. 3046–3074

2025
[10]

Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217(2023)

work page arXiv 2023
[11]

Louis Castricato, Nathan Lile, Rafael Rafailov, Jan-Philipp Fränken, and Chelsea Finn. 2025. Persona: A reproducible testbed for pluralistic alignment. InProceedings of the 31st International Conference on Computational Linguistics. 11348–11368

2025
[12]

Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. 2024. Maxmin-rlhf: Alignment with diverse human preferences.arXiv preprint arXiv:2402.08925(2024)

work page arXiv 2024
[13]

Chen Cheng, Hilal Asi, and John Duchi. 2022. How many labelers do you have? A closer look at gold-standard labels.arXiv preprint arXiv:2206.12041(2022)

work page arXiv 2022
[14]

2020.The alignment problem: Machine learning and human values

Brian Christian. 2020.The alignment problem: Machine learning and human values. WW Norton & Company

2020
[15]

Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H Holliday, Bob M Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, et al. 2024. Social choice should guide ai alignment in dealing with diverse human feedback.arXiv preprint arXiv:2404.10271(2024)

work page arXiv 2024
[16]

2009.The second-person standpoint: Morality, respect, and accountability

Stephen Darwall. 2009.The second-person standpoint: Morality, respect, and accountability. Harvard University Press

2009
[17]

David Enoch. 2011. Giving practical reasons.The Philosopher’s Imprint11, 4 (2011)

2011
[18]

2008.Democratic Authority: A Philosophical Framework

David Estlund. 2008.Democratic Authority: A Philosophical Framework. Princeton University Press

2008
[19]

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306(2024)

work page internal anchor Pith review arXiv 2024
[20]

Arduin Findeis, Timo Kaufmann, Eyke Hüllermeier, Samuel Albanie, and Robert Mullins. 2024. Inverse constitutional ai: Compressing preferences into principles.arXiv preprint arXiv:2406.06560(2024)

work page arXiv 2024
[21]

2007.Epistemic injustice: Power and the ethics of knowing

Miranda Fricker. 2007.Epistemic injustice: Power and the ethics of knowing. Oxford university press

2007
[22]

Iason Gabriel. 2020. Artificial intelligence, values, and alignment.Minds and machines30, 3 (2020), 411–437

2020
[23]

Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al . 2022. Improving alignment of dialogue agents via targeted human judgements.arXiv preprint arXiv:2209.14375(2022)

work page arXiv 2022
[24]

Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022. Jury learning: Integrating dissenting voices into machine learning models. InProceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–19

2022
[25]

Guerrero

Alexander A. Guerrero. 2014. Against Elections: The Lottocratic Alternative.Philosophy & Public Affairs42, 2 (2014), 135–178. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/papa.12029 doi:10.1111/papa.12029

work page doi:10.1111/papa.12029 2014
[26]

1991.Whose science? Whose knowledge?: Thinking from women’s lives

Sandra Harding. 1991.Whose science? Whose knowledge?: Thinking from women’s lives. Cornell University Press

1991
[27]

1982.Essays on Bentham: Jurisprudence and political philosophy

Herbert Lionel Adolphus Hart. 1982.Essays on Bentham: Jurisprudence and political philosophy. OUP Oxford

1982
[28]

democratizing AI

Johannes Himmelreich. 2023. Against “democratizing AI”.AI & SOCIETY38, 4 (2023), 1333–1346

2023
[29]

Saffron Huang, Divya Siddarth, Liane Lovitt, Thomas I Liao, Esin Durmus, Alex Tamkin, and Deep Ganguli. 2024. Collective constitutional ai: Aligning a language model with public input. InProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency. 1395–1417

2024
[30]

Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hannaneh Hajishirzi. 2024. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback.Advances in neural information processing systems37 (2024), 36602–36633. 16 Three Models of RLHF Annotation: Extension, Ev...

2024
[31]

Hannah R Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. 2024. The PRISM alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models.Advances in Neur...

2024
[32]

Niko Kolodny. 2014. Rule over none I: What justifies democracy?Philosophy & Public Affairs42, 3 (2014), 195–229

2014
[33]

Nathan Lambert. 2024. RLHF Roundup: Getting Good at PPO, Sketching RLHF’s Impact, RewardBench Retrospective, and a Reward Model Competition. Interconnects (Substack). https://www.interconnects.ai/p/rlhf-roundup-2024 Accessed: 2026-03-25

2024
[34]

Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2024. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. InProceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Lea...

2024
[35]

Andreas Matthias. 2004. The responsibility gap: Ascribing responsibility for the actions of learning automata.Ethics and information technology6, 3 (2004), 175–183

2004
[36]

Evi Micha. 2025. How Proportional Representation Can Shape Artificial Intelligence.AI Magazine46, 4 (2025), e70044. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/aaai.70044 doi:10.1002/aaai.70044

work page doi:10.1002/aaai.70044 2025
[37]

R Millière and C Buckner. 2024. A philosophical introduction to language models–part I: continuity with classic debates. arXiv

2024
[38]

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback.Advances in neural information processing systems35 (2022), 27730–27744

2022
[39]

Duncan Purves and Jeremy Davis. 2022. Public Trust, Institutional Legitimacy, and the Use of Algorithms in Criminal Justice.Public Affairs Quarterly36, 2 (2022), 136–162

2022
[40]

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems36 (2023), 53728–53741

2023
[41]

Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic. 2024. Group robust preference optimization in reward-free rlhf.Advances in Neural Information Processing Systems37 (2024), 37100–37137

2024
[42]

Alan Rubel, Clinton Castro, and Adam Pham. 2019. Agency laundering and information technologies.Ethical Theory and Moral Practice 22, 4 (2019), 1017–1041

2019
[43]

Filippo Santoni de Sio and Giulio Mecacci. 2021. Four responsibility gaps with artificial intelligence: Why they matter and how to address them.Philosophy & technology34, 4 (2021), 1057–1084

2021
[44]

1996.Questions and answers in attitude surveys: Experiments on question form, wording, and context

Howard Schuman and Stanley Presser. 1996.Questions and answers in attitude surveys: Experiments on question form, wording, and context. Sage

1996
[45]

Nick Schuster and Daniel Kilov. 2025. Moral disagreement and the limits of AI value alignment: a dual challenge of epistemic justification and political legitimacy.AI & SOCIETY(2025), 1–15

2025
[46]

Elizabeth Seger, Aviv Ovadya, Divya Siddarth, Ben Garfinkel, and Allan Dafoe. 2023. Democratising AI: Multiple meanings, goals, and methods. InProceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. 715–722

2023
[47]

Eric Shoemaker. 2024. Democratic Equality Requires Randomly Selecting Legislators.Public Affairs Quarterly38, 2 (2024), 132–152

2024
[48]

Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. 2024. A roadmap to pluralistic alignment.arXiv preprint arXiv:2402.05070(2024)

work page arXiv 2024
[49]

Andre Steingrüber and Kevin Baum. 2025. Justifications for Democratizing AI Alignment and Their Prospects. InInternational Conference on Bridging the Gap between AI and Reality. Springer, 146–159

2025
[50]

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano
[51]

Learning to summarize with human feedback.Advances in neural information processing systems33 (2020), 3008–3021

2020
[52]

Jake Iain Stone and Brent Mittelstadt. 2024. Legitimate Power, Illegitimate Automation: The problem of ignoring legitimacy in automated decision systems.ACM Journal on Responsible Computing(2024)

2024
[53]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971 (2023)

work page internal anchor Pith review arXiv 2023
[54]

Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Scowcroft, Neel Kant, Aidan Swope, et al . 2024. Helpsteer: Multi-attribute helpfulness dataset for steerlm. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T...

2024
[55]

Siyan Zhao, John Dang, and Aditya Grover. 2023. Group preference optimization: Few-shot alignment of large language models.arXiv preprint arXiv:2310.11523(2023)

work page arXiv 2023
[56]

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593(2019). 17

work page internal anchor Pith review arXiv 2019