Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges

Afrar Jahin; Arif Hassan Zidan; Bolun Sun; Chao Huang; Dajiang Zhu; Fei Dou; Hanqi Jiang; Haoran Lu; Haotian Xiang; Huimin Cheng

REVIEW 4 major objections 4 minor 1 cited by

Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This survey claims that supervised fine-tuning instills basic instruction-following, while preference-based methods such as RLHF and DPO better capture nuanced human intent, and it maps the trade-offs among safety, helpfulness, and honesty.

desk verdict A broad, useful survey of LLM alignment whose current text is unverifiable because it lacks a bibliography and contains placeholder citations, including in sections where quantitative claims sit. read the letter →

arxiv 2507.19672 v1 pith:J7EI3HMN submitted 2025-07-25 cs.AI cs.LGstat.ML

Haoran Lu , Luyang Fang , Ruidong Zhang , Xinliang Li , Jiazhang Cai , Huimin Cheng , Lin Tang , Ziyu Liu

show 42 more authors

Zeliang Sun Tao Wang Yingchuan Zhang Arif Hassan Zidan Jinwen Xu Jincheng Yu Meizhi Yu Hanqi Jiang Xilin Gong Weidi Luo Bolun Sun Yongkai Chen Terry Ma Shushan Wu Yifan Zhou Junhao Chen Haotian Xiang Jing Zhang Afrar Jahin Wei Ruan Ke Deng Yi Pan Peilong Wang Jiahui Li Zhengliang Liu Lu Zhang Lin Zhao Wei Liu Dajiang Zhu Xin Xing Fei Dou Wei Zhang Chao Huang Rongjie Liu Mengrui Zhang Yiwen Liu Xiaoxiao Sun Qin Lu Zhen Xiang Wenxuan Zhong Tianming Liu Ping Ma

This is my paper

classification cs.AIcs.LGstat.ML

keywords LLMalignmentAIsafetyreinforcementlearningfromhumanfeedbackdirectpreferenceoptimizationconstitutionaljailbreakattacksbenchmarksuncertainty

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey tries to establish a map of how large language models are aligned with human values and to identify which training approaches actually deliver which kind of alignment. Its central analytical claim is that supervised fine-tuning (SFT) reliably teaches basic instruction-following, but preference-based methods—reinforcement learning from human feedback (RLHF), direct preference optimization (DPO), and related techniques—are more flexible for capturing nuanced human intent. The paper organizes the field around a three-part objective—helpfulness, harmlessness, and honesty—and shows that these goals trade off against one another, motivating hierarchical safeguards. It also argues that current evaluation is limited by reward misspecification, distributional robustness gaps, and the difficulty of scalable oversight. A sympathetic reader would care because the survey turns a scattered literature into a structured set of choices about how to train, evaluate, and govern aligned models.

What carries the argument

The organizing mechanism is the contrast between demonstration-based and preference-based training, mounted on a three-objective frame of helpfulness, harmlessness, and honesty. SFT minimizes token-level cross-entropy against fixed reference responses, which is maximum-likelihood imitation; RLHF instead trains a reward model on human or AI preferences and optimizes the policy against it, using PPO or actor-only variants such as GRPO. DPO is treated as the key bridge: it shows the optimal RLHF policy has a closed form, so preference alignment can be done without a separate reward model. The survey uses this machinery to explain why SFT alone under-generalizes—its objective encodes a single correct answer and no graded preference—and why preference-based training is more flexible but introduces new failure modes in reward modeling and optimization. It also introduces hierarchical policy stacking as the design pattern that resolves objective trade-offs: safety filter first, then fact-checking, then helpfulness optimization.

What would settle it

A head-to-head benchmark in which supervised fine-tuning on a rich, preference-ranked dataset matches or beats RLHF and DPO on ambiguous, value-laden instructions would falsify the central claim that preference-based methods are more flexible for nuanced human intent; alternatively, discovering that the paper's placeholder citations do not correspond to verifiable published work would falsify the survey's synthesis.

Watch

Extended reading notes

Core claim

The paper's central claim, stated in its own terms, is that the alignment field has converged on a canonical three-objective framing—helpfulness, harmlessness, and honesty—and that the two dominant training paradigms sit at different points on a flexibility spectrum. Supervised fine-tuning on demonstrations teaches a model to imitate aligned behavior and provides a stable, cheap first stage, but it treats every target response as equally correct and therefore cannot represent graded or comparative judgments. Preference-based methods, by contrast, learn from relative judgments between candidate outputs, allowing the model to internalize subtle distinctions in human intent; this is why production alignment pipelines typically combine SFT with RLHF or with reward-free objectives like DPO. The survey further claims that this preference signal is the load-bearing component for handling ambiguous, value-laden, or safety-critical requests, and that remaining weaknesses—reward hacking, jailbreaks, and evaluation gaps—are best understood as failures of the preference signal or its proxies.

Load-bearing premise

The whole synthesis depends on the premise that the cited studies exist and are described accurately; the text uses placeholder-style citation names and gives no complete reference list, so a reader cannot currently verify that the summarized literature is real or correctly represented.

Editorial extensions

If this is right

If SFT is only a first stage, then alignment pipelines should spend their data and compute budget on preference collection rather than on expanding demonstration sets.
If preference-based methods are more flexible, then evaluation of aligned models should measure performance on ambiguous, value-laden prompts, not just on benchmark accuracy.
If DPO-style reward-free objectives match RLHF, then resource-constrained teams can adopt preference alignment without the memory and instability costs of PPO-style RL.
If hierarchical safety-first stacking is the correct design, then models should refuse or redact before optimizing for helpfulness, and deployments should audit that ordering.
If reward misspecification is inherent to single-reward aggregation, then multi-reward, uncertainty-aware, or constitutional approaches will be needed to avoid preference collapse.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper itself does not say this, but the flexibility advantage of preference-based methods should show up as better calibration on rare or novel preference patterns; a benchmark built from intentionally ambiguous prompts could test this directly.
The paper itself does not say this, but as AI-generated feedback scales, value lock-in—systematic bias inherited from the teacher model—will become the binding constraint, making small human spot-checks inside AI feedback loops a natural mitigation.
The paper itself does not say this, but its trade-off analysis implies that front-loading high-quality SFT and then applying DPO with uncertainty-weighted preferences may outperform either method alone on safety-critical domains, and this ordering hypothesis is directly testable.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a wide-ranging survey of LLM alignment, and that is both its strength and its problem. It is explicitly not a novel-results paper, so the test is whether it summarizes the literature faithfully and usefully. On the positive side, it covers a lot of ground: SFT versus RLHF, reward-model taxonomies, PPO variants, GRPO and its recent descendants, domain-specific safety benchmarks, code safety, and even brain-inspired and uncertainty-quantification directions. The equations that appear (SFT cross-entropy, Bradley-Terry, PPO clipped objective, GRPO advantage normalization) match standard treatments. The high-level claim that preference-based methods offer more flexibility than SFT is consistent with the general findings of the cited field. For a newcomer wanting a map of alignment techniques, the organization is genuinely helpful.

The soft spot is load-bearing: the survey cannot be verified in its current form. The provided text has no reference list, and several in-text citations are placeholder-style names that do not resolve to real bibliographic entries. To pick the obvious examples, Section 3.1.1 cites 'Patel and Singh, 2025', 'Chen and Zhao, 2024', and 'Johnson and Kumar, 2024' for concrete numbers like 'as few as twenty carefully chosen queries'; Section 3.1.2 attributes SelfCipher to 'Smith and Doe, 2024'; Section 3.1.3 attributes EasyJailbreak to 'Doe and Bloggs, 2023'. Maybe these entries exist in a fuller version, but from this manuscript a reader cannot tell whether the cited works are real or accurately represented. That is a critical problem for a survey, not a cosmetic one. There are also signs of incomplete editing: typos like 'probal policy optimization' and 'filteres', a truncated line in Section 8.2, and some categories asserted without supporting evidence. Those are minor relative to the citation problem.

Who is this for? A newcomer wanting a structured overview of alignment methods, especially the RLHF/PPO/GRPO families and safety benchmarks, could get value from the organization. But I would not rely on it as a reference and I would not cite it until the citations are fixed. The central synthesis is plausible, but plausibility is not enough for a survey.

Recommendation: desk reject with an invitation to resubmit after replacing every placeholder citation with a real reference and supplying a complete, accurate bibliography. In the current state, peer review would be wasted because reviewers cannot check the claims. If a revised version resolves all in-text citations, I would support sending it out.

Referee Report

4 major / 4 minor

Summary. This manuscript is a survey of large language model alignment and safety. It reviews alignment objectives (helpfulness, harmlessness, honesty), evaluation and benchmarking, supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), comparisons and hybrid training pipelines, direct preference optimization (DPO), group relative policy optimization (GRPO), efficient fine-tuning methods, brain-inspired approaches, alignment uncertainty quantification, societal and regulatory considerations, and strategies used by leading AI labs. The paper's central claim is that SFT enables basic instruction-following while preference-based methods offer more flexibility for aligning with nuanced human intent, and it aims to synthesize the state of practice for researchers and practitioners.

Significance. If the references were complete and accurately represented, this survey would be a useful broad resource for newcomers to LLM alignment, given its wide topical coverage and the effort to organize benchmarks, policy optimization methods, and recent variants such as GRPO. The paper's high-level characterization of SFT versus RLHF/DPO is broadly consistent with the field's consensus. However, the manuscript as provided contains no reference list and multiple unverifiable or placeholder citations, so the survey's factual and attributional claims cannot be checked. Because a survey's reliability rests on faithful citation of prior work, this is a load-bearing defect rather than a presentational one.

major comments (4)

[Entire manuscript (References)] The manuscript contains no References section, so every [Author, Year] citation in the text is unverifiable. For a survey, this is a load-bearing problem: the abstract's central claim that 'preference-based methods offer more flexibility' is a synthesis of cited works, and without a bibliography the reader cannot confirm that the summarized literature exists or is accurately described. A complete reference list must be supplied, and any citation that cannot be resolved to a real published work must be removed or replaced.
[Sections 3.1.1–3.1.3] Concrete quantitative and attributional claims are attached to unverifiable citations: 'as few as twenty carefully chosen queries suffice' is attributed to 'Johnson and Kumar, 2024', SelfCipher is attributed to 'Smith and Doe, 2024', and EasyJailbreak to 'Doe and Bloggs, 2023'. These placeholder-style identifiers do not resolve to the named works, and they appear in precisely the sections where the survey's empirical substance lies. Either real references must be provided for these claims, or the claims should be removed.
[Section 6.2] The subsection 'Overlap for loss function alignment' makes nontrivial theoretical assertions, including that SPIN's update 'asymptotically reduces to SFT' and that RLHF and SFT converge when generation complexity matches verification complexity. These claims are presented with equations but no derivation and rely on unverifiable citations. If the equivalence is known, the survey should cite the specific theorem or paper with enough detail to locate it; otherwise, a proof sketch is needed. As written, the section asserts rather than demonstrates.
[Section 2.3] The 'Hierarchical Policy Stacking' framework is presented as 'one potential method' and supported by citations to Bai et al. (2022a), Glaese et al. (2022), and Solaiman et al. (2021). Those works describe constitutional AI, rule-based safety layers, and content policy classifiers, respectively; they do not appear to propose a unified framework named 'Hierarchical Policy Stacking.' This appears to be an invented taxonomy imposed on the cited works. The paragraph should be rewritten to describe what the sources actually propose, or the framework should be explicitly identified as the authors' own synthesis.

minor comments (4)

[Throughout] There are numerous typographical and formatting issues, including 'F eedback' in several headings, 'probal policy optimization' in Section 7.4, 'significantlt' in Section 8.2, and inconsistent spacing in method names such as 'R TO' and 'T oPPO'. These should be corrected in revision.
[Table 3] The entry for 'LabSafetyBench' is cited as '[Zhou et al.]' without a year, which is inconsistent with the other rows and further complicates verification given the missing reference list.
[Section 3.3.3] The Enron Email Dataset is described as a 'widely used benchmark dataset in privacy alignment,' but the text does not explain how the dataset is turned into a benchmark or cite a specific evaluation suite that uses it. Clarify the protocol or remove the claim.
[Section 1] Bostrom's orthogonality thesis is attributed to a secondary source '[Mulgan, 2016]' rather than to Bostrom's original work. If the intent is to cite the conceptual origin, a primary reference should be used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the survey synthesizes external literature rather than deriving its conclusions from its own inputs.

full rationale

This manuscript is a survey, not a derivation. Its central claim that preference-based methods offer more flexibility than supervised fine-tuning is presented as a synthesis of cited external work (e.g., Ouyang et al., 2022; Bai et al., 2022a; Rafailov et al., 2023), and no step in the paper fits a parameter, constructs a prediction from a subset of data, or defines its target in terms of its own assumptions. Section 6 compares SFT and RLHF by citing external theoretical results, and Section 7 reviews DPO and related methods as prior art rather than as outputs of the survey's own framework. The absence of a reference list and the presence of placeholder-style citations such as 'Patel and Singh, 2025' and 'Smith and Doe, 2024' are serious verifiability and completeness defects, but they are not circularity: the survey does not reduce to its own inputs. No self-citation chain is shown to be load-bearing, and no equation or definition in the paper equates the conclusion with an input by construction. Therefore, on the circularity axis, the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

No free parameters are fitted because the paper proposes no quantitative model. The two axioms above are the main unproved inputs: a particular ethical framing and the assumption of accurate citations.

assumptions (2)

domain assumption The alignment problem is canonically framed as Helpfulness, Harmlessness, Honesty
Section 2 asserts this tripartite framework without deriving it; it is a value-laden framing, not a mathematical fact.
domain assumption The cited literature accurately represents the state of the art
Every synthesis claim assumes these references exist and are faithfully summarized; this is load-bearing for a survey.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges." pith.science (2026). https://pith.science/paper/J7EI3HMN

@misc{pith2026250719672,
  author       = {Pith},
  title        = {Pith review of: Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J7EI3HMN}},
  note         = {Machine review of arXiv:2507.19672}
}

read the original abstract

Due to the remarkable capabilities and growing impact of large language models (LLMs), they have been deeply integrated into many aspects of society. Thus, ensuring their alignment with human values and intentions has emerged as a critical challenge. This survey provides a comprehensive overview of practical alignment techniques, training protocols, and empirical findings in LLM alignment. We analyze the development of alignment methods across diverse paradigms, characterizing the fundamental trade-offs between core alignment objectives. Our analysis shows that while supervised fine-tuning enables basic instruction-following, preference-based methods offer more flexibility for aligning with nuanced human intent. We discuss state-of-the-art techniques, including Direct Preference Optimization (DPO), Constitutional AI, brain-inspired methods, and alignment uncertainty quantification (AUQ), highlighting their approaches to balancing quality and efficiency. We review existing evaluation frameworks and benchmarking datasets, emphasizing limitations such as reward misspecification, distributional robustness, and scalable oversight. We summarize strategies adopted by leading AI labs to illustrate the current state of practice. We conclude by outlining open problems in oversight, value pluralism, robustness, and continuous alignment. This survey aims to inform both researchers and practitioners navigating the evolving landscape of LLM alignment.

Figures

Figures reproduced from arXiv: 2507.19672 by the authors.

**Figure 1.** Example of the jailbreak attack. survey established benchmarks and stress-testing suites that provide standardized datasets, metrics, and evaluation pipelines. Together, these components create a solid foundation for quantifying, comparing, and improving the alignment of large language models. 3.1 Adversarial Attacks & Red-Teaming Adversarial jailbreak research seeks to map and stress-test the full attack surface of… view at source ↗

**Figure 2.** Overview of the RLHF workflow. A pretrained LLM generates responses to prompts, [PITH_FULL_IMAGE:figures/full_fig_p026_2.png] view at source ↗

**Figure 3.** Integration of SFT (Step 1) and RLHF (Step 2 and Step 3). [PITH_FULL_IMAGE:figures/full_fig_p046_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Workflow of the Constitutional AI (CAI) Framework. A helpful LLM first generates [PITH_FULL_IMAGE:figures/full_fig_p049_4.png]

**Figure 5.** Figure 5: Overview of LoRA in LLMs. LoRA introduces trainable low-dimensional weight matrices [PITH_FULL_IMAGE:figures/full_fig_p055_5.png]

**Figure 6.** Figure 6: Two paths to sparsity: (left) data-efficient fine-tuning filters the corpus before training, and (right) parameter-efficient fine-tuning updates only a sparse mask of model weights. We categorized sparse fine-tuning based on the aspects of the sparsity: (1) data-effici…

**Figure 7.** Figure 7: Overview of Knowledge Distillation in LLMs. Knowledge is distilled from a teacher [PITH_FULL_IMAGE:figures/full_fig_p058_7.png]

**Figure 8.** Figure 8: Overview of Adapter in LLMs. Adapters are lightweight, trainable modules inserted into [PITH_FULL_IMAGE:figures/full_fig_p060_8.png]

**Figure 9.** Figure 9: Different Perspectives that Brain-Inspired LLM can help Alignment. [PITH_FULL_IMAGE:figures/full_fig_p062_9.png]

**Figure 10.** Figure 10: Sources of Alignment Uncertainty. prompts, with inter-annotator agreement typically ranging from 0.6 to 0.8 Krippendorff’s alpha [Stiennon et al., 2020, Ziegler et al., 2019], reflecting noisy or subjective supervision. Moreover, human values are inherently diverse, …

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

PersonaFuse: A Personality Activation-Driven Framework for Enhancing Human-LLM Interactions
cs.CL 2025-09 reject novelty 5.0 of 10

A post-training framework with persona-specific LoRA experts and a situation-aware router improves LLM emotional responses, but the evidence on preserving general ability is undercut by missing base-model comparisons.

Reference graph

Works this paper leans on

294 extracted references · 9 canonical work pages · cited by 1 Pith paper

[1]

Superintelligence: Paths, dangers, strategies, 2016

Tim Mulgan. Superintelligence: Paths, dangers, strategies, 2016

2016
[2]

Artificial intelligence as a positive and negative factor in global risk

Eliezer Yudkowsky et al. Artificial intelligence as a positive and negative factor in global risk. Global catastrophic risks, 1 0 (303): 0 184, 2008

2008
[3]

Concrete problems in ai safety

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man \'e . Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016

arXiv 2016
[4]

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

2017
[5]

Bert: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171--4186, 2019

2019
[6]

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

2019
[7]

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

1901
[8]

Emergent abilities of large language models

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

arXiv 2022

Show all 294 references

[9]

Scaling laws for neural language models

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

2001 arXiv
[10]

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

2022
[11]

Constitutional ai: Harmlessness from ai feedback

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022 a . doi:10.48550/arXiv.2212.08073. URL https://doi.org/10.48550/arXiv.2212.08073

2022 doi arXiv
[12]

Direct preference optimization: Your language model is secretly a reward model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36: 0 53728--53741, 2023

2023
[13]

A mathematical framework for transformer circuits

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1 0 (1): 0 12, 2021

2021
[14]

Supervising strong learners by amplifying weak experts

Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575, 2018

2018 arXiv
[15]

A general language assistant as a laboratory for alignment, December 2021

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, and et al. A general language assistant as a laboratory for alignment, December 2021. URL https://arxiv.org/abs/2112.00861

2021 arXiv
[16]

Improving alignment of dialogue agents via targeted human judgements

Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig...

2022 doi arXiv
[17]

Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned

Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.0785...

2022 arXiv
[18]

ToxiGen : A large-scale machine-generated dataset for adversarial and implicit hate speech detection

Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. ToxiGen : A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3463-...

2022
[19]

Large language models in medicine

Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29 0 (8): 0 1930--1940, 2023

1930
[20]

Large legal fictions: Profiling legal hallucinations in large language models

Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E Ho. Large legal fictions: Profiling legal hallucinations in large language models. Journal of Legal Analysis, 16 0 (1): 0 64--93, 2024

2024
[21]

Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. Do users write more insecure code with ai assistants? In Proceedings of the 21st Workshop on Programming Languages and Software Engineering, PLAS '22, pages 69--78, New York, NY, USA, 2022. Association for Computing Ma...

2022
[22]

Autodan: Generating stealthy jailbreak prompts on aligned llms

Yue Liu, Miao Xiong, Jinlan Fu, Shumin Deng, and Bryan Hooi. Autodan: Generating stealthy jailbreak prompts on aligned llms. In International Conference on Learning Representations, 2024 a

2024
[23]

Proximal policy optimization algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

2017 arXiv
[24]

Training a helpful and harmless assistant with rlhf

Yuntao Bai, Saurav Kadavath, Amanda Askell, et al. Training a helpful and harmless assistant with rlhf. arXiv preprint arXiv:2204.05862, 2022 b

2022 arXiv
[26]

Gpt-4o system card

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

2024 arXiv
[27]

u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing...

2020
[28]

Mathematical discoveries from program search with large language models

Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625 0 (7...

2024
[29]

Introduction to llm agents

NVIDIA. Introduction to llm agents. https://developer.nvidia.com/blog/introduction-to-llm-agents/, February 2024

2024
[30]

Chemcrow: Augmenting large-language models with chemistry tools

Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXiv:2304.05376, 2023

2023 arXiv
[31]

Man ends his life after an ai chatbot ‘encouraged’him to sacrifice himself to stop climate change

Imane El Atillah. Man ends his life after an ai chatbot ‘encouraged’him to sacrifice himself to stop climate change. euronews. com, 2023

2023
[32]

Whoops, samsung workers accidentally leaked trade secrets via chatgpt

Cecily Mauran. Whoops, samsung workers accidentally leaked trade secrets via chatgpt. Mashable [online]. Dostupn \'e z: https://mashable. com/article/samsungchatgpt-leak-details , 2023

2023
[33]

Large language models in finance: A survey

Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. Large language models in finance: A survey. In Proceedings of the fourth ACM international conference on AI in finance, pages 374--382, 2023 a

2023
[34]

A survey on the honesty of large language models

Siheng Li, Cheng Yang, Taiqiang Wu, Chufan Shi, Yuji Zhang, Xinyu Zhu, Zesen Cheng, Deng Cai, Mo Yu, Lemao Liu, et al. A survey on the honesty of large language models. arXiv preprint arXiv:2409.18786, 2024 a

2024 arXiv
[35]

Process for adapting language models to society

Irene Solaiman, Sarah Dennison, Deep Ganguli, and et al. Process for adapting language models to society. arXiv preprint arXiv:2106.10328, 2021. doi:10.48550/arXiv.2106.10328. URL https://arxiv.org/abs/2106.10328

2021 doi arXiv
[36]

A lifelong agent for strategy self‐exploration to jailbreak llms (autodan‐turbo)

Liang Zhang, Ying Wu, Ming Li, and Hao Chen. A lifelong agent for strategy self‐exploration to jailbreak llms (autodan‐turbo). In International Conference on Learning Representations, 2025 a

2025
[37]

Jailbreaking leading safety‐aligned llms with simple adaptive attacks

Rohan Patel and Ananya Singh. Jailbreaking leading safety‐aligned llms with simple adaptive attacks. In International Conference on Learning Representations, 2025

2025
[38]

Cognitive overload: Jailbreaking large language models with overloaded logical thinking

Ming Chen and Li Zhao. Cognitive overload: Jailbreaking large language models with overloaded logical thinking. In NAACL, 2024

2024
[39]

How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms

Yue Liu and Jiaheng Zhang. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. In EMNLP, 2024

2024
[40]

Jailbreaking black‐box large language models in twenty queries

Emily Johnson and Raj Kumar. Jailbreaking black‐box large language models in twenty queries. In NeurIPS, 2024

2024
[41]

Gpt‐4 is too smart to be safe: Stealthy chat with llms via cipher (selfcipher)

John Smith and Jane Doe. Gpt‐4 is too smart to be safe: Stealthy chat with llms via cipher (selfcipher). In International Conference on Learning Representations, 2024

2024
[42]

Multilingual jailbreak challenges in large language models

Jong Lee and Soo Kim. Multilingual jailbreak challenges in large language models. In International Conference on Learning Representations, 2024 a

2024
[43]

Artprompt: Ascii‐art‐based jailbreak attacks

Wei Jiang, Mei Chen, and Zixi Sun. Artprompt: Ascii‐art‐based jailbreak attacks. In Proceedings of the 2024 Association for Computational Linguistics (ACL), 2024

2024
[44]

Smiles-prompting: A novel approach to llm jailbreak attacks in chemical synthesis, 2024

Aidan Wong, He Cao, Zijing Liu, and Yu Li. Smiles-prompting: A novel approach to llm jailbreak attacks in chemical synthesis, 2024. URL https://arxiv.org/abs/2410.15641

2024 arXiv
[45]

Dynamic guided and domain applicable safeguards for enhanced security in large language models

Weidi Luo, He Cao, Zijing Liu, Yu Wang, Aidan Wong, Bin Feng, Yuan Yao, and Yu Li. Dynamic guided and domain applicable safeguards for enhanced security in large language models. In Proceedings of the AAAI Conference on Artificial Intelligence , 2025 a

2025
[46]

Do anything now: Characterizing and evaluating in‐the‐wild jailbreak prompts on large language models (dan)

Yue Liu and Hongcheng Gao. Do anything now: Characterizing and evaluating in‐the‐wild jailbreak prompts on large language models (dan). In ACM Conference on Computer and Communications Security, 2024

2024
[47]

Make them spill the beans! coercive knowledge extraction from (production) llms

Ming Li, Hao Chen, and Jie Zhang. Make them spill the beans! coercive knowledge extraction from (production) llms. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023 b

2023
[48]

Ignore this title and hackaprompt: Exposing systemic vulnerabilities of llms through a global-scale prompt hacking competition

Kai Wang, Yu Zhao, and Wei Liu. Ignore this title and hackaprompt: Exposing systemic vulnerabilities of llms through a global-scale prompt hacking competition. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023 a

2023
[49]

Summon a demon and bind it: A grounded theory of llm red-teaming in the wild

Alice Brown, John Smith, and Emily Davis. Summon a demon and bind it: A grounded theory of llm red-teaming in the wild. In USENIX Security Symposium, 2023

2023
[50]

Easyjailbreak: A unified framework for jailbreaking large language models

Jane Doe and Joe Bloggs. Easyjailbreak: A unified framework for jailbreaking large language models. arXiv preprint arXiv:2309.01234, 2023

2023 arXiv
[51]

Tricking llms into disobedience: Formalizing, analyzing, and detecting jailbreaks

Sung Lee and Hyun Kim. Tricking llms into disobedience: Formalizing, analyzing, and detecting jailbreaks. In Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC), 2024 b

2024
[52]

Sleeper agents: Training deceptive llms that persist through safety training

Evan Hubinger, Christopher Denison, Jing Mu, et al. Sleeper agents: Training deceptive llms that persist through safety training. arXiv preprint arXiv:2401.05566, 2024

2024 arXiv
[53]

Fake alignment: Are llms really aligned well? arXiv preprint arXiv:2311.05915, 2023 b

Yilun Wang, Yixin Teng, Kai Huang, et al. Fake alignment: Are llms really aligned well? arXiv preprint arXiv:2311.05915, 2023 b

2023 arXiv
[54]

Competition report: Finding universal jailbreak backdoors in aligned llms

Javier Rando, Francesco Croce, Kryštof Mitka, Stepan Shabalin, Maksym Andriushchenko, Nicolas Flammarion, and Florian Tram \`e r. Competition report: Finding universal jailbreak backdoors in aligned llms. arXiv preprint arXiv:2404.14461, 2024 a

2024 arXiv
[55]

Alignment faking in large language models

Rebecca Greenblatt, Christopher Denison, Benjamin Wright, et al. Alignment faking in large language models. arXiv preprint arXiv:2412.14093, 2024

2024 arXiv
[56]

Rlhf trojan competition: Finding trojans in aligned language models

Javier Rando and Florian Tram \`e r. Rlhf trojan competition: Finding trojans in aligned language models. GitHub repository: https://github.com/ethz-spylab/rlhf_trojan_competition, 2024

2024
[57]

Tdc 2023 (llm edition): The trojan detection challenge

Mantas Mazeika, Andy Zou, Norman Mu, Long Phan, Zifan Wang, Chunru Yu, Adam Khoja, Dan Hendrycks, Radha Poovendran, Bo Li, and David Forsyth. Tdc 2023 (llm edition): The trojan detection challenge. NeurIPS Competition, 2023

2023
[58]

Trojan detection in large language models: Insights from the trojan detection challenge

Mantas Mazeika, Andy Zou, Norman Mu, Long Phan, Zifan Wang, Chunru Yu, Adam Khoja, et al. Trojan detection in large language models: Insights from the trojan detection challenge. arXiv preprint arXiv:2404.13660, 2024 a

2024 arXiv
[59]

Clas 2024: The competition for llm and agent safety

Zhen Xiang, Mintong Kang Yi Zeng, Chejian Xu, Jiawei Zhang, Zhuowen Yuan, Zhaorun Chen, Chulin Xie, Fengqing Jiang, Minzhou Pan, Junyuan Hong, Ruoxi Jia, Radha Poovendran, and Bo Li. Clas 2024: The competition for llm and agent safety. NeurIPS 2024 Competition Track, 2024

2024
[60]

Dataset and lessons learned from the 2024 satml llm capture-the-flag competition

Edoardo Debenedetti, Daniel Paleka, Ahmed Salem, et al. Dataset and lessons learned from the 2024 satml llm capture-the-flag competition. arXiv preprint arXiv:2406.07954, 2024 a

2024 arXiv
[61]

Llm evaluations hackathon: Trojan detection challenge

Paul Christiano et al. Llm evaluations hackathon: Trojan detection challenge. Alignment Jam hackathon page: https://alignmentjam.com/jam/evals, 2023 a

2023
[62]

Our competitions at ieee satml 2024: Llm ctf and trojan detection

Javier Rando, Edoardo Debenedetti, Daniel Paleka, and Florian Tram \`e r. Our competitions at ieee satml 2024: Llm ctf and trojan detection. SPY Lab Blog: https://spylab.ai/blog/results-competition/, 2024 b

2024
[63]

Flipattack: Jailbreak llms via flipping

Yue Liu, Xiaoxin He, Miao Xiong, Jinlan Fu, Shumin Deng, and Bryan Hooi. Flipattack: Jailbreak llms via flipping. arXiv preprint arXiv:2410.02832, 2024 b

2024 arXiv
[64]

Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast

Xiangming Gu, Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Ye Wang, Jing Jiang, and Min Lin. Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast. arXiv preprint arXiv:2402.08567, 2024

2024 arXiv
[65]

Gptscore: Evaluate as you desire

Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. In NAACL, 2023 a

2023
[66]

G-eval: Nlg evaluation using gpt-4 with better human alignment

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. In EMNLP, 2023 a

2023
[67]

Llms as narcissistic evaluators: When ego inflates evaluation scores

Yiqi Liu, Nafise Sadat Moosavi, and Chenghua Lin. Llms as narcissistic evaluators: When ego inflates evaluation scores. arXiv preprint arXiv:2311.09766, 2023 b

2023 arXiv
[68]

Large language models are not fair evaluators

Peiyi Wang, Lei Li, Liang Chen, et al. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023 c

2023 arXiv
[69]

Dhp benchmark: Are llms good nlg evaluators? In Findings of NAACL, 2024 a

Yicheng Wang, Jiayi Yuan, Yu-Neng Chuang, et al. Dhp benchmark: Are llms good nlg evaluators? In Findings of NAACL, 2024 a

2024
[70]

Judgelm: Fine-tuned large language models are scalable judges

Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.12345, 2023 a

2023 arXiv
[71]

Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization

Yidong Wang, Zhuohao Yu, Zhengran Zeng, et al. Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. arXiv preprint arXiv:2311.06789, 2023 d

2023 arXiv
[72]

Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations

Yen-Ting Lin and Yun-Nung Chen. Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations. In NLP4ConvAI Workshop, 2023

2023
[73]

Clair: Evaluating image captions with large language models

David Chan, Suzanne Petryk, Joseph Gonzalez, et al. Clair: Evaluating image captions with large language models. In EMNLP, 2023

2023
[74]

Fleur: An explainable reference-free evaluation metric for image captioning using a large multimodal model

Yebin Lee, Imseong Park, and Myungjoo Kang. Fleur: An explainable reference-free evaluation metric for image captioning using a large multimodal model. In ACL, 2024 a

2024
[75]

Zico Kolter, and Matt Fredrikson

Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Advbench: Universal and transferable adversarial attacks on aligned language models, Jul 2023. URL https://arxiv.org/abs/2307.15043

2023 arXiv
[76]

Salad-bench: A hierarchical and comprehensive safety benchmark for large language models, February 2024 b

Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models, February 2024 b . URL https://arxiv.org/abs/2402.05044

2024 arXiv
[77]

Safetybench: Evaluating the safety of large language models with multiple choice questions

Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. Safetybench: Evaluating the safety of large language models with multiple choice questions. Association for Computational Linguistics (ACL), 2023 a

2023
[78]

Cold: A benchmark for chinese offensive language detection, January 2022

Jiawen Deng, Jingyan Zhou, Hao Sun, Chujie Zheng, Fei Mi, Helen Meng, and Minlie Huang. Cold: A benchmark for chinese offensive language detection, January 2022. URL https://arxiv.org/abs/2201.06025

2022 arXiv
[79]

Beavertails: Towards improved safety alignment of llm via a human-preference dataset, 2024

Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset, 2024. URL https://aclanthology.org/2024.emnlp-main.xxx

2024
[80]

Sehwag, Kai Huang, Lei He, Bashuan Wei, Dazheng Li, and Yubo Sheng

Tianyang Xie, Xiao Qi, Yi Zeng, Yong Huang, Upendra M. Sehwag, Kai Huang, Lei He, Bashuan Wei, Dazheng Li, and Yubo Sheng. Sorry‐bench: Systematically evaluating large language model safety refusal behaviors, Jun 2024. URL https://arxiv.org/abs/2406.14598

2024 arXiv
[81]

Raparthy, Andre Lupu, Emily Hambro, and Jakob Foerster

Mario Samvelyan, Sachit C. Raparthy, Andre Lupu, Emily Hambro, and Jakob Foerster. Rainbow teaming: Open‐ended generation of diverse adversarial prompts, Feb 2024. URL https://arxiv.org/abs/2402.16822

2024 arXiv
[82]

Cosafe: Evaluating large language model safety in multi-turn dialogue coreference, June 2024 a

Erxin Yu, Jing Li, Ming Liao, Siqi Wang, Zuchen Gao, Fei Mi, and Lanqing Hong. Cosafe: Evaluating large language model safety in multi-turn dialogue coreference, June 2024 a . URL https://arxiv.org/abs/2406.17626

2024 arXiv
[83]

Sc-safety: A multi-round open-ended question adversarial safety benchmark for large language models in chinese, October 2023 a

Liang Xu, Kangkang Zhao, Lei Zhu, and Hang Xue. Sc-safety: A multi-round open-ended question adversarial safety benchmark for large language models in chinese, October 2023 a . URL https://arxiv.org/abs/2310.05818

2023 arXiv
[84]

Promptbench: A unified library for evaluation of large language models, December 2023 b

Kaijie Zhu, Qinlin Zhao, Hao Chen, Jindong Wang, and Xing Xie. Promptbench: A unified library for evaluation of large language models, December 2023 b . URL https://arxiv.org/abs/2312.07910

2023 arXiv
[85]

Safechain: Safety of language models with long chain-of-thought reasoning capabilities, 2025 a

Fengqing Jiang, Zhangchen Xu, Yuetai Li, Luyao Niu, Zhen Xiang, Bo Li, Bill Yuchen Lin, and Radha Poovendran. Safechain: Safety of language models with long chain-of-thought reasoning capabilities, 2025 a . URL https://arxiv.org/abs/2502.12025

2025 arXiv
[87]

Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, ...

2025 arXiv
[88]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

2024 arXiv
[89]

Qwen2.5 technical report, 2025

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

2025 arXiv
[90]

The hidden risks of large reasoning models: A safety assessment of r1, 2025 a

Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Shreedhar Jangam, Jayanth Srinivasa, Gaowen Liu, Dawn Song, and Xin Eric Wang. The hidden risks of large reasoning models: A safety assessment of r1, 2025 a . URL https://arxiv.org/abs/2502.12659

2025
[91]

Bartoldson, Bhavya Kailkhura, and Cihang Xie

Zijun Wang, Haoqin Tu, Yuhan Wang, Juncheng Wu, Jieru Mei, Brian R. Bartoldson, Bhavya Kailkhura, and Cihang Xie. Star-1: Safer alignment of reasoning llms with 1k data, 2025 a . URL https://arxiv.org/abs/2504.01903

2025
[92]

Saro: Enhancing llm safety through reasoning-based alignment, 2025

Yutao Mou, Yuxiao Luo, Shikun Zhang, and Wei Ye. Saro: Enhancing llm safety through reasoning-based alignment, 2025. URL https://arxiv.org/abs/2504.09420

2025 arXiv
[93]

Decodingtrust: A comprehensive assessment of trustworthiness in gpt models

Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. In NeurIPS, 2023 e

2023
[94]

Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric Xing, Furong H...

2024 arXiv
[95]

Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. The prism alignment dataset: What participatory, representative and individualised human fee...

2024 arXiv
[96]

Learning fair representations

Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, page...

2013
[97]

Equality of opportunity in supervised learning, 2016

Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning, 2016. URL https://arxiv.org/abs/1610.02413

2016 arXiv
[98]

Behonest: Benchmarking honesty in large language models

Steffi Chern, Zhulin Hu, Yuqing Yang, Ethan Chern, Yuan Guo, Jiahe Jin, Binjie Wang, and Pengfei Liu. Behonest: Benchmarking honesty in large language models. arXiv preprint arXiv:2406.13261, 2024

2024 arXiv
[99]

Safeagentbench: A benchmark for safe task planning of embodied llm agents

Sheng Yin, Xianghe Pang, Yuanzhuo Ding, Menglan Chen, Yutong Bi, Yichen Xiong, Wenhao Huang, Zhen Xiang, Jing Shao, and Siheng Chen. Safeagentbench: A benchmark for safe task planning of embodied llm agents. arXiv preprint arXiv:2412.13178, 2024 a . doi:10.48550/arXiv.2412.131...

2024 doi
[100]

Agent‐safetybench: Evaluating the safety of llm agents

Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. Agent‐safetybench: Evaluating the safety of llm agents. arXiv preprint arXiv:2412.14470, 2024 a . doi:10.48550/arXiv.2412.14470. URL https://doi.org/10.48550/arXiv.2412.14470

2024 doi arXiv
[101]

St‐webagentbench: A benchmark for evaluating safety and trustworthiness in web agents

Ido Levy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yaeli, and Segev Shlomov. St‐webagentbench: A benchmark for evaluating safety and trustworthiness in web agents. arXiv preprint arXiv:2410.06703, 2024. doi:10.48550/arXiv.2410.06703. URL https://doi.org/10.48550/arXiv.2410.06703

2024 doi arXiv
[102]

Earbench: Towards evaluating physical risk awareness for task planning of foundation model‐based embodied ai agents

Zihao Zhu, Bingzhe Wu, Zhengyou Zhang, Lei Han, Qingshan Liu, and Baoyuan Wu. Earbench: Towards evaluating physical risk awareness for task planning of foundation model‐based embodied ai agents. arXiv preprint arXiv:2408.04449, 2024 a . doi:10.48550/arXiv.2408.04449. URL https...

2024 doi arXiv
[103]

Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents

Hanrong Zhang, Qian Li, Roshni Patel, and Ming Chen. Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents. arXiv preprint arXiv:2410.02644, 2024 b

2024 arXiv
[104]

Agentdojo: A dynamic environment to evaluate attacks and defenses for llm agents

Edoardo Debenedetti, Clara Rossi, and Tuan Nguyen. Agentdojo: A dynamic environment to evaluate attacks and defenses for llm agents. arXiv preprint arXiv:2406.13352, 2024 b

2024 arXiv
[105]

Guardagent: Safeguard llm agents by a guard agent via knowledge-enabled reasoning

Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, Dawn Song, and Bo Li. Guardagent: Safeguard llm agents by a guard agent via knowledge-enabled reasoning. In Proceedings of the 42nd International Conference ...

2025
[106]

Agrail: A lifelong agent guardrail with effective and adaptive safety detection

Weidi Luo, Ananya Singh, Luis Gomez, and Yuxin Tan. Agrail: A lifelong agent guardrail with effective and adaptive safety detection. arXiv preprint arXiv:2502.11448, 2025 b

2025 arXiv
[107]

Shieldagent: Shielding agents via verifiable safety policy reasoning

Zhaorun Chen, Mintong Kang, and Bo Li. Shieldagent: Shielding agents via verifiable safety policy reasoning. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025 a

2025
[108]

Healthbench: Evaluating large language models towards improved human health

Rahul K Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Qui \ n onero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, et al. Healthbench: Evaluating large language models towards improved human health. arXiv preprint arXi...

2025 arXiv
[109]

Chemsafetybench: Benchmarking llm safety on chemistry domain

Haochen Zhao, Xiangru Tang, Ziran Yang, Xiao Han, Xuanzhi Feng, Yueqing Fan, Senhao Cheng, Di Jin, Yilun Zhao, Arman Cohan, et al. Chemsafetybench: Benchmarking llm safety on chemistry domain. arXiv preprint arXiv:2411.16736, 2024

2024 arXiv
[110]

The wmdp benchmark: Measuring and reducing malicious use with unlearning

Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218, 2024 c

2024 arXiv
[111]

Medsafetybench: Evaluating and improving the medical safety of large language models

Tessa Han, Aounon Kumar, Chirag Agarwal, and Himabindu Lakkaraju. Medsafetybench: Evaluating and improving the medical safety of large language models. arXiv preprint arXiv:2403.03744, 2024 a

2024 arXiv
[112]

Benchmarking llms on safety issues in scientific labs

Yujun Zhou, Jingdong Yang, Kehan Guo, Pin-Yu Chen, Tian Gao, Werner Geyer, Nuno Moniz, Nitesh V Chawla, and Xiangliang Zhang. Benchmarking llms on safety issues in scientific labs
[113]

Physreason: A comprehensive benchmark towards physics-based reasoning

Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang, and Jun Liu. Physreason: A comprehensive benchmark towards physics-based reasoning. arXiv preprint arXiv:2502.12054, 2025 b

2025 arXiv
[114]

Scisafeeval: a comprehensive benchmark for safety alignment of large language models in scientific tasks

Tianhao Li, Jingyu Lu, Chuangxin Chu, Tianyu Zeng, Yujia Zheng, Mei Li, Haotian Huang, Bin Wu, Zuoxian Liu, Kai Ma, et al. Scisafeeval: a comprehensive benchmark for safety alignment of large language models in scientific tasks. arXiv preprint arXiv:2410.03769, 2024 d

arXiv
[115]

Control risk for potential misuse of artificial intelligence in science

Jiyan He, Weitao Feng, Yaosen Min, Jingwei Yi, Kunsheng Tang, Shuai Li, Jie Zhang, Kejiang Chen, Wenbo Zhou, Xing Xie, et al. Control risk for potential misuse of artificial intelligence in science. arXiv preprint arXiv:2312.06632, 2023

2023 arXiv
[116]

Sos bench: Benchmarking safety alignment on scientific knowledge, 2025 b

Fengqing Jiang, Fengbo Ma, Zhangchen Xu, Yuetai Li, Bhaskar Ramasubramanian, Luyao Niu, Bo Li, Xianyan Chen, Zhen Xiang, and Radha Poovendran. Sos bench: Benchmarking safety alignment on scientific knowledge, 2025 b . URL https://arxiv.org/abs/2505.21605

2025 arXiv
[117]

Harmbench: A standardized evaluation framework for automated red teaming and robust refusal

Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. International Conference on...

2024
[118]

Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong

Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. Jailbreakbench: An open robustness benchmark for jailbreaking large la...

2024
[119]

Jailbreakv-28k: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks

Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao. Jailbreakv-28k: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks. Conference on Language Modeling (COLM), 2024

2024
[120]

Codelmsec benchmark: Systematically evaluating and finding security vulnerabilities in black-box code language models, 2023

Hossein Hajipour, Keno Hassler, Thorsten Holz, Lea Schönherr, and Mario Fritz. Codelmsec benchmark: Systematically evaluating and finding security vulnerabilities in black-box code language models, 2023. URL https://arxiv.org/abs/2302.04012

2023 arXiv
[121]

Redcode: Risky code execution and generation benchmark for code agents

Chengquan Guo, Xun Liu, Chulin Xie, Andy Zhou, Yi Zeng, Zinan Lin, Dawn Song, and Bo Li. Redcode: Risky code execution and generation benchmark for code agents. Thirty-Eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024 a

2024
[122]

Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models, 2024

Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, David Molnar, Spencer Whitman, and Joshua Saxe. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large langua...

2024 arXiv
[123]

Self-instruct: Aligning language models with self-generated instructions

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022 a

2022 arXiv
[124]

Stanford alpaca: An instruction-following llama model, 2023

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023

2023
[125]

Instruction tuning with gpt-4

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023 a

2023 arXiv
[126]

Scaling instruction-finetuned language models

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25 0 (70): 0 1--53, 2024

2024
[128]

Lamda: Language models for dialog applications

Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022

2022 arXiv
[129]

The flan collection: Designing data and methods for effective instruction tuning

Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pages 22631--22648. P...

2023
[130]

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

2020
[131]

o pf, Yannic Kilcher, Dimitri Von R \

Andreas K \"o pf, Yannic Kilcher, Dimitri Von R \"u tte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Rich \'a rd Nagyfi, et al. Openassistant conversations-democratizing large language model alignment. Advances in Neural Info...

2023
[132]

Gonzalez, Ion Stoica, and Eric P

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90\ URL https://lmsys.org/blog/2023-03-30-vicuna/

2023
[133]

From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning

Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308.12032, 2023 c

2023 arXiv
[134]

Cross-task generalization via natural language crowdsourcing instructions

Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.08773, 2021

2021 arXiv
[135]

A survey of multilingual large language models

Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. A survey of multilingual large language models. Patterns, 6 0 (1), 2025

2025
[136]

Judging llm-as-a-judge with mt-bench and chatbot arena

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 0 46595--46623, 2023 a

2023
[137]

Bias and fairness in large language models: A survey

Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, 50 0 (3): 0 1097--1179, 2024

2024
[138]

Diversity measurement and subset selection for instruction tuning datasets

Peiqi Wang, Yikang Shen, Zhen Guo, Matthew Stallone, Yoon Kim, Polina Golland, and Rameswar Panda. Diversity measurement and subset selection for instruction tuning datasets. arXiv preprint arXiv:2402.02318, 2024 b

2024 arXiv
[139]

Beyond scale: The diversity coefficient as a data quality metric for variability in natural language data

Brando Miranda, Alycia Lee, Sudharsan Sundar, Allison Casasola, and Sanmi Koyejo. Beyond scale: The diversity coefficient as a data quality metric for variability in natural language data. arXiv preprint arXiv:2306.13840, 2023

2023 arXiv
[140]

Lora: Low-rank adaptation of large language models

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1 0 (2): 0 3, 2022

2022
[141]

Parameter-efficient transfer learning for nlp

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790--2799. PMLR, 2019

2019
[142]

The power of scale for parameter-efficient prompt tuning

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021

2021 arXiv
[143]

Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping

Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. In arXiv preprint arXiv:2002.06305, 2020

2002 arXiv
[144]

Becoming self-instruct: introducing early stopping criteria for minimal instruct tuning

Waseem AlShikh, Manhal Daaboul, Kirk Goddard, Brock Imel, Kiran Kamble, Parikshith Kulkarni, and Melisa Russak. Becoming self-instruct: introducing early stopping criteria for minimal instruct tuning. arXiv preprint arXiv:2307.03692, 2023

2023 arXiv
[145]

An empirical study on hyperparameter optimization for fine-tuning pre-trained language models

Xueqing Liu and Chi Wang. An empirical study on hyperparameter optimization for fine-tuning pre-trained language models. arXiv preprint arXiv:2106.09204, 2021

2021 arXiv
[146]

Supervised contrastive learning for pre-trained language model fine-tuning

Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. Supervised contrastive learning for pre-trained language model fine-tuning. arXiv preprint arXiv:2011.01403, 2020

2011 arXiv
[147]

Supervised contrastive learning as multi-objective optimization for fine-tuning large pre-trained language models

Youness Moukafih, Mounir Ghogho, and Kamel Smaili. Supervised contrastive learning as multi-objective optimization for fine-tuning large pre-trained language models. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-...

2023
[148]

Self-play fine-tuning converts weak language models to strong language models

Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024 a

2024 arXiv
[149]

Asft: Aligned supervised fine-tuning through absolute likelihood

Ruoyu Wang, Jiachen Sun, Shaowei Hua, and Quan Fang. Asft: Aligned supervised fine-tuning through absolute likelihood. arXiv preprint arXiv:2409.10571, 2024 c

2024 arXiv
[150]

Preference-oriented supervised fine-tuning: Favoring target model over aligned large language models

Yuchen Fan, Yuzhong Hong, Qiushi Wang, Junwei Bao, Hongfei Jiang, and Yang Song. Preference-oriented supervised fine-tuning: Favoring target model over aligned large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23859--2386...

2025
[151]

Ethical and social risks of harm from language models

Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021

2021 arXiv
[152]

Preference fine-tuning of llms should leverage suboptimal, on-policy data

Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of llms should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404.14367, 2024

2024 arXiv
[153]

Brown, Miljan Martic, Shane Legg, and Dario Amodei

Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. arXiv preprint arXiv:1706.03741, 2023 b . doi:10.48550/arXiv.1706.03741. URL https://doi.org/10.48550/arXiv.1706.03741

2023 doi arXiv
[154]

Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano

Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. arXiv preprint arXiv:2009.01325, 2022. doi:10.48550/arXiv.2009.01325. URL https://doi.org/10.48550/arXi...

2009 doi arXiv
[155]

Training a helpful and harmless assistant with reinforcement learning from human feedback

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, April 2022 c . doi:10.48550/arXiv.2204.05862. URL http...

2022 doi arXiv
[156]

Banghua Zhu, Jiantao Jiao, and Michael I. Jordan. Principled reinforcement learning with human feedback from pairwise or \ k\ -wise comparisons. arXiv preprint arXiv:2301.11270, 2024 b . doi:10.48550/arXiv.2301.11270. URL https://doi.org/10.48550/arXiv.2301.11270

2024 doi arXiv
[158]

Bakker, Martin J

Michiel A. Bakker, Martin J. Chadwick, Hannah R. Sheahan, Michael Henry Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, et al. Fine-tuning language models to find agreement among humans with diverse preferences. arXiv preprint arXiv:2211.15006, 2022. doi:10.4855...

2022 doi arXiv
[159]

Miller, Sumit Chopra, Marc’Aurelio Ranzato, and Jason Weston

Jiwei Li, Alexander H. Miller, Sumit Chopra, Marc’Aurelio Ranzato, and Jason Weston. Dialogue learning with human-in-the-loop. arXiv preprint arXiv:1611.09823, 2017. doi:10.48550/arXiv.1611.09823. URL https://doi.org/10.48550/arXiv.1611.09823

2017 doi arXiv
[160]

FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback

Baicen Xiao, Qifan Lu, Bhaskar Ramasubramanian, Andrew Clark, Linda Bushnell, and Radha Poovendran. FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback . In Proceedings of the International Conference on Autonomous Agents and Multiagent Syst...

2020
[161]

Training language models with language feedback at scale

Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback at scale. arXiv preprint arXiv:2303.16755, 2024. doi:10.48550/arXiv.2303.16755. URL https://doi.org/10.48550/arXiv.2...

2024 doi arXiv
[162]

Gan-based interactive reinforcement learning from demonstration and human evaluative feedback

Jie Huang, Jiangshan Hao, Rongshun Juan, Randy Gomez, Keisuke Nakamura, and Guangliang Li. Gan-based interactive reinforcement learning from demonstration and human evaluative feedback. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2023...

2023
[163]

Continually improving extractive qa via human feedback

Ge Gao, Hung-Ting Chen, Yoav Artzi, and Eunsol Choi. Continually improving extractive qa via human feedback. arXiv preprint arXiv:2305.12473, 2023 a . doi:10.48550/arXiv.2305.12473. URL https://doi.org/10.48550/arXiv.2305.12473

2023 doi arXiv
[164]

Lam, Joey Hejna, Yijia Shao, Hyundong Cho, Michael S

Omar Shaikh, Michelle S. Lam, Joey Hejna, Yijia Shao, Hyundong Cho, Michael S. Bernstein, and Diyi Yang. Aligning language models with demonstrated feedback. arXiv preprint arXiv:2406.00888, 2025. doi:10.48550/arXiv.2406.00888. URL https://doi.org/10.48550/arXiv.2406.00888

2025 doi arXiv
[165]

Jason Ross Brown, Carl Henrik Ek, and Robert D. Mullins. Learning from preferences and mixed demonstrations in general settings. https://openreview.net/forum?id=Sfct4aXXcw, 2025

2025
[166]

Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. Learning from dialogue after deployment: Feed yourself, chatbot! In Anna Korhonen, David Traum, and Llu \'i s M \`a rquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computati...

2019 doi
[167]

Bowman, Kyunghyun Cho, and Ethan Perez

Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel R. Bowman, Kyunghyun Cho, and Ethan Perez. Improving code generation by training with natural language feedback. arXiv preprint arXiv:2303.16749, 2024 b . doi:10.48550/arXiv.2303.16749. URL...

2024 doi arXiv
[168]

Using interactive feedback to improve the accuracy and explainability of question answering systems post-deployment

Zichao Li, Prakhar Sharma, Xing Han Lu, Jackie Chi Kit Cheung, and Siva Reddy. Using interactive feedback to improve the accuracy and explainability of question answering systems post-deployment. arXiv preprint arXiv:2204.03025, 2022 a . URL https://api.semanticscholar.org/Cor...

2022 arXiv
[169]

Reward learning from multiple feedback types

Yannick Metz, Andras Geiszl, Raphael Baur, and Mennatallah El-Assady. Reward learning from multiple feedback types. https://arxiv.org/abs/2502.21038, 2025. Preprint

2025 arXiv
[170]

Segmenting text and learning their rewards for improved RLHF in language models, 2025

Yueqin Yin, Shentao Yang, Yujia Xie, Ziyi Yang, Yuting Sun, Hany Hassan Awadalla, Weizhu Chen, and Mingyuan Zhou. Segmenting text and learning their rewards for improved RLHF in language models, 2025. URL https://openreview.net/forum?id=cK7yrw5g5Q

2025
[171]

Smith, Mari Ostendorf, and Hannaneh Hajishirzi

Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. arXiv, oct 2023

2023
[172]

Aligning large language models via fine-grained supervision

Dehong Xu, Liang Qiu, Minseok Kim, Faisal Ladhak, and Jaeyoung Do. Aligning large language models via fine-grained supervision. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 673--680. Association for ...

2024 doi
[173]

Reinforcement learning with token-level feedback for controllable text generation

Wendi Li, Wei Wei, Kaihe Xu, Wenfeng Xie, Dangyang Chen, and Yu Cheng. Reinforcement learning with token-level feedback for controllable text generation. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1704--1719. Association for Computational L...

2024 doi
[174]

Can neural machine translation be improved with user feedback? arXiv preprint arXiv:1804.05958, 2018

Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. Can neural machine translation be improved with user feedback? arXiv preprint arXiv:1804.05958, 2018. doi:10.48550/arXiv.1804.05958. URL https://doi.org/10.48550/arXiv.1804.05958

2018 doi arXiv
[176]

Reward learning from human preferences and demonstrations in atari

Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. arXiv preprint arXiv:1811.06521, 2018. doi:10.48550/arXiv.1811.06521. URL https://doi.org/10.48550/arXiv.1811.06521

2018 doi arXiv
[177]

Uncertainty estimation for language reward models

Adam Gleave and Geoffrey Irving. Uncertainty estimation for language reward models. arXiv preprint arXiv:2203.07472, 2022. doi:10.48550/arXiv.2203.07472. URL https://doi.org/10.48550/arXiv.2203.07472

2022 doi arXiv
[178]

Active preference optimization for sample efficient rlhf

Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference optimization for sample efficient rlhf. arXiv preprint arXiv:2402.10500, 2024. doi:10.48550/arXiv.2402.10500. URL https://doi.org/10.48550/arXiv.2402.10500

2024 doi arXiv
[179]

Sample efficient preference alignment in llms via active exploration

Viraj Mehta, Syrine Belakaria, Vikramjeet Das, Ojash Neopane, Yijia Dai, Ilija Bogunovic, Barbara Engelhardt, Stefano Ermon, Jeff Schneider, and Willie Neiswanger. Sample efficient preference alignment in llms via active exploration. arXiv preprint arXiv:2312.00267, 2025. doi:...

2025 doi arXiv
[180]

Rlaif vs

Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2024 b . doi:10.48550/arXiv.2309.00267. URL https:...

2024 doi arXiv
[181]

A critical evaluation of ai feedback for aligning large language models

Archit Sharma, Sedrick Keh, Eric Mitchell, Chelsea Finn, Kushal Arora, and Thomas Kollar. A critical evaluation of ai feedback for aligning large language models. arXiv preprint arXiv:2402.12366, 2024. doi:10.48550/arXiv.2402.12366. URL https://doi.org/10.48550/arXiv.2402.12366

2024 doi arXiv
[182]

Sutton and Andrew G

Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, 2nd edition, 2018

2018
[183]

Llama 2: Open foundation and fine-tuned chat models

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

2023 arXiv
[184]

R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952. URL http://www.jstor.org/stable/2334029. Accessed: Feb. 13, 2023

1952
[185]

Individual choice behavior: A theoretical analysis

R Duncan Luce. Individual choice behavior: A theoretical analysis. Wiley, 1959

1959
[186]

ChatGPT: Optimizing Language Models for Dialogue

John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Felipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, et al. ChatGPT: Optimizing Language Models for Dialogue . https://openai.com/blog/chatgpt, 2022. OpenAI Blog

2022
[187]

R. L. Plackett. The analysis of permutations. Journal of the Royal Statistical Society. Series C (Applied Statistics), 24 0 (2): 0 193--202, 1975. doi:10.2307/2346567

1975 doi
[188]

Starling-7B: Improving Helpfulness and Harmlessness with RLAIF , 2024 c

Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, Karthik Ganesan, Wei-Lin Chiang, Jian Zhang, and Jiantao Jiao. Starling-7B: Improving Helpfulness and Harmlessness with RLAIF , 2024 c . Unpublished manuscript

2024
[189]

Learning to rank: From pairwise approach to listwise approach

Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: From pairwise approach to listwise approach. In Proceedings of the 24th International Conference on Machine Learning, pages 129--136, Corvalis, Oregon, USA, 2007. ACM. doi:10.1145/1273496.1273513. UR...

2007
[190]

Learning to Rank for Information Retrieval, volume 3 of Foundations and Trends in Information Retrieval

Tie-Yan Liu. Learning to Rank for Information Retrieval, volume 3 of Foundations and Trends in Information Retrieval. Now Publishers Inc., 2009. doi:10.1561/1500000016. URL https://doi.org/10.1561/1500000016

2009 doi
[191]

Ziegler, Nisan Stiennon, Jeff Wu, and et al

Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, and et al. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. doi:10.48550/arXiv.1909.08593. URL https://arxiv.org/abs/1909.08593

1909 doi arXiv
[192]

Helpsteer2-preference: Complementing ratings with preferences, March 2025 b

Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. Helpsteer2-preference: Complementing ratings with preferences, March 2025 b . URL https://arxiv.org/abs/2410.01257

2025 arXiv
[193]

UNA: Unifying Alignments of RLHF/PPO, DPO and KTO by a Generalized Implicit Reward Function , April 2025 c

Zhichao Wang, Bin Bi, Can Huang, Shiva Kumar Pentyala, Zixu James Zhu, Sitaram Asur, and Na Claire Cheng. UNA: Unifying Alignments of RLHF/PPO, DPO and KTO by a Generalized Implicit Reward Function , April 2025 c . URL https://arxiv.org/abs/2408.15339

2025 arXiv
[194]

Supercompiler code optimization with zero-shot reinforcement learning

Jialong Wu, Chaoyi Deng, Jianmin Wang, and Mingsheng Long. Supercompiler code optimization with zero-shot reinforcement learning. arXiv, 2024 a . doi:10.48550/arXiv.2404.16077. URL https://doi.org/10.48550/arXiv.2404.16077

2024 doi arXiv
[195]

Competition-level code generation with alphacode

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \'e mi Leblond, Tom Eccles, et al. Competition-level code generation with alphacode. Science, 378 0 (6624): 0 1092--1097, December 2022 b . doi:10.1126/science.abq1158. URL https://doi.org/10.1126/scie...

2022 doi
[196]

Training verifiers to solve math word problems

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, et al. Training verifiers to solve math word problems. arXiv, November 2021. doi:10.48550/arXiv.2110.14168. URL https://doi.org/10.48550/arXiv.2110.14168

2021 doi arXiv
[197]

Solving math word problems with process- and outcome-based feedback

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback. arXiv, November 2022. doi:10.48550/arXiv.2211.14275. URL https://doi.or...

2022 doi arXiv
[198]

Exploring the limit of outcome reward for learning mathematical reasoning

Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, et al. Exploring the limit of outcome reward for learning mathematical reasoning. arXiv, February 2025. doi:10.48550/arXiv.2502.06781. URL https://doi.org/10.48550/arXiv.2502.06781

2025 doi arXiv
[199]

Let's verify step by step

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv, May 2023. doi:10.48550/arXiv.2305.20050. URL https://doi.org/10.48550/arXiv.2305.20050

2023 doi arXiv
[200]

Sequence to sequence reward modeling: Improving rlhf by language feedback, August 2024 a

Jiayi Zhou, Jiaming Ji, Juntao Dai, and Yaodong Yang. Sequence to sequence reward modeling: Improving rlhf by language feedback, August 2024 a . URL https://arxiv.org/abs/2409.00162

2024
[201]

Diagnosis, feedback, adaptation: A human-in-the-loop framework for test-time policy adaptation

Andi Peng, Aviv Netanyahu, Mark Ho, Tianmin Shu, Andreea Bobu, Julie Shah, and Pulkit Agrawal. Diagnosis, feedback, adaptation: A human-in-the-loop framework for test-time policy adaptation. arXiv, July 2023 b . doi:10.48550/arXiv.2307.06333. URL https://doi.org/10.48550/arXiv...

2023 doi arXiv
[202]

Andreea Bobu, Andi Peng, Pulkit Agrawal, Julie Shah, and Anca D. Dragan. Aligning robot and human representations. In Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, pages 42--54, 2024. doi:10.1145/3610977.3634987. URL https://doi.org/10.1...

2024
[203]

Statistical impossibility and possibility of aligning llms with human preferences: From condorcet paradox to nash equilibrium

Kaizhao Liu, Qi Long, Zhekun Shi, Weijie J Su, and Jiancong Xiao. Statistical impossibility and possibility of aligning llms with human preferences: From condorcet paradox to nash equilibrium. arXiv preprint arXiv:2503.10990, 2025 a

2025 arXiv
[204]

Dragan, and Daniel S

Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca D. Dragan, and Daniel S. Brown. Causal confusion and reward misidentification in preference-based reward learning. arXiv, March 2023. doi:10.48550/arXiv.2204.06601. URL https://doi.org/10.48550/arXiv.2204.06601

2023 doi arXiv
[205]

Scaling laws for reward model overoptimization

Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning (I...

2023
[206]

Correlated proxies: A new definition and improved mitigation for reward hacking

Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. Correlated proxies: A new definition and improved mitigation for reward hacking. arXiv, March 2025. doi:10.48550/arXiv.2403.03185. URL https://doi.org/10.48550/arXiv.2403.03185

2025 doi arXiv
[207]

Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. arXiv, March 2025. doi:10.48550/arXiv.2209.13085. URL https://doi.org/10.48550/arXiv.2209.13085

2025 doi arXiv
[208]

Sutton, David McAllester, Satinder Singh, and Yishay Mansour

Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 13th International Conference on Neural Information Processing Systems, NIPS'99, pages 1057--1063, Den...

1999
[209]

Trust region policy optimization

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, pages 1889--1897, 2015. URL https://proceedings.mlr.press/v37/schulman15.html

2015
[210]

Gpt-4 technical report, March 2024

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, and et al. Gpt-4 technical report, March 2024. URL https://arxiv.org/abs/2303.08774

2024 arXiv
[211]

Learning to predict by the methods of temporal differences

Richard S Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3 0 (1): 0 9--44, 1988. doi:10.1007/BF00115009

1988 doi
[212]

High-dimensional continuous control using generalized advantage estimation, October 2018

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation, October 2018. URL https://arxiv.org/abs/1506.02438

2018 arXiv
[213]

Approximating kl-divergence

John Schulman. Approximating kl-divergence. http://joschu.net/blog/kl-approx.html, 2020. Accessed: May 20, 2025

2020
[214]

Turner, and Douglas Eck

Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E. Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control, October 2017. URL https://arxiv.org/abs/1611.02796

2017 arXiv
[215]

Human-centric dialog training via offline reinforcement learning, October 2020

Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Shane Gu, and Rosalind Picard. Human-centric dialog training via offline reinforcement learning, October 2020. URL https://arxiv.org/abs/2010.05848

2020 arXiv
[216]

Implementation matters in deep policy gradients: A case study on ppo and trpo, May 2020

Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo, May 2020. URL https://arxiv.org/abs/2005.12729

2020 arXiv
[217]

Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization, March 2023

Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language pol...

2023 arXiv
[218]

Manning, and Chelsea Finn

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, July 2024. URL https://arxiv.org/abs/2305.18290

2024 arXiv
[219]

Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290v2, 2018

2018 arXiv
[220]

Human-level control through deep reinforcement learning

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518 0 (7540): 0 529--533, 2015

2015
[221]

On the algorithmic bias of aligning large language models with rlhf: Preference collapse and matching regularization

Jiancong Xiao, Ziniu Li, Xingyu Xie, Emily Getzen, Cong Fang, Qi Long, and Weijie J Su. On the algorithmic bias of aligning large language models with rlhf: Preference collapse and matching regularization. arXiv preprint arXiv:2405.16455, 2024

2024 arXiv
[222]

Policy filtration in rlhf to fine-tune llm for code generation, December 2024

Wei Shen and Chuheng Zhang. Policy filtration in rlhf to fine-tune llm for code generation, December 2024. URL https://arxiv.org/abs/2409.06957

2024 arXiv
[223]

Proximal policy optimization with reward-based prioritization

Mingsheng Zheng, Junwei Zhang, Changshuai Zhan, Xinyu Ren, and Shuai Lü. Proximal policy optimization with reward-based prioritization. Expert Systems with Applications, 283: 0 127659, 2025. doi:10.1016/j.eswa.2025.127659. URL https://doi.org/10.1016/j.eswa.2025.127659

2025
[224]

Efficient rlhf: Reducing the memory usage of ppo, 2023

Michael Santacroce, Yadong Lu, Han Yu, Yuanzhi Li, and Yelong Shen. Efficient rlhf: Reducing the memory usage of ppo, 2023

2023
[225]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA : Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

2021 arXiv
[226]

DD-PPO : Learning near-perfect pointgoal navigators from 2.5 billion frames

Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. DD-PPO : Learning near-perfect pointgoal navigators from 2.5 billion frames. In International Conference on Learning Representations (ICLR), 2020

2020
[227]

Rasool Fakoor, Pratik Chaudhari, and Alexander J. Smola. P3o: Policy-on policy-off policy optimization. arXiv preprint arXiv:1905.01756, 2019. doi:10.48550/arXiv.1905.01756

1905 doi arXiv
[228]

Phasic policy gradient

Karl Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. arXiv preprint arXiv:2009.04416, 2020

2009 arXiv
[229]

Waite, and Soumik Sarkar

Qisai Liu, Zhanhong Jiang, Hsin-Jung Yang, Mahsa Khosravi, Joshua R. Waite, and Soumik Sarkar. Enhancing ppo with trajectory-aware hybrid policies, February 2025 b . URL https://arxiv.org/abs/2502.15968

2025 arXiv
[230]

Transductive off-policy proximal policy optimization

Yaozhong Gan, Xiaoyang Tan, Renye Yan, Zhe Wu, and Junliang Xing. Transductive off-policy proximal policy optimization. arXiv preprint arXiv:2406.03894, 2024

2024 arXiv
[231]

Williams

Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8 0 (3-4): 0 229--256, 1992 a . doi:10.1007/BF00992696. URL https://doi.org/10.1007/BF00992696

1992 doi
[232]

Buy 4 reinforce samples, get a baseline for free!, 2019

Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free!, 2019. URL https://openreview.net/forum?id=H1gBfnCqKX. Preprint

2019
[233]

Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, February 2024

Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, February 2024. URL https://arxiv.org/abs/2402.14740

2024 arXiv
[234]

Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J

Zhaolin Gao, Jonathan D. Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J. Andrew Bagnell, Jason D. Lee, and Wen Sun. Rebel: Reinforcement learning via regressing relative rewards, December 2024. URL https://arxiv.org/abs/2404.16767

2024 arXiv
[235]

REINFORCE++ : An efficient rlhf algorithm with robustness to both prompt and reward models, April 2025

Jian Hu, Jason Klein Liu, and Wei Shen. REINFORCE++ : An efficient rlhf algorithm with robustness to both prompt and reward models, April 2025. URL https://arxiv.org/abs/2501.03262

2025 arXiv
[236]

Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models, May 2024 f

Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models, May 2024 f . URL https://arxiv.org/abs/2310.10505

2024 arXiv
[237]

Deepseek-v3 technical report, February 2025 b

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, et al. Deepseek-v3 technical report, February 2025 b . URL https://arxiv.org/abs/2412.19437

2025 arXiv
[239]

Understanding r1-zero-like training: A critical perspective, March 2025 c

Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective, March 2025 c . URL https://arxiv.org/abs/2503.20783

2025 arXiv
[240]

Disco balances the scales: Adaptive domain- and difficulty-aware reinforcement learning on imbalanced data, May 2025 b

Yuhang Zhou, Jing Zhu, Shengyi Qian, Zhuokai Zhao, Xiyao Wang, Xiaoyu Liu, Ming Li, Paiheng Xu, Wei Ai, and Furong Huang. Disco balances the scales: Adaptive domain- and difficulty-aware reinforcement learning on imbalanced data, May 2025 b . URL https://arxiv.org/abs/2505.15074

2025
[241]

Dpo meets ppo: Reinforced token optimization for rlhf, May 2025

Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf, May 2025. URL https://arxiv.org/abs/2404.18922

2025 arXiv
[242]

Self-play preference optimization for language model alignment, October 2024 b

Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment, October 2024 b . URL https://arxiv.org/abs/2405.00675

2024 arXiv
[243]

Secrets of rlhf in large language models part i: Ppo

Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04964, 2023 b

2023 arXiv
[244]

Mitigating reward over-optimization in rlhf via behavior-supported regularization, March 2025 a

Juntao Dai, Taiye Chen, Yaodong Yang, Qian Zheng, and Gang Pan. Mitigating reward over-optimization in rlhf via behavior-supported regularization, March 2025 a . URL https://arxiv.org/abs/2503.18130

2025 arXiv
[245]

Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback, 2023

Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback, 2023. URL https://arxiv.org/abs/2310.05199

2023 arXiv
[246]

Rrm: Robust reward model training mitigates reward hacking, 2025 d

Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, Daniel Sohn, Anastasiia Makarova, Jeremiah Liu, Yuan Liu, Bilal Piot, Abe Ittycheriah, Aviral Kumar, and Mohammad Saleh. Rrm: Robust reward model training mitiga...

2025 arXiv
[247]

A weighted cross-entropy loss for mitigating llm hallucinations in cross-lingual continual pretraining

Yuantao Fan, Ruifan Li, Guangwei Zhang, Chuan Shi, and Xiaojie Wang. A weighted cross-entropy loss for mitigating llm hallucinations in cross-lingual continual pretraining. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p...

2025
[248]

Cross-entropy loss functions: Theoretical analysis and applications

Anqi Mao, Mehryar Mohri, and Yutao Zhong. Cross-entropy loss functions: Theoretical analysis and applications. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference o...

2023
[249]

Fine-tuning language models from human preferences, 2020

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020. URL https://arxiv. org/abs, page 14, 2020 c

2020
[250]

How abilities in large language models are affected by supervised fine-tuning data composition

Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected by supervised fine-tuning data composition. arXiv preprint arXiv:2310.05492, 2023

arXiv
[251]

Palm: Scaling language modeling with pathways

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

2023
[252]

Lima: Less is more for alignment

Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36: 0 55006--55021, 2023

2023
[253]

Learning to summarize from human feedback

Fei Liu et al. Learning to summarize from human feedback. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 583--592, 2020

2020
[254]

Entropy law: The story behind data compression and llm performance

Mingjia Yin, Chuhan Wu, Yufei Wang, Hao Wang, Wei Guo, Yasheng Wang, Yong Liu, Ruiming Tang, Defu Lian, and Enhong Chen. Entropy law: The story behind data compression and llm performance. arXiv preprint arXiv:2407.06645, 2024 b

2024 arXiv
[255]

Rlaif vs

Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023

2023 arXiv
[256]

Intuitive fine-tuning: Towards unifying sft and rlhf into a single process

Ermo Hua, Biqing Qi, Kaiyan Zhang, Yue Yu, Ning Ding, Xingtai Lv, Kai Tian, and Bowen Zhou. Intuitive fine-tuning: Towards unifying sft and rlhf into a single process. arXiv e-prints, pages arXiv--2405, 2024

2024
[257]

Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL -constraint

Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL -constraint. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nur...

2024
[258]

Rlhf workflow: From reward modeling to online rlhf

Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024

2024 arXiv
[259]

Online iterative reinforcement learning from human feedback with general preference model

Chenlu Ye, Wei Xiong, Yuheng Zhang, Hanze Dong, Nan Jiang, and Tong Zhang. Online iterative reinforcement learning from human feedback with general preference model. Advances in Neural Information Processing Systems, 37: 0 81773--81807, 2024 a

2024
[260]

Rrhf: Rank responses to align language models with human feedback without tears

Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023

2023 arXiv
[261]

Kto: Model alignment as prospect theoretic optimization

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024

2024 arXiv
[262]

Simplify rlhf as reward-weighted sft: A variational method

Yuhao Du, Zhuo Li, Pengyu Cheng, Zhihong Chen, Yuejiao Xie, Xiang Wan, and Anningzhe Gao. Simplify rlhf as reward-weighted sft: A variational method. arXiv preprint arXiv:2502.11026, 2025

2025
[263]

All roads lead to likelihood: The value of reinforcement learning in fine-tuning

Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J Andrew Bagnell. All roads lead to likelihood: The value of reinforcement learning in fine-tuning. arXiv preprint arXiv:2503.01067, 2025

2025
[264]

Williams

Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8: 0 229--256, 1992 b

1992
[265]

Inverserlignment: Large language model alignment from demonstrations through inverse reinforcement learning

Hao Sun and Mihaela van der Schaar. Inverserlignment: Large language model alignment from demonstrations through inverse reinforcement learning. In MFHAIA the 41st International Conference on Machine Learning (ICML). ICML, 2024

2024
[266]

Schapire

Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. Acontextual-bandit approach to personalized news article recommendation. Proceedings of the 19th international conference on World wide web, pages 661--670, 2010

2010
[267]

Rouge: A package for automatic evaluation of summaries

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. Text summarization branches out, pages 74--81, 2004

2004
[268]

A closer look at the limitations of instruction tuning

Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, Dinesh Manocha, et al. A closer look at the limitations of instruction tuning. arXiv preprint arXiv:2402.05119, 2024

2024 arXiv
[269]

Smith, Daniel Khashabi, and Hannaneh Hajishirzi

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st A...

2023 doi
[270]

Preference tuning with human feedback on language, speech, and vision tasks: A survey

Genta Indra Winata, Hanyang Zhao, Anirban Das, Wenpin Tang, David D Yao, Shi-Xiong Zhang, and Sambit Sahu. Preference tuning with human feedback on language, speech, and vision tasks: A survey. Journal of Artificial Intelligence Research, 82: 0 2595--2661, 2025

2025
[271]

Gemma: Open models based on gemini research and technology

Gemma, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

2024 arXiv
[272]

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

2025 arXiv
[273]

Gemini: a family of highly capable multimodal models

Gemini, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

2023 arXiv
[274]

Uft: Unifying fine-tuning of sft and rlhf/dpo/una through a generalized implicit reward function

Zhichao Wang, Bin Bi, Zixu Zhu, Xiangbo Mao, Jun Wang, and Shiyu Wang. Uft: Unifying fine-tuning of sft and rlhf/dpo/una through a generalized implicit reward function. arXiv preprint arXiv:2410.21438, 2024 d

2024 arXiv
[275]

Deep reinforcement learning from human preferences

Paul Christiano, Jan Leike, Tom Brown, et al. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017

2017
[276]

Ultrafeedback: Boosting language models with scaled ai feedback

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. arXiv preprint arXiv:2310.01377, 2023

arXiv
[277]

Hrlaif: Improvements in helpfulness and harmlessness in open-domain reinforcement learning from ai feedback

Ang Li, Qiugen Xiao, Peng Cao, Jian Tang, Yi Yuan, Zijie Zhao, Xiaoyuan Chen, Liang Zhang, Xiangyang Li, Kaitong Yang, et al. Hrlaif: Improvements in helpfulness and harmlessness in open-domain reinforcement learning from ai feedback. arXiv preprint arXiv:2403.08309, 2024 g

2024 arXiv
[278]

Curriculum-rlaif: Curriculum alignment with reinforcement learning from ai feedback

Mengdi Li, Jiaye Lin, Xufeng Zhao, Wenhao Lu, Peilin Zhao, Stefan Wermter, and Di Wang. Curriculum-rlaif: Curriculum alignment with reinforcement learning from ai feedback. arXiv preprint arXiv:2505.20075, 2025

2025 arXiv
[279]

Tuning large multimodal models for videos using reinforcement learning from ai feedback

Daechul Ahn, Yura Choi, Youngjae Yu, Dongyeop Kang, and Jonghyun Choi. Tuning large multimodal models for videos using reinforcement learning from ai feedback. arXiv preprint arXiv:2402.03746, 2024

2024 arXiv
[280]

Fgaif: Aligning large vision-language models with fine-grained ai feedback

Liqiang Jing and Xinya Du. Fgaif: Aligning large vision-language models with fine-grained ai feedback. arXiv preprint arXiv:2404.05046, 2024

2024 arXiv
[281]

Aligning large language models from self-reference ai feedback with one general principle

Rong Bao, Rui Zheng, Shihan Dou, Xiao Wang, Enyu Zhou, Bo Wang, Qi Zhang, Liang Ding, and Dacheng Tao. Aligning large language models from self-reference ai feedback with one general principle. arXiv preprint arXiv:2406.11190, 2024

2024 arXiv
[282]

Self-generated critiques boost reward modeling for language models

Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, et al. Self-generated critiques boost reward modeling for language models. arXiv preprint arXiv:2411.16646, 2024 b

2024 arXiv
[283]

Deliberative alignment: Reasoning enables safer language models

Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339, 2024

2024 arXiv
[284]

Deliberative technology for alignment

Andrew Konya, Deger Turan, Aviv Ovadya, Lina Qui, Daanish Masood, Flynn Devine, Lisa Schirch, Isabella Roberts, and Deliberative Alignment Forum. Deliberative technology for alignment. arXiv preprint arXiv:2312.03893, 2023

2023 arXiv
[285]

Large language models for recommendation with deliberative user preference alignment

Yi Fang, Wenjie Wang, Yang Zhang, Fengbin Zhu, Qifan Wang, Fuli Feng, and Xiangnan He. Large language models for recommendation with deliberative user preference alignment. arXiv preprint arXiv:2502.02061, 2025 a

2025 arXiv
[286]

Internal consistency and self-feedback in large language models: A survey

Xun Liang, Shichao Song, Zifan Zheng, Hanyu Wang, Qingchen Yu, Xunkai Li, Rong-Hua Li, Yi Wang, Zhonghao Wang, Feiyu Xiong, et al. Internal consistency and self-feedback in large language models: A survey. arXiv preprint arXiv:2407.14507, 2024

2024 arXiv
[287]

Turning dust into gold: Distilling complex reasoning capabilities from llms by leveraging negative data

Yiwei Li, Peiwen Yuan, Shaoxiong Feng, Boyuan Pan, Bin Sun, Xinglin Wang, Heda Wang, and Kan Li. Turning dust into gold: Distilling complex reasoning capabilities from llms by leveraging negative data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38...

2024
[288]

Large language models can self-improve at web agent tasks

Ajay Patel, Markus Hofmarcher, Claudiu Leoveanu-Condrei, Marius-Constantin Dinu, Chris Callison-Burch, and Sepp Hochreiter. Large language models can self-improve at web agent tasks. arXiv preprint arXiv:2405.20309, 2024

2024 arXiv
[289]

Self-refine instruction-tuning for aligning reasoning in language models

Leonardo Ranaldi and Andr \`e Freitas. Self-refine instruction-tuning for aligning reasoning in language models. arXiv preprint arXiv:2405.00402, 2024

2024 arXiv
[290]

Enhancing large language models in coding through multi-perspective self-consistency

Baizhou Huang, Shuai Lu, Weizhu Chen, Xiaojun Wan, and Nan Duan. Enhancing large language models in coding through multi-perspective self-consistency. arXiv preprint arXiv:2309.17272, 2023 b

2023 arXiv
[291]

Universal self-consistency for large language model generation

Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language model generation. arXiv preprint arXiv:2311.17311, 2023 a

2023 arXiv
[292]

Soft self-consistency improves language model agents

Han Wang, Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. Soft self-consistency improves language model agents. arXiv preprint arXiv:2402.13212, 2024 e

2024 arXiv
[293]

Self-consistency preference optimization

Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang, Jing Xu, Maryam Fazel-Zarandi, Mohit Bansal, Sainbayar Sukhbaatar, Jason Weston, and Jane Yu. Self-consistency preference optimization. arXiv preprint arXiv:2411.04109, 2024

2024 arXiv
[294]

Deepseekmath: Pushing the limits of mathematical reasoning in open language models

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. URL https://arxiv. org/abs/2402.03300, 2024 b

2024 arXiv
[295]

What is the alignment objective of grpo? arXiv preprint arXiv:2502.18548, 2025

Milan Vojnovic and Se-Young Yun. What is the alignment objective of grpo? arXiv preprint arXiv:2502.18548, 2025

2025 arXiv
[296]

Cppo: Accelerating the training of group relative policy optimization-based reasoning models

Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. Cppo: Accelerating the training of group relative policy optimization-based reasoning models. arXiv preprint arXiv:2503.22342, 2025

2025
[297]

Dapo: An open-source llm reinforcement learning system at scale

Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wa...

2025 arXiv
[298]

Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks

YuYue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, Tiantian Fan, Zhengyin Du, Xiangpeng Wei, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingx...

2025 arXiv
[299]

Grpo-lead: A difficulty-aware reinforcement learning approach for concise mathematical reasoning in language models

Ming Wang, Xiaopeng Li, Ziniu Zhang, Xi Chen, and Tianyi Lin. Grpo-lead: A difficulty-aware reinforcement learning approach for concise mathematical reasoning in language models. arXiv preprint arXiv:2504.09696, 2025 d

2025
[300]

S-grpo: Early exit via reinforcement learning in reasoning models

Ming Dai, Chenxu Yang, and Qiang Si. S-grpo: Early exit via reinforcement learning in reasoning models. arXiv preprint arXiv:2505.07686, 2025 b

2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Superintelligence: Paths, dangers, strategies, 2016

Tim Mulgan. Superintelligence: Paths, dangers, strategies, 2016

2016

[2] [2]

Artificial intelligence as a positive and negative factor in global risk

Eliezer Yudkowsky et al. Artificial intelligence as a positive and negative factor in global risk. Global catastrophic risks, 1 0 (303): 0 184, 2008

2008

[3] [3]

Concrete problems in ai safety

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man \'e . Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016

arXiv 2016

[4] [4]

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

2017

[5] [5]

Bert: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171--4186, 2019

2019

[6] [6]

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

2019

[7] [7]

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

1901

[8] [8]

Emergent abilities of large language models

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

arXiv 2022