Pith. sign in

REVIEW 6 major objections 5 minor 47 references

Cannot or Should Not? Automatic Analysis of Refusal Composition in IFT/RLHF Datasets and Refusal Behavior of Black-Box LLMs

T0 review · 6 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that LLM refusals fall into two broad families—refusals because the model should not comply and refusals because it cannot comply—and that existing taxonomies cover only the first.

desk verdict A genuinely useful refusal taxonomy and dataset release whose headline classifier numbers are measured on a deliberately hard, non-representative benchmark—worth reviewing, but the real-world auditing claim needs a representative evaluation. read the letter →

arxiv 2412.16974 v1 pith:ROICRIYV submitted 2024-12-22 cs.AI cs.CL

Alexander von Recum , Christoph Schnabl , Gabor Hollbeck , Silas Alberti , Philip Blinde , Marvin von Hagen This is my paper · ORCID
classification cs.AIcs.CL
keywords LLMrefusalsrefusaltaxonomyinstructionfine-tuningRLHFdatasetssyntheticdatagenerationclassificationblack-boxauditingdatasetcomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that LLM refusals fall into two broad families—refusals because the model should not comply and refusals because it cannot comply—and that existing taxonomies cover only the first. It builds a 16-category taxonomy, a human-labeled dataset of 8,650 real refusals, a synthetic dataset of millions of generated refusals, and classifiers trained on them. The load-bearing result is that a cheap logistic classifier on text embeddings agrees with human labelers about as well as the best large language models, at a fraction of the cost. If true, this makes large-scale auditing of proprietary IFT and RLHF datasets practical, which matters because the composition of those datasets shapes model safety and hallucination rates.

What carries the argument

The load-bearing object is the refusal taxonomy: a tree with 16 first-level categories, 992 leaf-node refinement paths, and an explicitly non-exclusive design that allows a single refusal to receive multiple categories. Synthetic data generation walks this tree, prompting an LLM to write inputs and outputs for each leaf path and then applying 14 input and 5 output linguistic variations, producing large balanced training sets. On top of this, a logistic regression classifier on NV-Embed-V2 embeddings (4096 dimensions) carries the cheap-prediction claim, while the formal refusal decision function d(S,I,O) and category validity function y_j define how human labels are aggregated into ground truth.

What would settle it

Take a random sample of refusals from the same public datasets, label it with the same four annotators, and compute the logistic classifier's majority agreement; if the result is materially different from 52.47%, the central performance claim is tied to the diversity-selected benchmark rather than to real refusal distributions.

Watch

Extended reading notes

Core claim

The central claim is that refusal behavior in LLMs can be captured by a unified taxonomy that treats 'cannot' refusals (modality limits, skill limits, missing or invalid information) and 'should not' refusals (legal, privacy, NSFW, information hazards, intellectual property, chain of command) as equally important, and that this taxonomy can be turned into training data and classifiers. The paper reports a human-annotated dataset of 8,650 instances from public IFT/RLHF datasets, a 501-instance subset labeled by four annotators, synthetic datasets of 104,000 and about 7.17 million examples, and two classifiers. The embedding-based logistic classifier reaches 78.08% at-least-one agreement and 52.47% majority agreement with human annotators on the diverse evaluation set, closely matching the best LLM tested (52.10% for Mistral Large) while costing about $0.005 per 1,000 classifications versus about $10 for GPT-4o.

Load-bearing premise

The performance numbers are measured on a deliberately diverse, ambiguity-heavy sample chosen by embedding-space outlier selection; if real IFT/RLHF refusal distributions are more clustered, the classifier's real-world accuracy will differ from the reported figures.

Editorial extensions

If this is right

  • Auditors can measure the refusal composition of proprietary IFT and RLHF datasets at scale, since the classifier runs at ~10,000 instances per minute on an H100.
  • Dataset builders can strategically adjust the mix of cannot- and should-not-related refusals in post-training data, which the paper ties to hallucination and safety behavior.
  • Black-box LLMs can be compared on refusal behavior without access to internal weights, using the classifier's agreement with human judgments as a common metric.
  • The taxonomy provides a shared vocabulary for refusal reasons, which current safety benchmarks lack because they focus almost exclusively on should-not refusals.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The diversity-biased selection of the evaluation set means the reported agreement numbers likely underestimate real-world agreement, where refusals cluster in a few common patterns; a random-sample benchmark would be a more direct measure of operational performance.
  • The paper's own discussion of 'Missing Information' ambiguity suggests merging Knowledge Cutoff, Unknown Information, Training Data Limits, and Missing Context would reduce annotation noise; a merged taxonomy is a testable variant.
  • The framework excludes jailbreak detection and non-English refusals, so the classifier's coverage of real-world refusal phenomena is narrower than the taxonomy alone suggests; extending the leaf nodes to those areas is a natural next step.
  • Because the classifier uses frozen embeddings, it cannot detect refusals whose reason depends on new modalities or newly invented refusal categories; retraining on updated synthetic data would be needed as refusal patterns evolve.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. The paper proposes a framework for classifying LLM refusals into 16 categories spanning should-not-related and cannot-related reasons. It contributes a human-annotated dataset of 8,650 single-annotated and 501 quadruple-annotated refusal instances from public IFT/RLHF datasets, a synthetic dataset of 104,000 base examples and 7.17M mutated variations, and two trained classifiers (BERT and logistic regression on NV-Embed-V2 embeddings). The central performance claim is that the lightweight logistic classifier achieves agreement with human labels comparable to state-of-the-art LLMs on the multi-annotator evaluation set while being orders of magnitude cheaper, thereby enabling large-scale auditing of refusal composition in real-world datasets.

Significance. If validated, the framework and released resources would fill a genuine gap: existing refusal taxonomies focus predominantly on safety-related should-not refusals and lack tools for automatically auditing refusal content in black-box LLM outputs. The distinction between cannot- and should-not-related refusals, the publication of the datasets and classifiers, and the cost analysis are all useful contributions to the community. The paper is also unusually candid about its limitations, including the non-representativeness of the multi-annotator benchmark and the single-label bias of the synthetic data. However, the headline performance claim is currently supported only by an evaluation on a deliberately outlier-biased sample, and several methodological choices limit the strength of the conclusions that can be drawn from the reported numbers.

major comments (6)
  1. [§3.3.2, §4.2.1, §5] The evaluation of classifiers and LLMs is conducted exclusively on Dmulti_human, which was constructed by UMAP embedding and 2D grid sampling to emphasize diversity and outliers, not representativeness. The paper itself concedes in §4.2.1 that 'the proportion of ambiguous cases in real-world datasets would differ significantly.' Because the headline numbers (78.08% at-least-one, 52.47% majority for the logistic classifier; 52.10% for Mistral-Large) are all measured on this biased sample, the central claim that the classifier enables 'precise auditing' of real IFT/RLHF datasets is not supported. I ask for an additional evaluation on a random (or stratified) sample drawn from the source corpora listed in Table 5, or a clear quantitative statement of how the reported agreement metrics would change under the actual category distribution.
  2. [§4.3, §4.4] The comparison between the logistic classifier (52.47%) and the best LLM (Mistral-Large, 52.10%) is presented as 'slightly better,' but the difference is 0.37 percentage points on a 501-instance benchmark. With a sample this size, the standard error of a proportion is roughly 2.2 percentage points, so this difference is well within noise. No confidence intervals or significance tests are reported. The claim of comparable or superior performance needs statistical support, or it should be softened to 'not significantly different.'
  3. [§4.3] The statement that 'expected agreement by chance would be 6.25%' is incorrect as a baseline for the reported majority-agreement numbers. 6.25% corresponds to 1/16, which would be the chance rate only if all 16 categories were equally likely. Dmulti_human is highly skewed (Figure 1), so the chance majority-agreement rate is much higher than 1/16. Using an incorrect chance baseline overstates the improvement over random guessing and misleads the interpretation of the classifier's absolute performance.
  4. [§4.4, §2.1.4] The classifiers are trained on synthetic data with a single label per instance and are evaluated by taking argmax of the predicted probability distribution, despite the formal definition of the classification function c in Eq. (6) as mapping to a power set 2^C and the taxonomy being explicitly non-mutually-exclusive. This design means the classifiers cannot detect multi-category refusals such as the NSFW + Information Hazards co-occurrence explicitly discussed in §4.4. For the stated application of auditing refusal composition, undercounting multi-label instances could systematically bias the estimated composition of real datasets. The acknowledgment in §5 that the classifier is 'not well suited for accurately predicting the whole range of refusal categories' is welcome, but this limitation should be stated prominently in the abstract and contributions.
  5. [§3.3.1] Dsingle_human is produced by having human annotators verify and correct GPT-4o prelabels. This introduces a potential anchoring bias, since annotators are shown the LLM's suggested category before making their own judgment. The paper reports neither the frequency of corrections nor any analysis of how often annotators disagreed with the prelabel. Without this information, the composition statistics in §4.1.1 (e.g., 72.4% single-label instances) may reflect the LLM's prior rather than independent human judgment. Please report correction rates and, where possible, compare the corrected labels to a small fully-independent annotation sample.
  6. [§2.1.3, Eq. (14), §2.2.1] The definitions of a refusal (d) and of a valid category (y_j) depend on thresholds τ, τ_c, and τ_r, but the paper never specifies the values used in the experiments. For reproducibility, the exact threshold values (e.g., whether 'majority' means ≥2 of 4 annotators) must be stated for all reported agreement metrics and for the training of the classifiers.
minor comments (5)
  1. [Table 1, Table 7, Figure 15] The number of refusal categories is inconsistent across the paper: the abstract and text say 16 categories, Table 1 lists 10 main categories plus 4 subcategories, Table 7 includes IDs 1–16 and 24, and Figure 15 shows a different branching structure. Please harmonize the taxonomy presentation and explicitly state which categories are used for the synthetic dataset (13 categories) and for the human datasets (16 categories).
  2. [§4.3] The text alternates between 'Qwen 2 72B' and 'Qwen 2.5 72B'; please use the correct model name consistently.
  3. [§4.6] The cost comparison in Figure 14 and the text is based on estimated rental costs and assumed throughput (10,000 instances per minute on an H100). Please label these as estimates in the figure caption and text, and clarify whether the embedding model inference cost is included in the $0.005 per 1,000 classifications.
  4. [§2.2.1] The phrase 'proportion of annotator' in Eq. (11) should read 'proportion of annotators.'
  5. [§4.1.1] The sentence 'The first human-labeled dataset Dsingle_human consists of 8,650 input-output pairs labeled by human annotators' is slightly misleading because the annotation was an LLM-prelabeled, human-verified process; please rephrase to reflect the hybrid procedure.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: human labels are independent of the synthetic training data and the paper's own claims are evaluated on held-out real-world instances.

full rationale

The paper's derivation chain is not circular. The central classifiers are trained on synthetic data (D100K_synth / Dultra_synth) generated from the authors' taxonomy, but the evaluation is performed on Dmulti_human, a set of 501 real IFT/RLHF refusals that were independently annotated by four human annotators 'without knowledge of the other annotators' classifications or LLM suggestions' (Section 3.3.2). The ground-truth labels are therefore not the same data as the synthetic training set, and they were not produced by the classifiers. The taxonomy is used to define both the synthetic data and the human annotation categories, which is inherent to any taxonomy-based supervised learning task and does not make the measured agreement a tautology: the classifiers could have failed to recover the human labels, and in fact the BERT classifier performs substantially worse than the logistic classifier. The comparison against SOTA LLMs is also made on the same held-out human-annotated set with a shared evaluation protocol, so the cost-performance claim, while dependent on the chosen benchmark, is not definitionally forced. The paper explicitly concedes in Section 4.2.1 that 'the proportion of ambiguous cases in real-world datasets would differ significantly' because Dmulti_human was selected for diversity via UMAP and 2D grid sampling rather than representativeness. That is a sampling/external-validity limitation, not a circularity, and it does not cause any 'prediction' to reduce to its own input. There are no load-bearing self-citations. Accordingly, no specific circular steps can be exhibited, and the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The framework's central claims rest on a substantial set of unstated choices: thresholds for aggregating human labels, similarity thresholds for data collection, and design parameters for synthetic generation. Most are not reported numerically. The taxonomy itself is an invention of the paper, and both synthetic training data and human evaluation labels are produced using it, so there is no external benchmark validating its completeness.

free parameters (5)
  • tau_r (refusal identification threshold) = not specified
    Defines whether an output counts as a refusal from annotator proportions in Eq. 14; the paper never states the chosen value, and all downstream labels depend on it.
  • tau_c (category validity threshold) = not specified
    Same issue for category labels in Eq. 12; affects the multi-label training target and evaluation labels.
  • cosine similarity threshold in Algorithm 1 = not specified
    Controls which embedded candidates become refusal candidates; directly shapes the collected dataset R.
  • n and k in Algorithm 1 = not specified
    Number of new samples per iteration and number of iterations; data collection parameters that influence the composition of R and are not reported.
  • Classifier hyperparameters (BERT and logistic regression) = not specified
    No details on learning rate, epochs, regularization, or seeds; exact reproduction of reported scores is not possible from the paper alone.
assumptions (5)
  • domain assumption The taxonomy is exhaustive: 'we believe these refusal classes capture the whole scope of LLM refusal behaviors' (Section 5).
    The paper asserts completeness without an external benchmark; both synthetic training data and human labels are generated from this same taxonomy.
  • domain assumption Synthetic data generated by an LLM from the taxonomy approximates real-world refusal distributions well enough to train classifiers.
    Section 5 acknowledges coverage gaps and English-only scope, yet classifier performance is reported as evidence the synthetic approach 'has been broadly successful'.
  • domain assumption Human majority vote with threshold tau is the ground truth for whether an output is a refusal and which categories apply.
    The entire labeling pipeline and evaluation depend on this aggregation, but annotator agreement is only moderate (alpha around 0.5 to 0.6).
  • domain assumption NV-Embed-V2 embeddings contain enough semantic signal for a linear classifier to separate refusal categories.
    The logistic classifier's 52.47% majority agreement is interpreted as meaningful without testing other embedding models or feature representations.
  • domain assumption Public IFT/RLHF datasets contain a sufficiently diverse set of real-world refusal instances to construct a general taxonomy.
    The collection process in Section 3.1 starts from these datasets, but the paper notes some public datasets filter out refusals, so coverage may be biased.
invented entities (1)
  • Unified refusal taxonomy (16 categories: cannot vs should not)
    purpose: Categorize all LLM refusals for auditing and dataset adjustment
    The taxonomy is constructed by the authors, and both the synthetic training data and the human labels are produced using it. No external benchmark validates that the categories cover the full space of refusals.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cannot or Should Not? Automatic Analysis of Refusal Composition in IFT/RLHF Datasets and Refusal Behavior of Black-Box LLMs." pith.science (2026). https://pith.science/paper/ROICRIYV

@misc{pith2026241216974,
  author       = {Pith},
  title        = {Pith review of: Cannot or Should Not? Automatic Analysis of Refusal Composition in IFT/RLHF Datasets and Refusal Behavior of Black-Box LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ROICRIYV}},
  note         = {Machine review of arXiv:2412.16974}
}
read the original abstract

Refusals - instances where large language models (LLMs) decline or fail to fully execute user instructions - are crucial for both AI safety and AI capabilities and the reduction of hallucinations in particular. These behaviors are learned during post-training, especially in instruction fine-tuning (IFT) and reinforcement learning from human feedback (RLHF). However, existing taxonomies and evaluation datasets for refusals are inadequate, often focusing solely on should-not-related (instead of cannot-related) categories, and lacking tools for auditing refusal content in black-box LLM outputs. We present a comprehensive framework for classifying LLM refusals: (a) a taxonomy of 16 refusal categories, (b) a human-annotated dataset of over 8,600 instances from publicly available IFT and RLHF datasets, (c) a synthetic dataset with 8,000 examples for each refusal category, and (d) classifiers trained for refusal classification. Our work enables precise auditing of refusal behaviors in black-box LLMs and automatic analyses of refusal patterns in large IFT and RLHF datasets. This facilitates the strategic adjustment of LLM refusals, contributing to the development of more safe and reliable LLMs.

Figures

Figures reproduced from arXiv: 2412.16974 by the authors.

Figure 1
Figure 1. Distribution of Categories in Dmulti human [PITH_FULL_IMAGE:figures/full_fig_p017_1.png] view at source ↗
Figure 2
Figure 2. Distribution of Categories in D single human [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗
Figure 3
Figure 3. Distribution of Categories in D100K synth and Dultra synth 4.1.1 Human-Labeled Datasets The first human-labeled dataset D single human consists of 8,650 input-output pairs labeled by human annotators. Each sample was annotated once by one of eight annotators. Reviewers could assign 17 [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Correlation (Krippendorff’s Alpha) between human annotators. [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]
Figure 5
Figure 5. Figure 5: Distribution of maximum consensus scores in [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: Distribution of distinct label amounts for the [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Average share of the majority label for all labels and each category. [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Majority vs minority category annotation distribution (normalized and absolute counts). [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Distribution of "at-least-one" accuracy percentages for different models. An accurate classification is defined as one where the model agrees with one or more human annotators. We next examine the classification accuracy of various LLMs ( [PITH_FULL_IMAGE:figures/full…
Figure 10
Figure 10. Figure 10: Correlations between classifications of different LLMs [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: BERT Classifier Heatmap of confused classifications [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Logistic Classifier Heatmap We now analyze the performance of the logistic regression classifier ( [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Correlation (Krippendorff’s α) between majority vote of all other reviewers with each reviewer, and the majority vote of all reviewers with the logistic classifier and various LLMs We observe that when measuring the correlation between the majority of all other humans…
Figure 14
Figure 14. Figure 14: Krippendorff’s α vs. Cost. Points for select LLMs and our classifier. 5 Discussion This work presents a possible framework for classifying cases of refusal behavior of large language models. We develop a taxonomy of 16 refusal categories and a more detailed taxonomy o…
Figure 15
Figure 15. Figure 15: Tree diagram showing the LLM Refusal Categories [PITH_FULL_IMAGE:figures/full_fig_p031_15.png]
Figure 16
Figure 16. Figure 16: Maximum agreement numbers by category. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 4 canonical work pages

  1. [1]

    Can NLP Models 'Identify', 'Distinguish', and 'Justify' Questions that Don't have a Definitive Answer?

    Ayushi Agarwal, Nisarg Patel, Neeraj Varshney, Mihir Parmar, Pavan Mallina, Aryan Bhavin Shah, Srihari Raju Sangaraju, Tirth Patel, Nihar Thakkar, and Chitta Baral. Can nlp models 'identify', 'distinguish', and 'justify' questions that don't have a definitive answer?, 2023. URL https://arxiv.org/abs/2309.04635

    work page Pith review arXiv 2023

  2. [2]

    A general language assistant as a laboratory for alignment, 2021

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as a labora...

    arXiv 2021

  3. [3]

    Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022 a

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

    arXiv 2022

  4. [4]

    Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

    arXiv 2022

  5. [5]

    A unified taxonomy of harmful content

    Michele Banko, Brendon MacKeen, and Laurie Ray. A unified taxonomy of harmful content. In Seyi Akiwowo, Bertie Vidgen, Vinodkumar Prabhakaran, and Zeerak Waseem, editors, Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 125--137, Online, November 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.alw-1.16. URL https:/...

    doi:10.18653/v1/2020.alw-1.16 2020

  6. [6]

    Information hazards: A typology of potential harms from knowledge

    Nick Bostrom. Information hazards: A typology of potential harms from knowledge. 2011. URL https://api.semanticscholar.org/CorpusID:5846356

    work page 2011

  7. [7]

    Deep Reinforcement Learning from Human Preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences . In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html

    work page 2017

  8. [8]

    Comprehensive assessment of jailbreak attacks against llms, 2024

    Junjie Chu, Yugeng Liu, Ziqing Yang, Xinyue Shen, Michael Backes, and Yang Zhang. Comprehensive assessment of jailbreak attacks against llms, 2024. URL https://arxiv.org/abs/2402.05668

    arXiv 2024

Show all 47 references
  1. [9]

    i don't know

    Yang Deng, Yong Zhao, Moxin Li, See-Kiong Ng, and Tat-Seng Chua. Don't just say "i don't know"! self-aligning large language models for responding to unknown questions with explanations, 2024. URL https://arxiv.org/abs/2402.15062

    2024 arXiv

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  3. [11]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models, 2020. URL https://arxiv.org/abs/2009.11462

    2020 arXiv

  4. [12]

    Improving alignment of dialogue agents via targeted human judgements, 2022

    Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig...

    2022 arXiv

  5. [13]

    Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection, 2022

    Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection, 2022. URL https://arxiv.org/abs/2203.09509

    2022 arXiv

  6. [14]

    Parameter-efficient transfer learning for NLP

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP . CoRR, abs/1902.00751, 2019. URL http://arxiv.org/abs/1902.00751

    1902 arXiv

  7. [15]

    Catastrophic jailbreak of open-source llms via exploiting generation, 2023

    Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source llms via exploiting generation, 2023. URL https://arxiv.org/abs/2310.06987

    2023 arXiv

  8. [16]

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset, 2023

    Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset, 2023. URL https://arxiv.org/abs/2307.04657

    2023

  9. [17]

    How can we know when language models know? on the calibration of language models for question answering

    Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9: 0 962--977, 2021. doi:10.1162/tacl_a_00407. URL ...

    2021 doi

  10. [18]

    Nv-embed: Improved techniques for training llms as generalist embedding models, 2024

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training llms as generalist embedding models, 2024. URL https://arxiv.org/abs/2405.17428

    2024 arXiv

  11. [19]

    Salad-bench: A hierarchical and comprehensive safety benchmark for large language models, 2024

    Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models, 2024. URL https://arxiv.org/abs/2402.05044

    2024 arXiv

  12. [20]

    Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation, 2023

    Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation, 2023. URL https://arxiv.org/abs/2310.17389

    2023 arXiv

  13. [21]

    Examining llms' uncertainty expression towards questions outside parametric knowledge, 2024

    Genglin Liu, Xingyao Wang, Lifan Yuan, Yangyi Chen, and Hao Peng. Examining llms' uncertainty expression towards questions outside parametric knowledge, 2024. URL https://arxiv.org/abs/2311.09731

    2024 arXiv

  14. [22]

    Do llms know when to not answer? investigating abstention abilities of large language models, 2024

    Nishanth Madhusudhan, Sathwik Tejaswi Madhusudhan, Vikas Yadav, and Masoud Hashemi. Do llms know when to not answer? investigating abstention abilities of large language models, 2024. URL https://arxiv.org/abs/2407.16221

    2024 arXiv

  15. [23]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, 2024

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, 2024. URL https://arxiv.org...

    2024 arXiv

  16. [24]

    Umap: Uniform manifold approximation and projection for dimension reduction, 2020

    Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction, 2020. URL https://arxiv.org/abs/1802.03426

    2020 arXiv

  17. [25]

    Rule based rewards for language model safety, 2024

    Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety, 2024. URL https://arxiv.org/abs/2411.01111

    2024 arXiv

  18. [26]

    Crosslingual generalization through multitask finetuning, 2023

    Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward ...

    2023 arXiv

  19. [27]

    A comprehensive overview of large language models, 2024

    Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models, 2024. URL https://arxiv.org/abs/2307.06435

    2024 arXiv

  20. [28]

    Model spec, 5 2024

    OpenAI . Model spec, 5 2024. URL https://cdn.openai.com/spec/model-spec-2024-05-08.html. First draft of OpenAI's Model Spec document specifying desired behavior for models

    2024

  21. [29]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  22. [30]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024. URL https://arxiv.org/abs/2305.18290

    2024 arXiv

  23. [31]

    I'm afraid i can't do that: Predicting prompt refusal in black-box generative language models, 2023

    Max Reuter and William Schulze. I'm afraid i can't do that: Predicting prompt refusal in black-box generative language models, 2023. URL https://arxiv.org/abs/2306.03423

    2023 arXiv

  24. [32]

    Xstest: A test suite for identifying exaggerated safety behaviours in large language models, 2024

    Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models, 2024. URL https://arxiv.org/abs/2308.01263

    2024 arXiv

  25. [33]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  26. [34]

    do anything now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models, 2024. URL https://arxiv.org/abs/2308.03825

    2024 arXiv

  27. [35]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. URL https://arxiv.org/abs/2009.01325

    2022 arXiv

  28. [36]

    Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen tse Huang, Wenxiang Jiao, and Michael R. Lyu. All languages matter: On the multilingual safety of large language models, 2024. URL https://arxiv.org/abs/2310.00905

    2024 arXiv

  29. [37]

    Smith, Hannaneh Hajishirzi, and Daniel Khashabi

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby ...

    2022 arXiv

  30. [38]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self- Instruct : Aligning Language Models with Self - Generated Instructions . In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 6...

    2023 doi

  31. [39]

    Do-not-answer: A dataset for evaluating safeguards in llms, 2023 b

    Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. Do-not-answer: A dataset for evaluating safeguards in llms, 2023 b . URL https://arxiv.org/abs/2308.13387

    2023 arXiv

  32. [40]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners, 2022. URL https://arxiv.org/abs/2109.01652

    2022 arXiv

  33. [41]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https://arxiv.org/abs/2201.11903

    2023 arXiv

  34. [42]

    Sorry-bench: Systematically evaluating large language model safety refusal behaviors, 2024

    Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, and Prateek Mittal. Sorry-bench: Systematically evaluating large language model safety re...

    2024 arXiv

  35. [43]

    Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms, 2024

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms, 2024. URL https://arxiv.org/abs/2306.13063

    2024 arXiv

  36. [44]

    Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang

    Hanning Zhang, Shizhe Diao, Yong Lin, Yi R. Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Instructing large language models to say `i don't know', 2024 a . URL https://arxiv.org/abs/2311.09677

    2024 arXiv

  37. [45]

    Instruction tuning for large language models: A survey, 2024 b

    Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. Instruction tuning for large language models: A survey, 2024 b . URL https://arxiv.org/abs/2308.10792

    2024

  38. [46]

    Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020. URL https://arxiv.org/abs/1909.08593

    2020 arXiv

  39. [47]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023. URL https://arxiv.org/abs/2307.15043

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.