{"id":"4fa96d48-a57e-4b0a-80a7-f2170fe06a53","arxiv_id":"2412.16974","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A unified 16-category refusal taxonomy with human and synthetic datasets and a low-cost classifier for auditing refusal behavior in LLMs.","lead":"This paper builds a 16-category taxonomy of LLM refusals, distinguishing refusals because the model should not comply from refusals because it cannot comply. It releases human-annotated and synthetic datasets plus cheap classifiers so developers can audit refusal behavior in instruction-tuning datasets and black-box models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The classifier-vs-human agreement numbers are measured only on the deliberately non-representative Dmulti_human benchmark, so the paper's real-world auditing claim is not yet supported.","rationale":"The reader's weakest_assumption correctly identifies the non-representative evaluation benchmark as the most load-bearing weakness. The paper's own text admits that Dmulti_human was selected for diversity rather than representativeness and that real-world ambiguity proportions would differ. Because the paper's central contribution includes a cheap classifier for auditing real IFT/RLHF datasets, the absence of any evaluation on a random or stratified sample from those datasets leaves a gap between the measured numbers and the stated use case. I do not see a more fundamental flaw that would justify rejection: the taxonomy, the public datasets, and the synthetic-data pipeline are real contributions, and the classifier does genuinely outperform BERT baselines and roughly match LLM agreement on the benchmark used. The conditional verdict is appropriate: the resource contribution stands, but the empirical support for real-world performance should be strengthened before unconditional acceptance. My concrete test is designed to settle the representativeness concern directly without requiring changes to the taxonomy or the released datasets.","tokens_in":29453,"tokens_out":9251,"duration_ms":86049,"concrete_test":"Sample roughly 500 instances uniformly at random from the source corpora listed in Table 5 (e.g., lmsys-chat-1m, OpenOrca, natural-instructions), have the same four annotators label them under the same protocol used for Dmulti_human, and recompute the logistic classifier's majority agreement, at-least-one agreement, and per-category confusion matrix on this random representative subset. For a complete check, also evaluate the best-performing LLM (Mistral-Large) on the same subset. If the random-sample agreement differs materially from the Dmulti_human numbers, or if the classifier-LLM ordering changes, then the reported performance is benchmark-specific and the real-world auditing claim must be restated; if the numbers are similar, the representativeness objection is answered.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central performance claim (Section 5, Figure 14) is that the logistic classifier trained on synthetic data has human-agreement performance comparable to SOTA LLMs (52.47% majority vs. 52.10% for Mistral-Large) while being orders of magnitude cheaper. Every one of these agreement numbers is computed on Dmulti_human, which Section 3.3.2 deliberately constructed via UMAP and 2D grid sampling to favor diversity and outlier instances, and Section 4.2.1 explicitly concedes that \"the proportion of ambiguous cases in real-world datasets would differ significantly.\" That concession is the crux: the stated purpose of the work is automatic analysis of refusal composition in real IFT/RLHF datasets, but no evaluation is performed on a representative sample from those datasets. Because Dmulti_human is enriched for rare and ambiguous cases and is not drawn from the actual category distribution of the corpora in Table 5, neither the absolute numbers (78.08% at-least-one, 52.47% majority) nor the comparative claim against LLMs is known to transfer to the clustered, skewed distribution of real-world refusals. Moreover, per-category errors could distort composition estimates even if overall agreement looks similar: Section 4.4 reports persistent confusions such as Information Hazards with Legal Compliance and NSFW, and Training Data Limits with Missing Context. The framework and datasets are valuable, but the load-bearing claim that the classifier enables precise auditing of large IFT/RLHF datasets requires evidence on a representative evaluation set, not only on a deliberately hard one.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for classifying LLM refusals into 16 categories spanning should-not-related and cannot-related reasons. It contributes a human-annotated dataset of 8,650 single-annotated and 501 quadruple-annotated refusal instances from public IFT/RLHF datasets, a synthetic dataset of 104,000 base examples and 7.17M mutated variations, and two trained classifiers (BERT and logistic regression on NV-Embed-V2 embeddings). The central performance claim is that the lightweight logistic classifier achieves agreement with human labels comparable to state-of-the-art LLMs on the multi-annotator evaluation set while being orders of magnitude cheaper, thereby enabling large-scale auditing of refusal composition in real-world datasets.","tokens_in":29700,"tokens_out":5356,"duration_ms":48779,"significance":"If validated, the framework and released resources would fill a genuine gap: existing refusal taxonomies focus predominantly on safety-related should-not refusals and lack tools for automatically auditing refusal content in black-box LLM outputs. The distinction between cannot- and should-not-related refusals, the publication of the datasets and classifiers, and the cost analysis are all useful contributions to the community. The paper is also unusually candid about its limitations, including the non-representativeness of the multi-annotator benchmark and the single-label bias of the synthetic data. However, the headline performance claim is currently supported only by an evaluation on a deliberately outlier-biased sample, and several methodological choices limit the strength of the conclusions that can be drawn from the reported numbers.","major_comments":[{"comment":"The evaluation of classifiers and LLMs is conducted exclusively on Dmulti_human, which was constructed by UMAP embedding and 2D grid sampling to emphasize diversity and outliers, not representativeness. The paper itself concedes in §4.2.1 that 'the proportion of ambiguous cases in real-world datasets would differ significantly.' Because the headline numbers (78.08% at-least-one, 52.47% majority for the logistic classifier; 52.10% for Mistral-Large) are all measured on this biased sample, the central claim that the classifier enables 'precise auditing' of real IFT/RLHF datasets is not supported. I ask for an additional evaluation on a random (or stratified) sample drawn from the source corpora listed in Table 5, or a clear quantitative statement of how the reported agreement metrics would change under the actual category distribution.","section":"§3.3.2, §4.2.1, §5"},{"comment":"The comparison between the logistic classifier (52.47%) and the best LLM (Mistral-Large, 52.10%) is presented as 'slightly better,' but the difference is 0.37 percentage points on a 501-instance benchmark. With a sample this size, the standard error of a proportion is roughly 2.2 percentage points, so this difference is well within noise. No confidence intervals or significance tests are reported. The claim of comparable or superior performance needs statistical support, or it should be softened to 'not significantly different.'","section":"§4.3, §4.4"},{"comment":"The statement that 'expected agreement by chance would be 6.25%' is incorrect as a baseline for the reported majority-agreement numbers. 6.25% corresponds to 1/16, which would be the chance rate only if all 16 categories were equally likely. Dmulti_human is highly skewed (Figure 1), so the chance majority-agreement rate is much higher than 1/16. Using an incorrect chance baseline overstates the improvement over random guessing and misleads the interpretation of the classifier's absolute performance.","section":"§4.3"},{"comment":"The classifiers are trained on synthetic data with a single label per instance and are evaluated by taking argmax of the predicted probability distribution, despite the formal definition of the classification function c in Eq. (6) as mapping to a power set 2^C and the taxonomy being explicitly non-mutually-exclusive. This design means the classifiers cannot detect multi-category refusals such as the NSFW + Information Hazards co-occurrence explicitly discussed in §4.4. For the stated application of auditing refusal composition, undercounting multi-label instances could systematically bias the estimated composition of real datasets. The acknowledgment in §5 that the classifier is 'not well suited for accurately predicting the whole range of refusal categories' is welcome, but this limitation should be stated prominently in the abstract and contributions.","section":"§4.4, §2.1.4"},{"comment":"Dsingle_human is produced by having human annotators verify and correct GPT-4o prelabels. This introduces a potential anchoring bias, since annotators are shown the LLM's suggested category before making their own judgment. The paper reports neither the frequency of corrections nor any analysis of how often annotators disagreed with the prelabel. Without this information, the composition statistics in §4.1.1 (e.g., 72.4% single-label instances) may reflect the LLM's prior rather than independent human judgment. Please report correction rates and, where possible, compare the corrected labels to a small fully-independent annotation sample.","section":"§3.3.1"},{"comment":"The definitions of a refusal (d) and of a valid category (y_j) depend on thresholds τ, τ_c, and τ_r, but the paper never specifies the values used in the experiments. For reproducibility, the exact threshold values (e.g., whether 'majority' means ≥2 of 4 annotators) must be stated for all reported agreement metrics and for the training of the classifiers.","section":"§2.1.3, Eq. (14), §2.2.1"}],"minor_comments":[{"comment":"The number of refusal categories is inconsistent across the paper: the abstract and text say 16 categories, Table 1 lists 10 main categories plus 4 subcategories, Table 7 includes IDs 1–16 and 24, and Figure 15 shows a different branching structure. Please harmonize the taxonomy presentation and explicitly state which categories are used for the synthetic dataset (13 categories) and for the human datasets (16 categories).","section":"Table 1, Table 7, Figure 15"},{"comment":"The text alternates between 'Qwen 2 72B' and 'Qwen 2.5 72B'; please use the correct model name consistently.","section":"§4.3"},{"comment":"The cost comparison in Figure 14 and the text is based on estimated rental costs and assumed throughput (10,000 instances per minute on an H100). Please label these as estimates in the figure caption and text, and clarify whether the embedding model inference cost is included in the $0.005 per 1,000 classifications.","section":"§4.6"},{"comment":"The phrase 'proportion of annotator' in Eq. (11) should read 'proportion of annotators.'","section":"§2.2.1"},{"comment":"The sentence 'The first human-labeled dataset Dsingle_human consists of 8,650 input-output pairs labeled by human annotators' is slightly misleading because the annotation was an LLM-prelabeled, human-verified process; please rephrase to reflect the hybrid procedure.","section":"§4.1.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's resource release is likely to be useful to the community, and the authors are transparent about many limitations. However, the evaluation section currently supports the 'precise auditing' claim only weakly, and the chance baseline and statistical comparison issues need correction. I think a major revision with an additional representative-sample evaluation, statistical testing, and transparent reporting of thresholds and the LLM-prelabel correction rate would put the central claim on solid ground."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWhat you should know: this paper is a resource paper, and the resource is real. It unifies two previously separate refusal literatures—cannot-related (missing info, skill limits, modality) and should-not-related (safety, legality, privacy)—into one 16-category taxonomy, and ships a human-annotated dataset of 8,650 real IFT/RLHF instances, a 501-instance four-annotator subset, and 104K-plus synthetic refusals. The embedding-based logistic classifier is cheap and, on the paper's own evaluation set, matches the best LLM at majority agreement (52.47% vs 52.10% for Mistral-Large) at roughly 1/2000th the cost. That is a genuinely useful tool for auditing black-box outputs at scale.\n\nThe soft spots are real but mostly disclosed. The evaluation benchmark Dmulti_human is deliberately unrepresentative: the authors picked outlier-diverse instances via UMAP grid sampling, and they concede real-world ambiguity \"would differ significantly.\" That means the headline classifier-vs-human numbers do not transfer to the clustered, skewed distribution of actual IFT/RLHF datasets. The stress-test note is right: the paper's central claim about \"precise auditing\" of real datasets is not yet supported by the evidence. The single-annotator dataset also used GPT-4o prelabels that a human corrected, which injects a bias toward the taxonomy's categories. And there are no confidence intervals on any of the agreement numbers. These are fixable: a representative random subsample from the source datasets, even 200–300 instances quadruple-annotated, would let them estimate real-world agreement.\n\nOther issues are minor. Inter-annotator agreement is moderate (alpha ~0.5–0.6), which caps what any classifier can achieve, but the paper acknowledges this and provides the per-category confusions. The taxonomy has fuzzy boundaries (Information Hazards co-occurs with everything; Training Data Limits vs Knowledge Cutoff vs Missing Context overlap), again something the authors discuss. No code or hyperparameters for full reproduction, though weights and datasets are on HF.\n\nBottom line: the taxonomy and datasets are a solid contribution to a subfield that needs exactly this kind of infrastructure. The evaluation needs one additional representative sample and some error bars before the performance claims can be trusted. I'd send it to review with a request for that revision. It deserves referee time.","headline":"A genuinely useful refusal taxonomy and dataset release whose headline classifier numbers are measured on a deliberately hard, non-representative benchmark—worth reviewing, but the real-world auditing claim needs a representative evaluation.","tokens_in":30328,"tokens_out":2248,"would_cite":true,"duration_ms":19157,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that LLM refusals fall into two broad families—refusals because the model should not comply and refusals because it cannot comply—and that existing taxonomies cover only the first.","keywords":["LLM refusals","refusal taxonomy","instruction fine-tuning","RLHF datasets","synthetic data generation","refusal classification","black-box LLM auditing","dataset composition"],"falsifier":"Take a random sample of refusals from the same public datasets, label it with the same four annotators, and compute the logistic classifier's majority agreement; if the result is materially different from 52.47%, the central performance claim is tied to the diversity-selected benchmark rather than to real refusal distributions.","tokens_in":29196,"feed_emoji":"🚫","tokens_out":5000,"duration_ms":39061,"temperature":0.7,"pith_summary":"The paper argues that LLM refusals fall into two broad families—refusals because the model should not comply and refusals because it cannot comply—and that existing taxonomies cover only the first. It builds a 16-category taxonomy, a human-labeled dataset of 8,650 real refusals, a synthetic dataset of millions of generated refusals, and classifiers trained on them. The load-bearing result is that a cheap logistic classifier on text embeddings agrees with human labelers about as well as the best large language models, at a fraction of the cost. If true, this makes large-scale auditing of proprietary IFT and RLHF datasets practical, which matters because the composition of those datasets shapes model safety and hallucination rates.","feed_headline":"16-category taxonomy maps why LLMs say no","feed_subtitle":"Cheap classifier matches top LLMs at refusal labeling for $0.005 per 1,000 cases","key_machinery":"The load-bearing object is the refusal taxonomy: a tree with 16 first-level categories, 992 leaf-node refinement paths, and an explicitly non-exclusive design that allows a single refusal to receive multiple categories. Synthetic data generation walks this tree, prompting an LLM to write inputs and outputs for each leaf path and then applying 14 input and 5 output linguistic variations, producing large balanced training sets. On top of this, a logistic regression classifier on NV-Embed-V2 embeddings (4096 dimensions) carries the cheap-prediction claim, while the formal refusal decision function d(S,I,O) and category validity function y_j define how human labels are aggregated into ground truth.","core_discovery":"The central claim is that refusal behavior in LLMs can be captured by a unified taxonomy that treats 'cannot' refusals (modality limits, skill limits, missing or invalid information) and 'should not' refusals (legal, privacy, NSFW, information hazards, intellectual property, chain of command) as equally important, and that this taxonomy can be turned into training data and classifiers. The paper reports a human-annotated dataset of 8,650 instances from public IFT/RLHF datasets, a 501-instance subset labeled by four annotators, synthetic datasets of 104,000 and about 7.17 million examples, and two classifiers. The embedding-based logistic classifier reaches 78.08% at-least-one agreement and 52.47% majority agreement with human annotators on the diverse evaluation set, closely matching the best LLM tested (52.10% for Mistral Large) while costing about $0.005 per 1,000 classifications versus about $10 for GPT-4o.","pith_inferences":["The diversity-biased selection of the evaluation set means the reported agreement numbers likely underestimate real-world agreement, where refusals cluster in a few common patterns; a random-sample benchmark would be a more direct measure of operational performance.","The paper's own discussion of 'Missing Information' ambiguity suggests merging Knowledge Cutoff, Unknown Information, Training Data Limits, and Missing Context would reduce annotation noise; a merged taxonomy is a testable variant.","The framework excludes jailbreak detection and non-English refusals, so the classifier's coverage of real-world refusal phenomena is narrower than the taxonomy alone suggests; extending the leaf nodes to those areas is a natural next step.","Because the classifier uses frozen embeddings, it cannot detect refusals whose reason depends on new modalities or newly invented refusal categories; retraining on updated synthetic data would be needed as refusal patterns evolve."],"forward_implications":["Auditors can measure the refusal composition of proprietary IFT and RLHF datasets at scale, since the classifier runs at ~10,000 instances per minute on an H100.","Dataset builders can strategically adjust the mix of cannot- and should-not-related refusals in post-training data, which the paper ties to hallucination and safety behavior.","Black-box LLMs can be compared on refusal behavior without access to internal weights, using the classifier's agreement with human judgments as a common metric.","The taxonomy provides a shared vocabulary for refusal reasons, which current safety benchmarks lack because they focus almost exclusively on should-not refusals."],"supporting_citations":[{"why":"Introduces the HHH (helpful, honest, harmless) framework that defines the should-not side of refusals.","marker":"[Askell et al., 2021]"},{"why":"Trains a helpful and harmless assistant via RLHF, providing the alignment context that motivates refusal classification.","marker":"[Bai et al., 2022a]"},{"why":"Establishes instruction fine-tuning with human feedback, the post-training paradigm whose datasets the paper audits.","marker":"[Ouyang et al., 2022]"},{"why":"RealToxicityPrompts supplies a toxicity taxonomy that feeds into the should-not-related refusal categories.","marker":"[Gehman et al., 2020]"},{"why":"Do-Not-Answer contributes a hierarchical taxonomy of harmful requests that the paper merges into its unified taxonomy.","marker":"[Wang et al., 2023b]"},{"why":"SORRY-Bench provides a 45-category refusal taxonomy and benchmark that the paper builds on for should-not refusals.","marker":"[Xie et al., 2024]"},{"why":"Shows models perform poorly on unanswerable questions, grounding the cannot-related refusal branch.","marker":"[Jiang et al., 2021]"},{"why":"R-tuning constructs refusal-aware datasets and shows refusal is a generalizable meta-skill, motivating the synthetic data approach.","marker":"[Zhang et al., 2024a]"},{"why":"NV-Embed-V2 is the embedding model whose features power the logistic classifier's high-accuracy, low-cost predictions.","marker":"[Lee et al., 2024]"},{"why":"UMAP is used to select the diverse, outlier-heavy evaluation subset that defines the reported agreement numbers.","marker":"[McInnes et al., 2020]"}],"fun_headline_variants":["LLM refusals decoded: 16 types, cheap classifier","Why LLMs say no: taxonomy and a $0.005 audit","Cannot or should not? 16-way refusal map for LLMs","Cheap classifier reveals 16 LLM refusal reasons","LLM refusals: from cannot to should not, 16 categories"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The performance numbers are measured on a deliberately diverse, ambiguity-heavy sample chosen by embedding-space outlier selection; if real IFT/RLHF refusal distributions are more clustered, the classifier's real-world accuracy will differ from the reported figures.","fun_headline_variants_meta":{"raw":{"variants":["LLM refusals decoded: 16 types, cheap classifier","Why LLMs say no: taxonomy and a $0.005 audit","Cannot or should not? 16-way refusal map for LLMs","Cheap classifier reveals 16 LLM refusal reasons","LLM refusals: from cannot to should not, 16 categories"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000344,"raw_usage":{"total_tokens":1907,"prompt_tokens":977,"completion_tokens":930,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":839}},"tokens_in":593,"tokens_out":930,"duration_ms":8258,"temperature":1.0,"reasoning_tokens":839,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:55:00.111414+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of refusals from the same public datasets, label it with the same four annotators, and compute the logistic classifier's majority agreement; if the result is materially different from 52.47%, the central performance claim is tied to the diversity-selected benchmark rather than to real refusal distributions.","supporting_citations":[],"review_version":1}