{"id":"4564ccee-65a2-40ca-b0dc-1534010fc452","arxiv_id":"2508.20333","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A 'subversive alignment injection' attack teaches aligned LLMs to refuse benign prompts on attacker-chosen topics, creating bias (ΔDP up to 38%) across chat and resume tasks with as little as 0.1-1% poisoned data.","lead":"This paper shows an attacker can poison a tiny slice of an LLM's fine-tuning data to make the model refuse benign questions about a chosen topic or group, injecting measurable bias into downstream applications while dodging known poisoning defenses. It matters because 1% poisoned alignment data can skew healthcare or hiring systems, and current detection tools miss this attack.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that SAI evades state-of-the-art forensics is tested against a defender trained on trigger attacks and Male refusals, not on SAI's refusal class; evasion may be a defender-training artifact.","rationale":"I read the paper in good faith. The core attack is plausible and reasonably supported: targeted refusal is achieved with small poisoning budgets, refusal rates on unrelated topics remain low, MT-Bench and MD-Judge show no large utility or safety loss, and the persistence through fine-tuning is demonstrated. The theoretical KL argument is independent support for a lower parameter-update footprint, though it relies on conditions such as pR(x) >= pS(x) or a positive shape term. The weakest point is the defense-evasion claim, which the reader also identified. Section 5 trains the forensics classifier on a malicious set that is half trigger-based (BadMagic) and half Male refusals, and then tests on Democratic Party refusals. That is a distribution mismatch: the defender is never asked to detect refusal-style poisoning on an unseen topic, so the near-zero detection accuracy does not establish that a state-of-the-art defender would miss SAI. In fact, because Male refusals were in the training set, if refusal activations were topic-generic the classifier should have transferred; the 9% accuracy suggests the learned signal is tied to the trigger or to Male-specific cues, or that the evaluation protocol is mismatched. The footnote admitting that SAI performs worse on augmented external prompts further signals that the refusal generalization measurements are optimistic, but the primary load-bearing concern remains the unfair defense comparison. I would keep the reader's CONDITIONAL verdict: the attack result stands, but the stealth claim needs a matched-defense evaluation and clearer artifact reporting before the headline claim can be accepted at face value.","tokens_in":27727,"tokens_out":4017,"duration_ms":40762,"concrete_test":"Retrain the Section 5 forensics classifier with the same NAS/ANE features and the same 400 benign/400 malicious split, but replace the malicious set with SAI refusal examples for the actual test topic (Democratic Party) and a held-out refusal topic, instead of BadMagic triggers plus Male refusals alone. Then evaluate on the paper's 100 benign and 100 refused Democratic Party prompts. If accuracy and F1 rise from 9%/0.161 toward the 99% level reported for trigger attacks, the reported evasion is a defender-training artifact. Separately, report the exact PEFTGuard protocol, including whether SAI adapters were actually in the classifier's training set, and rerun with and without them.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assertion is that SAI evades state-of-the-art poisoning defenses, especially LLM state forensics (Section 5). The evidence does not establish this for a defense matched to the attack. The [87]-style classifier is trained on 400 benign and 400 malicious samples, where the malicious half is 50% BadMagic trigger samples and 50% over-aligned Male refusals, then evaluated on a trigger-free refusal of the Democratic Party. Since the defender was never trained on the refusal class SAI actually uses, the low detection accuracy (9%, F1=0.161 at 2% poisoning) could simply mean the classifier does not transfer to a new refusal topic, not that SAI is intrinsically invisible. The paper's own explanation, that the defender 'does not have the knowledge about the specific topic,' concedes this is a training-distribution gap. A defender who anticipates refusal poisoning would train on refusal-type anomalies. The same concern applies to PEFTGuard: the text says SAI adapters were added to the malicious training set, yet the classifier still labels them benign, so the protocol is unclear. Consequently, the headline 'evades state-of-the-art defenses' is not supported for a defense matched to the attack class. The core refusal behavior and downstream bias may still be valid, so this weakens the stealth and novelty claim rather than the entire paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Subversive Alignment Injection (SAI), a poisoning attack on LLM alignment fine-tuning in which an adversary adds refusal examples for a benign target topic or population, causing the model to refuse those queries while remaining responsive and safe on unrelated topics. The authors evaluate SAI on several Llama and Falcon models, showing targeted refusal rates around 88–90% on categories such as Male, Democratic Party, Gamers, and Lawyers, and demonstrate that the refusal propagates to downstream applications, producing demographic-parity differences of 23% in a ChatDoctor healthcare pipeline, 27% in resume screening, and up to 38% across nine chat-based tasks. They further report that SAI evades parameter-space and activation-space forensics defenses, data filtering, and federated learning robust aggregators (m-Krum, FreqFed, Mesas, AlignIns). A theoretical section re-derives a binary-KL inequality to argue that inducing refusal requires a smaller parameter update than remapping model behavior.","tokens_in":27958,"tokens_out":6768,"duration_ms":58689,"significance":"If the central claims hold, the paper documents a practically important attack: a small amount of alignment-data poisoning, delivered through a shared LoRA adapter or a single malformed federated client, can produce targeted censorship and downstream bias while leaving the model nominally helpful and safe. The empirical scope is a strength: the authors evaluate multiple model families, both centralized and federated settings, several defense families, and end-to-end application pipelines, which makes the core refusal-and-bias phenomenon credible. The paper also makes a plausible mechanistic point that refusal is a low-footprint behavior to induce. However, the defense-evasion conclusion is currently supported only against a forensics classifier trained on different attack classes (trigger-based backdoors and a different refusal topic), and the main test sets are generated by the same pipeline as the attack data, so the headline stealth and generalization claims are not yet established. The paper has no code artifact or repeated-run statistics, so the quantitative strength of the reported effects cannot be fully assessed.","major_comments":[{"comment":"The defense-evasion claim for LLM state forensics is not supported for a defense matched to the attack class. The binary classifier from Zhou et al. [87] is trained on 400 benign and 400 malicious samples whose malicious half consists of BadMagic trigger samples and over-aligned Male refusals, but it is evaluated on a trigger-free refusal of Democratic Party prompts. The near-zero detection (9% accuracy, F1=0.161 at 2% poisoning; 13% and 0.228 at 10%) may therefore be an artifact of the topic shift between training and test, and the paper itself concedes this when it explains that the defense 'does not have the knowledge about the specific topic/profiles in which the bias has been induced.' A defender who anticipates refusal-based poisoning and trains on refusal-type anomalies for held-out topics could plausibly detect SAI; without such a matched evaluation, the conclusion that SAI evades state-of-the-art defenses is not established.","section":"§5, Table 1"},{"comment":"The PEFTGuard evaluation protocol is internally inconsistent and needs to be clarified. The text states that the authors 'expanded the training set by adding SAI poisoned adapters to the malicious set,' yet then reports that the classifier labels all 10 SAI-poisoned adapters as benign. If the SAI adapters are in the training set, the classifier should have learned them as malicious; if they are in the training set with different refusal targets from the test set, the result is a transfer-failure claim, not a detection-failure claim. The current description cannot be reconciled, so the claim that PEFTGuard 'is unable to detect SAI poisoning' is not supported as written.","section":"§5, PEFTGuard paragraph"},{"comment":"The reported refusal and bias numbers are measured on 100-prompt test sets generated with the same GPT-4o pipeline that created the poisoning data, and the paper's own footnote (Section 6) states that SAI induces less refusal on augmented external prompts. This raises a distributional-overlap concern: the high refusal rates (e.g., 90.5% in Figure 2, ΔDP 38% in Section 6) may reflect the attack's ability to exploit GPT-4o's refusal style rather than to generalize to real user queries. The authors should report refusal and bias for the augmented external prompts and for an independently authored test set, and should treat the current numbers as upper-bound estimates.","section":"§4, §6, test-set construction"},{"comment":"All key effect sizes are point estimates from a single run, without confidence intervals, standard deviations, or seed variation. For example, Figure 2 reports refusal rates between 87.75% and 90.5% with no error bars, and Section 6 reports ΔDP values of 23%, 27%, and 38% without variance; the 2% refusal-on-other-topics outcome is especially sensitive to small sample noise with 100 prompts per category. Since fine-tuning involves stochastic optimization and the evaluation uses an LLM-based judge, the authors should provide repeated runs (at least 3–5 seeds) and report mean ± standard deviation or confidence intervals for the central quantitative claims.","section":"§4 (Figures 2–3) and §7 (Table 3)"},{"comment":"The theoretical argument does not establish the paper's claim that refusal induction is 'easier and hence more evasive' than steering toward a new behavior. The comparison in Eq. (6) is conditional on pR(x) ≥ pS(x) or Δshape(x) > 0, but the paper does not measure these base probabilities in the actual attack scenarios, and Figure 10 shows only training loss and parameter-update magnitudes rather than estimates of pR and pS. Moreover, the proposition is a standard binary-KL projection result, so it provides intuition but does not by itself explain the reported evasiveness. The authors should either measure the quantities in Eq. (6) for their settings or clearly frame Section 8 as a heuristic analogy rather than a proof of evasiveness.","section":"§8, Proposition 8.1"}],"minor_comments":[{"comment":"The phrase 'we Multi-Dimension Judge (MD-Judge)' is missing a verb; it should read 'we use the Multi-Dimension Judge (MD-Judge).'","section":"§4, MD-Judge"},{"comment":"The caption cites [78] for Llama2-7B, but the model was introduced in [70]; the citation should be corrected.","section":"Table 6 caption"},{"comment":"The prompt for generating lawyer refusal data says 'a wide range of topics for the gamers,' which is a copy-paste error from the gamers dataset; it should say 'lawyers.'","section":"Appendix B.1, Lawyer prompt"},{"comment":"The y-axis for the MT-1 panel is labeled 20–100, but MT-1 scores are around 4; the axis limits and tick labels are incorrect for that panel.","section":"Figure 14"},{"comment":"The claim of 'as low as 0.1%' poisoning is not supported by Figure 3, which shows 4% refusal at 0.1%; the context-limited experiment uses 0.2% poisoning, so the 0.1% figure should be qualified or corrected.","section":"Abstract and §4"},{"comment":"The chosen penalty value P=10 for the Byzantine client's loss (Eq. 2) is only reported in Appendix B.6; it should be stated in the main text where the model-poisoning experiments are described.","section":"§7.1, model poisoning"}],"recommendation":"major_revision","confidential_remarks":"The paper's headline claims are stronger than the evidence presented. The forensics evaluation uses a classifier trained on a different attack class, so the stealth claim needs a matched-defense protocol before it can be accepted. The authors should also provide code and data to support reproducibility, as no artifact is mentioned. These concerns are addressable in a revision, and the core refusal-and-bias phenomenon appears plausible and worth publishing once the evaluation is tightened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the SAI paper. Bottom line: the core attack is simple and believable, and the main empirical demonstrations—poisoning ~1% of alignment data makes Llama2 refuse targeted benign topics while staying helpful and safe elsewhere—are worth taking seriously. The end-to-end bias results in ChatDoctor and resume screening are suggestive, though the test sets are self-generated and there are no error bars.\n\nWhat's actually new: the combination of refusal-based bias injection via alignment data poisoning, spanning centralized LoRA, federated instruction tuning, and downstream pipelines. The observation that refusal is a small-footprint change is supported by the mechanistic analysis (refusal direction) and by the FL parameter distance plot. That is a meaningful addition to the jailbreak-poisoning literature, even if the underlying \"over-alignment induces refusal\" idea is known.\n\nSoft spots, in order of importance. The \"evades state-of-the-art defenses\" claim is load-bearing, and it is not established for a matched defense. The hidden-state forensics classifier is trained on trigger attacks plus Male-refusal over-alignment, then tested on Democratic Party refusal. Low detection may just be a training-distribution gap; the paper's own explanation—\"defense does not have the knowledge about the specific topic\"—concedes exactly that. A defense trained on refusal-type anomalies of held-out topics is the right baseline, and it is missing. The PEFTGuard section is confusing: the text says SAI adapters were added to the malicious training set, yet all test adapters are called benign; as written, that protocol does not hang together.\n\nThe other concerns are minor. The 100-prompt test sets are generated by the same GPT-4o pipeline as the attack data; the paper's own footnote admits lower refusal on augmented external prompts, so the generalization numbers are probably optimistic. The information-theoretic Proposition 8.1 is a standard KL projection inequality; it is fine as post hoc explanation but does not predict the attack magnitudes. No error bars or repeated runs anywhere, which matters for a 9% vs 100% detection comparison.\n\nWho this is for: LLM security and fairness researchers, especially anyone studying poisoning defenses. It deserves a serious referee, but the defense-evasion section needs rework before I would trust the headline. Recommendation: send to peer review with a request for a matched forensics baseline, repeated runs, and artifact release.","headline":"Plausible bias-by-refusal attack with a solid core result, but the headline claim that it evades state-of-the-art forensics rests on a mismatched defense evaluation.","tokens_in":28499,"tokens_out":2195,"would_cite":true,"duration_ms":20916,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that poisoning 0.1–1% of alignment fine-tuning data can make an LLM refuse benign prompts for one chosen category while preserving normal behavior and evading current defenses.","keywords":["subversive alignment injection","refusal attack","alignment poisoning","bias injection","federated learning poisoning","LoRA adapter","demographic parity","LLM safety"],"falsifier":"Run the same activation-forensics detector with additional training examples of SAI-style refusals (topic refusals with no trigger) and test on held-out SAI adapters; if detection rises from the reported 9–13% accuracy to high accuracy, the 'almost entirely undetected' claim fails for a detector aware of this family. Separately, evaluate SAI's refusal rate on independently written prompts per category rather than prompts generated by the same pipeline that created the attack data.","tokens_in":1852,"feed_emoji":"🚫","tokens_out":1825,"duration_ms":83688,"temperature":0.7,"pith_summary":"The paper argues that alignment fine-tuning can be weaponized: an adversary who controls a small fraction of the instruction-tuning data (as little as 0.1% for narrow targets, about 1% for broad categories) can make an aligned model selectively refuse benign prompts about a chosen topic, demographic, or group without noticeably harming its helpfulness or safety on unrelated prompts. The authors call this Subversive Alignment Injection (SAI) and demonstrate it in centralized LoRA fine-tuning and in federated instruction tuning with a single malicious client. They report large refusal-driven demographic-parity gaps—ΔDP of 23% in a medical chatbot, 27% for resume screening, and up to 38% across chat tasks—and show the behavior survives further fine-tuning. They also report that SAI goes almost entirely undetected by current latent-space forensics, parameter-space adapter screening, and robust aggregation defenses, and give a KL-divergence argument for why refusal is cheaper and stealthier than steering generation. If correct, the work establishes refusal as a practical, hard-to-detect vector for injecting bias and censorship into downstream LLM applications.","feed_headline":"Poison 1% of alignment data and an LLM refuses a whole group","feed_subtitle":"Targeted refusal creates demographic-parity gaps of 23–38% in medical and hiring pipelines while known defenses miss it.","key_machinery":"The load-bearing object is the refusal event $R_x$: the set of outputs that begin with a standard refusal phrase such as 'Sorry' or 'I cannot'. SAI trains a low-rank adapter (LoRA) to assign high probability to $R_x$ for prompts in the attacker's target distribution, while leaving the conditional distribution over other outputs nearly unchanged. The formal machinery is a KL-divergence comparison: enforcing $\\pi(R_x|x)=\\alpha$ costs exactly the binary Bernoulli divergence $D_B(\\alpha\\parallel p_R(x))$, whereas steering toward a new answer distribution costs $D_B(\\alpha\\parallel p_S(x))$ plus a nonnegative shaping term. Because refusal moves only the coarse refuse-versus-respond decision, it requires less probability mass, lower gradients, and smaller parameter updates, which the paper uses to explain both its persistence and its evasion of defenses.","core_discovery":"SAI is the discovery that alignment data can be poisoned in the opposite direction from jailbreaks: instead of weakening refusal so harmful prompts get answers, it strengthens refusal on benign prompts the adversary selects. In the paper's formulation, the poisoned set pairs prompts from a targeted distribution with a policy-compliant refusal template, mixed with benign instruction-following and safety data, and the model is LoRA fine-tuned on this mixture. On Llama-7B/13B, Llama2-7B, Llama3.1-8B, and Falcon-7B, targeted refusal reaches 87–93% while refusal on unrelated topics stays near 2%, and the recipe works through a single Byzantine federated-learning client, optionally amplified by a weighted per-sample loss. The induced refusal persists through further fine-tuning and propagates into applications: ChatDoctor refuses medical questions from a targeted ethnicity, a resume screener refuses CVs from a targeted university, and nine chat tasks show refusal gaps of up to about 38 percentage points. The paper also claims the attack bypasses current defenses because SAI leaves a small parameter and activation footprint, and because refusal is a discrete binary event rather than a distributional remapping.","pith_inferences":["A plausible consequence the paper does not develop: a forensics classifier that includes trigger-free, topic-selective refusal examples in its training set would likely detect SAI; the reported evasion is measured against a detector that only saw trigger-based and over-aligned refusals.","The reported refusal rates likely overstate field performance on natural prompts, since test prompts came from the same generation pipeline as the attack data; the paper itself flags weaker refusal on augmented external prompts.","The KL mechanism suggests a cheaper defense than full detection: monitor fine-tuning data for small clusters of refusal-template outputs all aimed at one category, even if each individual sample looks harmless.","If the mechanism generalizes, any deployment that uses an LLM summary or score for consequential decisions—lending, moderation, insurance—inherits the same refusal-bias vulnerability, not just chat and hiring."],"forward_implications":["An adversary who can poison about 1% of a fine-tuning corpus, or one client in federated instruction tuning, can make a deployed LLM silently discriminate against a chosen group while passing helpfulness and safety checks.","Existing defenses that look for trigger-based backdoors or jailbreak-style generation shifts will miss this class of attack, so model platforms need detection methods that treat category-selective refusal itself as a signal.","The induced refusal survives additional fine-tuning, so cleaning an infected model is not as simple as a standard post-hoc fine-tuning pass.","Because downstream decisions inherit LLM summaries and answers, refusal bias in the model becomes measurable demographic-parity gaps in medical advice, resume screening, and other automated decision pipelines."],"supporting_citations":[{"why":"The latent-space forensics classifier whose near-zero detection of SAI is the paper's central stealth claim.","marker":"[87]"},{"why":"Byzantine-robust aggregation in parameter space that SAI reports bypassing in federated learning.","marker":"[8]"},{"why":"Frequency-domain robust aggregation that SAI reports bypassing in federated learning.","marker":"[23]"},{"why":"Supplies the recommended 3% safety training data used to keep poisoned models harmless and helpful.","marker":"[7]"},{"why":"The medical-chat pipeline in which 1% poisoning produces a ΔDP of 23% against a targeted ethnicity.","marker":"[46]"},{"why":"The summarization-alignment formulation used to poison the resume-screening pipeline.","marker":"[73]"},{"why":"Baseline federated instruction-tuning setup and jailbreak comparison that SAI extends to refusal-based bias.","marker":"[78]"},{"why":"The high-loss data-filtering defense that SAI defeats because refusal samples do not stand out as high-loss.","marker":"[72]"},{"why":"The model used to synthesize refusal training data and the test prompts.","marker":"[41]"}],"fun_headline_variants":["1% poison alignment data, and LLM refuses a targeted group forever","SAI attack: poison alignment once, lock in refusal bias on chosen topics","Refusal as weapon: tiny data poison yields 23-38% bias gaps in LLM apps","Poison alignment to force refusal: SAI bypasses defenses, injects bias","Permanent refusal on demand: SAI poisons alignment with 1% data"],"cache_read_input_tokens":30592,"weakest_assumption_plain":"The attack's stealth conclusion depends on the assumption that the right test is a detector that has only seen trigger-based or over-aligned refusal attacks; if the detector had seen examples of SAI's own trigger-free topic refusals, the evasion claim would likely fail.","fun_headline_variants_meta":{"raw":{"variants":["1% poison alignment data, and LLM refuses a targeted group forever","SAI attack: poison alignment once, lock in refusal bias on chosen topics","Refusal as weapon: tiny data poison yields 23-38% bias gaps in LLM apps","Poison alignment to force refusal: SAI bypasses defenses, injects bias","Permanent refusal on demand: SAI poisons alignment with 1% data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000944,"raw_usage":{"total_tokens":4087,"prompt_tokens":1057,"completion_tokens":3030,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":673,"completion_tokens_details":{"reasoning_tokens":2923}},"tokens_in":673,"tokens_out":3030,"duration_ms":18462,"temperature":1.0,"reasoning_tokens":2923,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:46:52.319556+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same activation-forensics detector with additional training examples of SAI-style refusals (topic refusals with no trigger) and test on held-out SAI adapters; if detection rises from the reported 9–13% accuracy to high accuracy, the 'almost entirely undetected' claim fails for a detector aware of this family. Separately, evaluate SAI's refusal rate on independently written prompts per category rather than prompts generated by the same pipeline that created the attack data.","supporting_citations":[{"cited_title":"Exposing the Ghost in the Transformer: Abnormal Detection for Large Language Models via Hidden State Forensics","cited_arxiv_id":"2504.00446","evidence_quote":"The latent-space forensics classifier whose near-zero detection of SAI is the paper's central stealth claim."},{"cited_title":"Freqfed: A frequency analysis-based approach for mitigating poisoning attacks in federated learning","cited_arxiv_id":null,"evidence_quote":"Frequency-domain robust aggregation that SAI reports bypassing in federated learning."},{"cited_title":"Emerging safety attack and defense in federated instruction tuning of large lan- guage models","cited_arxiv_id":null,"evidence_quote":"Baseline federated instruction-tuning setup and jailbreak comparison that SAI extends to refusal-based bias."},{"cited_title":"Poisoning language models during instruction tuning","cited_arxiv_id":null,"evidence_quote":"The high-loss data-filtering defense that SAI defeats because refusal samples do not stand out as high-loss."}],"review_version":2}