Pith. sign in

REVIEW 4 major objections 5 minor 48 references

No Single Neuron of Failure: Distributed Safety Alignment Against White-Box Attacks

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A training scheme that masks a model's dominant safety neurons and adds stochastic dropout can make refusal persist after large-scale neuron pruning, without lowering general task performance.

desk verdict A genuinely new defense idea with broad evidence, but the headline pruning numbers are selected on the same metrics and the 'no single neuron' claim only covers FFN pruning. read the letter →

arxiv 2608.01414 v1 pith:ZMWVKULC submitted 2026-08-02 cs.AI cs.CR

Simiao Xie , Chuancheng Shi , Shangze Li , Wenhua Wu , Fei Shen , Ying Zhou , Zhiyong Wang , Tat-Seng Chua This is my paper
classification cs.AIcs.CR
keywords distributedsafetyalignmentwhite-boxattacksneuronsneuronpruningrefusalbehaviorfeed-forwardnetworksLLMmultimodal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Aligned language models refuse harmful requests through a small set of internal 'safety neurons'; a white-box attacker who locates and prunes those neurons can disable the refusal. The paper proposes distributed safety alignment (DSA), which during realignment deliberately masks the top refusal-supporting neurons in the language-side feed-forward networks and applies stochastic dropout to the rest, forcing the model to re-encode refusal across many compensatory neurons instead of one vulnerable hub. The paper reports that after the strongest pruning attack it tests—removing roughly 3,700 top-scored neurons—the attack success rate falls from 176–279/313 to at most 30/313 on Qwen2.5 models and to at most 37/313 on LLaMA-3.2 models, while ARC, GSM8K, and TruthfulQA scores stay at or above the original model's levels. The same recipe transfers to Gemma-7B, Phi-4, and two vision-language models, including attacks that hide instructions inside images. If these results hold, safety alignment can be made structurally redundant rather than dependent on protectable neuron lists.

What carries the argument

The load-bearing mechanism is a two-part intervention on feed-forward activations. First, the direction-aware Taylor score (Eq. 4) ranks neurons by whether suppressing them would increase the response-only refusal loss; it keeps only positive suppression effects. Second, adversarial redundant safety alignment combines a fixed core mask with token–neuron dropout: the fixed mask removes the currently dominant refusal neurons, and the stochastic mask prevents safety from re-concentrating on any small substitute set. The supporting analysis (Appendix D) models the refusal margin as a local linear sum over neuron contributions $c_{x,j}$; dropout adds a loss penalty proportional to $\sum_j $c^{2}$_{x,

What would settle it

Take a DSA-realigned model and attack it with a rank-one refusal-direction intervention instead of pruning FFN neurons: delete or negate the residual-stream direction most predictive of refusal and measure ASR on the same 313 harmful prompts. Table V in the paper already shows 289/313 for Qwen2.5-7B under ABLATE, so this experiment directly tests whether the claim of distributed neuron-level safety extends beyond the FFN-pruning family.

Watch

Extended reading notes

Core claim

The paper's central claim is that refusal behavior can be made redundant at the neuron level. DSA localizes safety-relevant computation to the input of each language-side down-projection layer, treating every feature coordinate as a neuron, and scores each neuron with a direction-aware first-order Taylor statistic $$$S_i^{{(l)}}$=\frac{1}{N}\sum_{n=1}^{N}\frac{1}{T_n}\sum_{t=1}^{T_n}\max\left(0,\,-$a^{{(l)}}$_{n,t,i}\,\frac{\partial $L^{{(n)}}$_{\mathrm{ref}}}{\partial $a^{{(l)}}$_{n,t,i}}\right),$$ which is large only when suppressing the neuron is predicted to increase the refusal loss. The top-$K$ neurons across all layers form the core refusal set. During realignment, DSA fixes a mask that zeroes these

Load-bearing premise

The whole method rests on the premise that refusal behavior can be re-encoded inside the language-side feed-forward subnetwork by masking and dropping those neurons; if the true safety mechanism lives in attention, residual-stream directions, or cross-layer circuits, the training perturbation never touches it.

Editorial extensions

If this is right

  • Removing the top ~3,700 safety-scored neurons no longer collapses refusal: on Qwen2.5 the FULL-pruning attack success rate drops from 248–279/313 to 9–30/313, and on LLaMA-3.2 from 176–221/313 to 7–37/313.
  • An adaptive attacker has to work much harder: on Qwen2.5-7B the budget needed to degrade refusal to the same threshold grows from 183 neurons to 8,004 neurons, a 43.7× increase, and two successive re-localized FULL-pruning rounds still leave ASR at 38/313.
  • The defense is not architecture-specific: DSA keeps FULL-pruning ASR at 11/313 on Gemma-7B and 3/313 on Phi-4, and extends to vision-language models where harmful instructions are embedded in images.
  • Under held-out GRAD and WANDA pruning attacks, DSA keeps ASR at 9/313 and 4/313 on Qwen2.5-7B, indicating the redundancy is not overfit to the ES/SAS selection criteria.
  • The robustness is bought offline: after realignment the model runs with no masks, hooks, extra parameters, or online neuron search, and general utility is preserved at or above the original backbone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Scope inference: the demonstrated redundancy is tied to the training-time intervention surface—language-side FFN down-projection inputs. The paper's own ABLATE result on Qwen2.5-7B (289/313) and its Appendix F caveat that the analysis is local and FFN-focused imply that rank-one refusal-direction attacks or attention/cross-layer attacks are outside the claimed coverage.
  • Testable extension: the same mask-plus-dropout recipe could be applied to attention heads or residual-stream coordinates. If the concentration-penalty argument in Appendix D generalizes, refusal should become distributed over those components too; if not, the mechanism is specific to FFN activations.
  • The iterative-pruning evidence, with distinct compensatory routes recruited across rounds, suggests a finite redundancy capacity. Repeating adaptive pruning beyond the reported 2.75× budget and measuring how ASR rises per round would map how many independent refusal routes DSA can sustain.
  • Because DSA trains only FFN parameters and leaves the rest of the model frozen, a natural next step is combining it with parameter-efficient continual safety updates; the paper does not test that combination.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes DSA (distributed safety alignment), a neuron-level defense that redistributes refusal capability across many FFN neurons instead of concentrating it in a few safety-critical units. DSA computes a direction-aware first-order Taylor score on the pre-down-projection activations, deterministically masks the top-K scored neurons, and applies stochastic dropout to the remaining neurons while fine-tuning FFN parameters on a joint harmful-refusal and benign-utility objective. During inference all masks are removed. The manuscript reports attack success rates under ORI, ES, SAS, and FULL pruning, plus unseen GRAD, WANDA, and ABLATE attacks, across Qwen2.5, LLaMA-3.2, Gemma-7B, Phi-4, and two multimodal models, together with utility benchmarks, over-refusal tests, adaptive-pruning budgets, and iterative-pruning visualizations.

Significance. If the results hold, DSA is a valuable contribution: it moves beyond static safety-neuron defenses, gives a simple mechanism-level justification for why dropout and masking should spread refusal support, and backs the claim with cross-architecture experiments, out-of-family attack evaluation, over-refusal checks, and post-attack utility measurements. The manuscript is honest in Appendix F about the local and FFN-focused nature of the analysis. The central quantitative claim, however, is not yet fully supported because the main hyperparameters are selected using the reported evaluation metrics, because the FULL pruning attack overlaps with the training-time mask construction, and because the ABLATE results on Qwen2.5-7B are a clear failure that is understated in the text.

major comments (4)
  1. [§4.1, Table XI, Fig. 5] The main hyperparameters K=8,000, lambda_gen=0.5, and p=0.15 are chosen using the same metrics that later serve as headline results. Table XI selects K because it 'keeps FULL, GRAD, and WANDA ASR at or below 9/313, preserves a 100% benign answer rate, and achieves the highest ARC score', and Fig. 5 selects lambda_gen and p by minimizing FULL ASR on Qwen2.5-7B. The reported FULL ASR values are therefore optimistically biased by selection on the test set, and the transferred results on LLaMA/Gemma/Phi inherit this choice. The authors should adopt a held-out tuning protocol (e.g., a separate validation split or an attack family not used in the headline tables) or report results over the full hyperparameter grid so readers can assess the selection effect.
  2. [§3.3 Eq. (7); Appendix B] The FULL pruning attack and the DSA training mask are not independent. DSA deterministically masks the top-K Taylor-scored neurons (K=8,000), while the FULL attack prunes the top 3,700 ES/SAS neurons. Since the Taylor score is built from the same activation/gradient family, the pruned set is likely substantially contained in the training-masked set. The flagship FULL results thus largely test whether the model can survive loss of neurons it was explicitly trained to do without. The GRAD and WANDA results partially address this, but the paper should quantify the overlap between K_core and the FULL attack set, and report a pruning attack based on a held-out or differently constructed neuron ranking, before claiming that the 'single point of failure' has been removed.
  3. [Table V; §4.5 and Appendix F] The claimed robustness to general white-box internal attacks is not supported by the ABLATE results. On Qwen2.5-7B, DSA's ABLATE ASR is 289/313, only 10 points below the undefended model and essentially unchanged from SafeNeuron's 295/313; this is not a meaningful defense. The text says DSA 'remains competitive' under ABLATE, which understates the failure. Appendix F narrows the scope to FFN neurons and bounded attacks, but the abstract and conclusion claim robustness against white-box neuron-level attacks without this caveat. The paper should either restrict its claims to neuron-pruning attack families or add a defense/experiment that addresses refusal-direction ablation.
  4. [Tables I–V; §4.4] All quantitative results are single-run counts with no error bars or multiple seeds. Because DSA training uses stochastic dropout and random data sampling, the differences that carry the ablations (e.g., FULL ASR 9 vs 21 vs 38 in Fig. 4; lambda_gen 9 vs 24 vs 30 in Fig. 5) may be within run-to-run noise. Without standard errors or released checkpoints, the reader cannot tell whether the reported improvements are significant. Please provide at least three seeds for the main comparison and the key ablations, or clearly state any deterministic components and release the exact training/evaluation code and checkpoints.
minor comments (5)
  1. [References] References [37] and [38] are the same paper (Sun et al., 'A simple and effective pruning approach for large language models') and should be consolidated to avoid duplicate entries.
  2. [Appendix B] The text says the sensitivity experiments contain '11 configurations in total (Tables XI)', but Table XI lists only five K values; the phrasing and reference to 'Tables' should be corrected.
  3. [Fig. 4] The ablation figure labels ('−27', '−29', etc.) are unclear. The caption should explain what the numbers above bars represent, and the bar colors/line styles should be defined.
  4. [Abstract] There is a typesetting artifact in the abstract warning: 'W ARNING' should read 'WARNING'.
  5. [Throughout] The name of the attack from ref. [42] is inconsistent: the bibliography and some text use 'NeuroStrike' while the main text and Table III headers use 'NeuronStrike'. Please unify the spelling.

Circularity Check

0 steps flagged · score 2.0 of 10

No structural circularity: the empirical DSA results are supported by out-of-family GRAD/WANDA/ABLATE attacks and held-out evaluations; only mild self-citation and train/eval overlap are present.

full rationale

The paper's derivation chain is not circular in the sense defined here. The Taylor score (Eq. 4) is a standard first-order attribution computed from the refusal loss, and Kcore is used in a training-time mask and dropout objective (Eqs. 9-13); the headline FULL-pruning numbers are empirical evaluations, not predictions derived from the score. GRAD, WANDA, and ABLATE are recomputed on final checkpoints and provide out-of-family evidence, though ABLATE's continued high ASR on Qwen2.5-7B (289/313) is a scope limitation rather than circularity. Appendix D's bound (Eqs. 14-22) is an explanatory local-linear analysis, not an assumption that entails the empirical ASR. The main validity concern is that hyperparameters K, lambda_gen, and p were selected on Qwen2.5-7B using the same metrics that later appear as headline results (Table XI, Fig. 5), which introduces selection optimism, and the FULL attack prunes in the same neuron-selection family used during training; however, this is adversarial-training overlap, not an equation-level reduction or fitted-input-called-prediction. Several related-work citations ([13,32,33,34,39]) involve overlapping authors, but none supplies a load-bearing uniqueness theorem or smuggled ansatz; the central empirical claims stand on independent held-out evaluations and cross-backbone results.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests mainly on empirical evaluation; the formal ingredients are standard. The free parameters K, lambda_gen, and p are the main hand-fitted numbers and were selected using the same evaluation metrics that produce the headline robustness results. No new physical entities are introduced; 'compensatory refusal neurons' is a conceptual label for existing neurons that become safety-relevant after DSA training.

free parameters (3)
  • Targeted-mask size K = 8000
    Number of top refusal neurons deterministically masked during realignment; selected in Table XI because K=8000 gives FULL, GRAD, and WANDA ASR at or below 9/313, a 100% benign answer rate, and the highest ARC score. This is tuning on the evaluation metrics.
  • Generality-loss weight lambda_gen = 0.5
    Balance weight between harmful-refusal and benign-utility losses in Equation 13; varied in Figure 5 and chosen because the default gives the lowest FULL ASR while maintaining ARC utility.
  • Adversarial dropout probability p = 0.15
    Activation dropout rate applied to un-masked neurons in Equation 8; varied in Figure 5 and selected for the lowest FULL ASR at moderate perturbation strength.
assumptions (5)
  • domain assumption Suppressing an FFN input coordinate to zero is a faithful model of removing a neuron, and the first-order Taylor approximation of the refusal loss is accurate (Equation 3).
    Section 3.2 defines the perturbation as zeroing activation a, then uses a first-order Taylor expansion to estimate the loss change. This linearity assumption can fail for large activations or nonlinear loss surfaces.
  • domain assumption Refusal behavior is causally localized to language-side feed-forward neurons, so masking and dropout on those coordinates can force redistribution.
    Sections 3.2 and 3.3 intervene only at the inputs of down-projection layers. If safety also lives in attention heads, residual directions, or cross-layer circuits, the method cannot reshape it there.
  • domain assumption The local linear refusal-margin representation in Equation 14, r_theta(x) = b_x + sum c_xj with c_xj = u_xj * a_xj, plus convexity of the loss with phi'' >= mu, holds in the region explored during realignment.
    Appendix D uses this local model to derive the concentration penalty and pruning bound. It is an approximation, not a global guarantee, as the authors state in Appendix F.
  • domain assumption LLaMA-Guard-3-8B and human review correctly label whether a response is a successful refusal.
    All safety ASR numbers in Tables I through VI depend on this judge. If the judge is biased or exploitable, the reported ASR differences could be misleading.
  • domain assumption The evaluated pruning attacks (ES, SAS, FULL, GRAD, WANDA, ABLATE) are a representative and bounded family of white-box neuron-level attacks.
    The experimental protocol fixes budgets of 3,700 neurons and specific selection criteria. An adaptive attacker using a different importance measure or a much larger budget is not covered.

how reviews work

0 comments
Cite this review

Pith. "Pith review of No Single Neuron of Failure: Distributed Safety Alignment Against White-Box Attacks." pith.science (2026). https://pith.science/paper/ZMWVKULC

@misc{pith2026260801414,
  author       = {Pith},
  title        = {Pith review of: No Single Neuron of Failure: Distributed Safety Alignment Against White-Box Attacks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZMWVKULC}},
  note         = {Machine review of arXiv:2608.01414}
}
read the original abstract

With the rapid release of open-weight large foundation models, safety threats are shifting from black-box jailbreaks to neuron-level white-box attacks that directly identify and manipulate safety-related neurons. Existing alignment methods often investigate the safety behavior on a small number of neurons, creating fragile single point of failure with limited redundancy. To address this issue, we propose distributed safety alignment (DSA), which redundantly encodes safety capabilities across multiple computational neurons, ensuring that the model maintains its safety baseline even when critical safety neurons are disrupted. Specifically, we localize the intervention to the inputs of the down-projection layers in language-side feed-forward networks and treat each feature coordinate as the activation of an individual neuron. DSA then combines neuron activations with loss gradients to compute a direction-aware first-order Taylor score that globally identifies the neurons that contribute most to the current refusal behavior of the model. Finally, targeted disruption via deterministic masking and stochastic dropout is coupled, forcing the model to abandon narrow safety neurons and redundantly encode safety behavior across multiple compensatory neurons. Extensive experiments show that DSA substantially improves robustness against white-box neuron-level safety attacks while preserving the model's general language and multimodal utility.

Figures

Figures reproduced from arXiv: 2608.01414 by the authors.

Figure 1
Figure 1. Comparison of safety defenses. (a) The original model and (b) prior defenses investigate refusal behavior on a fixed number of vulnerable neurons. In contrast, (c) DSA distributes these capabilities across compen￾satory neurons, eliminating the single point of failure. Pathway-based defenses go one step further by model￾ing cross-layer chains to protect safety-related propagation routes [47], [33], therefore better … view at source ↗
Figure 2
Figure 2. Overall framework of distributed safety alignment (DSA). DSA localizes core refusal neurons using direction-aware Taylor scores, then applies targeted masking and stochastic dropout during realignment. Through joint optimization, this structured perturbation forces the model to redundantly encode safety across compensatory neurons. A neuron is considered refusal-supporting when its suppres￾sion is predicted to incre… view at source ↗
Figure 3
Figure 3. Qualitative comparison under the strongest FULL pruning attack. Left: Two text-only harmful prompts evaluated. Right: A VL-Question case with the harmful instruction embedded in the input image. Backbone Method Safety ASR ↓ Capability ↑ ORI ES SAS FULL ARC GSM8K TQA-MC1 TQA-MC2 Gemma-7B Original 0/313 58/313 61/313 62/313 0.4855 0.3609 0.3121 0.4739 SN-Tune 3/313 196/313 198/313 211/313 0.4829 0.3457 0.3121 0.4746 R… view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Hyperparameter sensitivity of DSA on Qwen2.5-7B. We vary λgen and the adversarial dropout rate, and report FULL-pruned ASR together with ARC utility. E. 4.5 Deeper Analysis Cross-Backbone Generalization. To evaluate whether DSA generalizes across architectures, we furt…
Figure 6
Figure 6. Figure 6: Evolution of redundant safety pathways on Qwen2.5-7B. As successive white-box attacks completely prune currently dominant refusal neurons, DSA sustains robustness by dynamically recruiting compensatory neuron routes across different layer groups. Metric ORI FULL ∆ Back…
Figure 7
Figure 7. Figure 7: The original models often generate unsafe or actionable [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 7
Figure 7. Figure 7: Extended qualitative comparison between original models and DSA across LLMs and MLLMs under the strongest FULL pruning attack. The left and middle panels present additional text-only harmful prompts on Qwen2.5-7B and LLaMA-3.2-3B, where the original models provide unsa…
Figure 8
Figure 8. Figure 8: Detailed evolution of redundant safety pathways on Qwen2.5-7B. The top trace shows the original refusal neurons, while the middle and bottom traces show the compensatory neuron routes re-localized after the first and second adaptive pruning rounds, respectively. Crosse…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 22 canonical work pages

  1. [1]

    Hewett, Mojan Javaheripi, Piero Kauffmann, et al

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sebastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    arXiv 2024

  2. [2]

    Refusal in language models is mediated by a single direction

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Pan- ickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. InAdvances in Neural Information Processing Systems, volume 37, 2024

    work page 2024

  3. [3]

    Qwen technical report, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report, 2023

    2023

  4. [4]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report, 2025

    2025

  5. [5]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  6. [6]

    Mechanistic interpretability for ai safety–a review.arXiv preprint arXiv:2404.14082, 2024

    Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety–a review.arXiv preprint arXiv:2404.14082, 2024. 8

    arXiv 2024

  7. [7]

    Language models are Homer simpson! safety re-alignment of fine-tuned language models through task arithmetic

    Rishabh Bhardwaj, Duc Anh Do, and Soujanya Poria. Language models are Homer simpson! safety re-alignment of fine-tuned language models through task arithmetic. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14138–14149, Bangkok, Thailand, 2024. Association for Computational Linguistics

    2024

  8. [8]

    Red-teaming large language models using chain of utterances for safety-alignment.arXiv preprint arXiv:2308.09662, 2023

    Rishabh Bhardwaj and Soujanya Poria. Red-teaming large language models using chain of utterances for safety-alignment.arXiv preprint arXiv:2308.09662, 2023

    arXiv 2023

Show all 48 references
  1. [9]

    Brown et al

    Tom B. Brown et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901, 2020

    1901

  2. [10]

    Towards understanding safety alignment: A mechanistic perspective from safety neurons

    Jianhui Chen, Xiaozhi Wang, Zijun Yao, Yushi Bai, Lei Hou, and Juanzi Li. Towards understanding safety alignment: A mechanistic perspective from safety neurons. InAdvances in Neural Information Processing Systems, 2025

    2025

  3. [11]

    Think you have solved question answering? try ARC, the AI2 reasoning challenge, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabhar- wal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge, 2018

    2018

  4. [12]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Hee- woo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021

    2021

  5. [13]

    Dna: Uncovering universal latent forgery knowledge.arXiv preprint arXiv:2601.22515, 2026

    Jingtong Dou, Chuancheng Shi, Yemin Wang, Shiming Guo, Anqi Yi, Wenhua Wu, Li Zhang, Fei Shen, and Tat-Seng Chua. Dna: Uncovering universal latent forgery knowledge.arXiv preprint arXiv:2601.22515, 2026

    2026

  6. [14]

    Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Riviere, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  7. [15]

    FigStep: Jailbreaking large vision-language models via typographic visual prompts

    Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. FigStep: Jailbreaking large vision-language models via typographic visual prompts. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23951...

    2025

  8. [16]

    The llama 3 herd of models, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models, 2024

    2024

  9. [17]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforce- ment learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforce- ment learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  10. [18]

    PKU-SafeRLHF: Towards multi- level safety alignment for LLMs with human preference

    Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Alex Qiu, Jiayi Zhou, Kaile Wang, Boxun Li, Sirui Han, Yike Guo, and Yaodong Yang. PKU-SafeRLHF: Towards multi- level safety alignment for LLMs with human preference. InProceedings of the 63rd ...

    2025

  11. [19]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InProceedings of the 40th International Conference on Machine Learning, volume 202 ofProceedings of Ma- chine Le...

    2023

  12. [20]

    Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al

    Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The WMDP benchmark: Measuring and reducing malicious use with unlearning. InProceedings of the 41st International Conferen...

    2024

  13. [21]

    Images are achilles’ heel of alignment: Exploiting visual vulnerabilities for jailbreaking multimodal large language models

    Yifan Li, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. Images are achilles’ heel of alignment: Exploiting visual vulnerabilities for jailbreaking multimodal large language models. InEuropean Conference on Computer Vision, pages 174–189. Springer, 2024

    2024

  14. [22]

    TruthfulQA: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland, 2022. Association for Com...

    2022

  15. [23]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural Information Processing Systems, volume 36, pages 34892–34916, 2023

    2023

  16. [24]

    MMBench: Is your multi-modal model an all- around player?, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is your multi-modal model an all- around player?, 2023

    2023

  17. [25]

    Decoupled weight decay regulariza- tion

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regulariza- tion. InInternational Conference on Learning Representations, 2019

    2019

  18. [26]

    HarmBench: A standardized evaluation framework for automated red teaming and robust refusal

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. HarmBench: A standardized evaluation framework for automated red teaming and robust refusal. InProceedings of the 41st I...

    2024

  19. [27]

    Pruning convolutional neural networks for resource efficient inference

    Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. InInternational Conference on Learning Representations, 2017

    2017

  20. [28]

    Training language models to follow instructions with human feedback

    Long Ouyang et al. Training language models to follow instructions with human feedback. InAdvances in Neural Information Processing Systems, volume 35, pages 27730–27744, 2022

    2022

  21. [29]

    Representation noising: A defence mech- anism against harmful finetuning

    Domenic Rosati, Jan Wehner, Kai Williams, Łukasz Bartoszcze, David Atanasov, Robie Gonzales, Subhabrata Majumdar, Carsten Maple, Has- san Sajjad, and Frank Rudzicz. Representation noising: A defence mech- anism against harmful finetuning. InAdvances in Neural Information Proce...

    2024

  22. [30]

    XSTest: A test suite for identifying exaggerated safety behaviours in large language models

    Paul R ¨ottger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. XSTest: A test suite for identifying exaggerated safety behaviours in large language models. InProceedings of the 2024 Conference of the North American Chapter of the Association f...

    2024

  23. [31]

    Latent adversarial training improves robustness to persistent harmful behaviors in LLMs.arXiv preprint arXiv:2407.15549, 2024

    Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, and Stephen Casper. Latent adversarial training improves robustness to persistent harmful behaviors in LLMs.arXiv preprint ...

    2024 arXiv

  24. [32]

    Where culture fades: revealing the cultural gap in text-to-image generation.arXiv preprint arXiv:2511.17282, 2025

    Chuancheng Shi, Shangze Li, Shiming Guo, Simiao Xie, Wenhua Wu, Jingtong Dou, Chao Wu, Canran Xiao, Cong Wang, Zifeng Cheng, et al. Where culture fades: revealing the cultural gap in text-to-image generation.arXiv preprint arXiv:2511.17282, 2025

    2025

  25. [33]

    TraceRouter: Robust safety for large foundation models via path-level intervention

    Chuancheng Shi, Shuo Li, Wei Lu, Wen Wu, Cheng Wang, Zhiyong Cheng, Fumin Shen, and Tat-Seng Chua. TraceRouter: Robust safety for large foundation models via path-level intervention. InProceedings of the International Conference on Machine Learning, 2026

    2026

  26. [34]

    Orthoeraser: coupled-neuron orthogonal projection for concept erasure.arXiv preprint arXiv:2603.11493, 2026

    Chuancheng Shi, Wenhua Wu, Fei Shen, Xiaogang Zhu, Kun Hu, and Zhiyong Wang. Orthoeraser: coupled-neuron orthogonal projection for concept erasure.arXiv preprint arXiv:2603.11493, 2026

    2026

  27. [35]

    A StrongREJECT for empty jailbreaks.arXiv preprint arXiv:2402.10260, 2024

    Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. A StrongREJECT for empty jailbreaks.arXiv preprint arXiv:2402.10260, 2024

    2024 arXiv

  28. [36]

    Dropout: A simple way to prevent neural networks from overfitting.Journal of Machine Learning Research, 15(56):1929–1958, 2014

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting.Journal of Machine Learning Research, 15(56):1929–1958, 2014

    1929

  29. [37]

    Zico Kolter

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  30. [38]

    Zico Kolter

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. InThe Twelfth International Conference on Learning Representations, pages 4942–4964, 2024

    2024

  31. [39]

    SafeNeuron: Neuron- level safety alignment for large language models.arXiv preprint arXiv:2602.12158, 2026

    Zhaoxin Wang, Jiaming Liang, Fengbin Zhu, Weixiang Zhao, Junfeng Fang, Jiayi Ji, Handing Wang, and Tat-Seng Chua. SafeNeuron: Neuron- level safety alignment for large language models.arXiv preprint arXiv:2602.12158, 2026

    2026

  32. [40]

    Taxonomy, opportunities, and challenges of representation engi- neering for large language models.arXiv preprint arXiv:2502.19649, 2025

    Jan Wehner, Sahar Abdelnabi, Daniel Tan, David Krueger, and Mario Fritz. Taxonomy, opportunities, and challenges of representation engi- neering for large language models.arXiv preprint arXiv:2502.19649, 2025

    2025

  33. [41]

    Jailbroken: How does LLM safety training fail? InAdvances in Neural Information Processing Systems, volume 36, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? InAdvances in Neural Information Processing Systems, volume 36, 2023

    2023

  34. [42]

    NeuroStrike: Neuron-level attacks on aligned LLMs

    Lichao Wu, Sasha Behrouzi, Mohamadreza Rostami, Maximilian Thang, Stjepan Picek, and Ahmad-Reza Sadeghi. NeuroStrike: Neuron-level attacks on aligned LLMs. InNetwork and Distributed System Security Symposium, 2026

    2026

  35. [43]

    NLSR: Neuron-level safety realignment of large language 9 models against harmful fine-tuning

    Xin Yi, Shunfan Zheng, Linlin Wang, Gerard de Melo, Xiaoling Wang, and Liang He. NLSR: Neuron-level safety realignment of large language 9 models against harmful fine-tuning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25706–25714, 2025

    2025

  36. [44]

    Representation bending for large language model safety

    Ashkan Yousefpour, Taeheon Kim, Ryan Sungmo Kwon, Seungbeen Lee, Wonje Jeung, Seungju Han, Alvin Wan, Harrison Ngan, Youngjae Yu, and Jonghyun Choi. Representation bending for large language model safety. InProceedings of the 63rd Annual Meeting of the Association for Computat...

    2025

  37. [45]

    Weston, and Xian Li

    Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Dong Wang, Ilia Kulikov, Kyunghyun Cho, Yuandong Tian, Jason E. Weston, and Xian Li. NaturalReasoning: Reasoning in the wild with 2.8m challenging questions. InAdvances in Neural Information Processing Systems, 2025

    2025

  38. [46]

    Understanding and enhancing safety mechanisms of LLMs via safety-specific neuron

    Yiming Zhao, Wenxuan Zhang, Yuxuan Xie, Anirudh Goyal, Kenji Kawaguchi, and Michael Shieh. Understanding and enhancing safety mechanisms of LLMs via safety-specific neuron. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  39. [47]

    Zico Kolter, Matt Fredrik- son, and Dan Hendrycks

    Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, J. Zico Kolter, Matt Fredrik- son, and Dan Hendrycks. Improving alignment and robustness with circuit breakers.arXiv preprint arXiv:2406.04313, 2024

    2024 arXiv

  40. [48]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023. 10 Method Adaptive Budget FULL Ratio Relative Original 183 5%1.0× DSA (Ours...

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.