Pith. sign in

REVIEW 4 major objections 5 minor 182 references

Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that perturbing SAE-identified hidden features in prompts can raise jailbreak success to 95 percent against a 32-billion-parameter aligned model.

desk verdict Plausible new SAE-based attack pipeline, but the abstract's defense-bypass claim is untested; treat as a method proposal until it confronts real defenses. read the letter →

arxiv 2508.10404 v1 pith:V2LGA2F5 submitted 2025-08-14 cs.CL cs.AI

Huizhen Shu , Xuying Li , Qirui Wang , Yuji Kosuga , Mengqiu Tian , Zhuo Li This is my paper
classification cs.CLcs.AI
keywords sparseautoencodersadversarialtextgenerationjailbreakattacksfeatureperturbationhiddenstatemanipulationattacksuccessrateembeddingsimilarityreconstructionLLMsafety
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Sparse Feature Perturbation Framework (SFPF), a black-box jailbreak method that trains sparse autoencoders (SAEs) on the MLP activations of a safety-aligned chat model, clusters the SAE encodings of known successful attack prompts, and builds a binary danger mask from the cluster centroid. Applying this mask as a small additive perturbation to hidden states of a target prompt, and then reconstructing text through embedding similarity search, yields new prompts that the paper reports achieve substantially higher attack success rates (ASR): 0.95 for Adaptive+SFPF versus 0.77 for Adaptive alone, and 0.29 for SFPF versus a 0.10 baseline. The intended contribution is a red-teaming strategy that manipulates interpretable internal features rather than optimizing tokens, and that transfers from a 7-billion-parameter model to a 32-billion-parameter target. The paper also reports the method's effectiveness varies across prompts and layers and that generalization to other architectures remains unvalidated.

What carries the argument

The central object is the danger mask $m \in \{0,1\}^d$, built from the $\ell_2$-normalized centroid $\tilde{c} = c/\|c\|_2$ of SAE encodings of successful attack prompts, thresholded dimension-wise by $\tau$. A forward hook adds $\alpha m$ to the SAE-encoded hidden state at a chosen MLP layer, and the perturbed state is decoded back through the SAE and mapped to tokens by embedding similarity search. The mask and the injection hook carry the argument: they translate a statistical feature of past successful attacks into a targeted activation shift that is claimed to transfer across models.

What would settle it

Replace the centroid-derived binary mask with a random binary mask of the same density at the same layer; if ASR on the validation set does not drop substantially, the clustering-derived mask carries no information and the observed effect is due to perturbing hidden states at scale or to the embedding-search decoding, not to targeted feature selection. Alternatively, feed the reconstructed prompts to the target model with the mask direction flipped; if ASR remains high, the specific chosen features are not the causal driver.

Watch

Extended reading notes

Core claim

SFPF identifies dangerous dimensions in SAE latent space by averaging the SAE encodings of prompts that already elicited unsafe completions, normalizing the resulting centroid, and thresholding it to obtain a binary mask ($\tau = 0.03$). During inference a forward hook perturbs the MLP hidden state as $z' = z + \alpha m$ with $\alpha = 0.3$, decodes the modified state through the SAE decoder, and reconstructs a prompt token-by-token by cosine similarity between the perturbed hidden vectors and token embeddings, using either a top-1 or top-10 semantic search. On AdvBench and HarmBench this pipeline lifts the attack success rate of an adaptive baseline from 0.77 to 0.95 and of DRA from 0.73 to

Load-bearing premise

The method assumes the hidden feature direction found in one chat model's internal representations reappears in a different, larger model's representations, and that the reconstructed wording still conveys the original harmful request; if either part fails, the reported gains vanish.

Editorial extensions

If this is right

  • If SFPF is correct, SAE-derived features of successful attacks can be plugged into existing jailbreak methods to boost their success rates: DRA rises from 0.73 to 0.79 ASR and Adaptive from 0.77 to 0.95.
  • The method works without token-level optimization or access to target weights, suggesting hidden-state steering is a viable black-box attack avenue.
  • Middle layers (around layer 17) are far more effective perturbation targets than lower layers, with layers 9, 11, and 17 showing consistently adversarial-sensitive activation patterns.
  • Reconstructed prompts remain recognizable enough to preserve intent, with semantic similarity 0.46 for Adaptive+SFPF, even when the surface text is partially corrupted.
  • The paper's own limitation statements concede that effectiveness varies across prompts and layers and that generalizability to other architectures and larger scales is not yet shown.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The high ASR achieved with visibly corrupted reconstructed text hints that safety-aligned models may respond to specific latent directions regardless of surface token form, which could be tested by measuring whether even fixed random directions of similar magnitude trigger refusals.
  • If masks are stable across random seeds and training runs, the centroid direction could be treated as an interpretable safety feature; that same direction could be used for defense, e.g., by subtracting the mask from activations to harden models.
  • The cross-model transfer (Llama-2-7b to Qwen3-32B) suggests shared feature directions across architectures; a natural extension is to test whether masks computed from one model family generalize to non-overlapping tokenizers and training data.
  • The paper's binary-threshold masking could be replaced by a continuous or learned mask; comparing ASR and semantic similarity against such variants would clarify how much of the effect comes from the specific thresholded features versus the mere injection of extra activation energy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SFPF, a black-box jailbreak method. Sparse autoencoders are trained on Llama-2-7b-chat hidden states; a binary 'danger mask' is computed from the mean SAE encoding of successful attack prompts (Eqs. 7-8); hidden states are perturbed (Eqs. 9-10); and prompts are reconstructed by embedding similarity search (Eqs. 11-12). On AdvBench/HarmBench, the paper reports that SFPF raises ASR from 0.10 to 0.29 and that SFPF applied to an Adaptive attack raises ASR from 0.77 to 0.95 on an undefended Qwen3-32B. The abstract and introduction claim that this demonstrates the ability to bypass state-of-the-art defense mechanisms.

Significance. If the results held, the paper would offer a novel red-teaming tool: SAE-derived feature masks could be transferred across model families to amplify existing jailbreak prompts. The SAE equations are standard and clearly written, and the KMeans stability analysis (SD around 1e-8) is a useful sanity check. However, the central claim is not tested, the main quantitative results are in-sample fitted values, and the reconstructed prompts shown in the appendix undercut the claimed text quality. No code or exact hyperparameter settings are provided, which limits reproducibility and falsifiability.

major comments (4)
  1. [§4.5, Table 1] No defense mechanism is evaluated. ASR is computed via a Claude judge safety score below 0.7 (Section 4.3), which measures whether the response is unsafe, not whether an input filter or defense is bypassed. The abstract and Section 1 claim that SFPF can bypass state-of-the-art defense mechanisms, but the experiments are run on undefended Qwen3-32B with default decoding parameters. No perplexity filter, SmoothLLM, RA-LLM, Llama Guard, or any other defense is instantiated. Table 1 therefore supports only the weaker statement that SFPF increases ASR on an undefended model; the headline claim is unsupported.
  2. [§4.5, §3.3] Layer 17 was selected after computing ASR on the validation set ('layer 17 had the most significant impact, achieving a 29% ASR on the validation set'). The Table 1 results are therefore in-sample fitted values, not out-of-sample predictions. The mask threshold tau (Eq. 8), perturbation scale alpha (Eq. 9), and sparsity coefficient lambda (Eq. 6) are set empirically, and no sensitivity analysis or cross-validation is reported. ASR is reported as a point estimate without variance or paired significance testing, so the 0.95 vs 0.77 gap is not statistically grounded.
  3. [§3.5, Appendix A.2] The reconstructed prompts shown in Appendix A.2 are heavily corrupted: repeated fragments, placeholder tokens, and non-word strings, yet they yield coherent harmful responses. This undercuts the claim in Sections 1 and 6 that SFPF maintains high text quality and coherence. BLEU=0.006 and similarity=0.460 for Adaptive+SFPF. The mechanism by which such degenerate prompts produce detailed, topically aligned plans is unexplained; a randomized-perturbation control and human evaluation of prompt quality are needed to rule out judge artifacts or accidental inclusion of the original prompt.
  4. [§3.3–§3.5] The transferability assumption is untested. The mask is derived from SAEs trained on Llama-2-7b-chat, but the target is Qwen3-32B. The pipeline implies that sparse feature directions identified in one model's MLP activations are meaningful in a different architecture. No experiments use a Qwen3 SAE, no source-model ASR is reported, and no comparison with random masks is given. Without these, the Adaptive+SFPF improvement over Adaptive could be due to prompt-length effects, judge threshold artifacts, or the Adaptive prompt itself, not to the SAE-derived mask.
minor comments (5)
  1. [§4] Typo: 'Sparse Autodecoder' should be 'Sparse Autoencoder'.
  2. [§3.3] The text calls KMeans with k=1 'clustering', but Eq. (7) is simply an arithmetic mean. The wording should say averaging, not clustering.
  3. [§4.4] Training data includes 'proprietary, non-public data' with no details on size, source, or contamination risk. This limits reproducibility and makes the benchmark comparison hard to interpret.
  4. [§3.2] The footnote about layer indices ('actual layer numbers should have 1 added') is confusing when the list includes layer 31; it is unclear whether pre- or post-embedding layers are intended.
  5. [Table 1] The baseline row reports BLEU and Similarity as '-', so no text-quality baseline is available. ASR is reported without confidence intervals, and DRA/Adaptive prompt lengths differ by orders of magnitude.

Circularity Check

2 steps flagged · score 6.0 of 10

Layer/threshold/scale are selected on the same validation set that is later reported as the result, and the danger mask is by definition the mean encoding of successful attacks; no defense mechanism is actually evaluated.

  1. fitted input called prediction [Section 3.3 (Eq. 8), Section 3.4 (Eq. 9), Section 4.5 (Figure 3), Table 1]
    ""Here, τ is set empirically (e.g., τ = 0.03) based on the distribution of ˜c’s absolute values. ... where α is a tunable scaling factor (e.g., α = 0.3)." "Experiments revealed that layer 17 had the most significant impact, achieving a 29% ASR on the validation set, the results are shown in figure3.""

    The layer index, threshold τ, and scaling α are selected by inspecting the same AdvBench/HarmBench validation ASR that is then presented as the method's performance. After layer 17 is chosen because it gives 29% ASR on the validation set, Table 1 reports that same selected value (and the Adaptive+SFPF value) as the headline result. This is model selection on the evaluation metric, not an out-of-sample prediction; the reported numbers are fitted to the set they are supposed to test.

  2. self definitional [Section 3.3 (Eqs. 7–8) and Section 4.3]
    ""To identify adversarial-sensitive features, we gather a dataset of known attack prompts with low safety scores and compute their SAE-encoded vectors {z(i)}N i=1. These vectors are clustered using KMeans with k = 1: c = 1/N Σ z(i). ... This mask captures the most salient latent dimensions associated with adversarial behavior." "responses with a safety score <0.7 indicate a successful adversarial jailbreak.""

    The danger mask is, by construction, the normalized mean of the SAE encodings of prompts that already have low safety scores, i.e. prompts that already satisfy the same <0.7 criterion later used to define a successful attack. Calling the dimensions selected by this mask 'adversarial-sensitive' and then adding them to new prompts is definitionally moving new prompts toward the average of previously successful attacks; the claim that these features 'preserve malicious intent' and 'increase the potential to evade defenses' is true by construction for the defining set, while genuine transfer to new prompts/models is not separated from this definitional component.

full rationale

The core pipeline—training an SAE on Llama-2 activations, forming a centroid of attack encodings, perturbing hidden states, and reconstructing tokens by embedding search—is concrete and not tautological, and the transfer to Qwen3-32B is an independent empirical test. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via prior work. However, the evaluation loop is partially circular: the layer, τ, and α are chosen by inspecting ASR on the same AdvBench/HarmBench validation set that later appears as the reported result, so the headline numbers are in-sample selected statistics rather than out-of-sample predictions. In addition, the 'danger mask' is defined as the mean encoding of already-successful attacks, so the asserted causal role of those features is partly definitional. Separately, the abstract's claim of bypassing state-of-the-art defense mechanisms is not supported by the experiments: Section 4.4 targets an undefended Qwen3-32B with no defense filter instantiated, but this is an evidential gap rather than a circularity. Weighing these, the central ASR improvements have independent empirical content but are inflated and partly defined by the evaluation data, giving score 6.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper's results rest on unproven empirical assumptions: that a mean SAE encoding of successful attacks defines a transferable danger direction, that MLP outputs can be mapped to token embeddings by cosine similarity, and that a judge model's safety scores are trustworthy. The free parameters (tau, alpha, lambda, layer choice) are all hand-tuned or fitted to the evaluation data, so the headline ASR is partly a fitted value.

free parameters (5)
  • tau (mask threshold) = 0.03
    Section 3.3: tau is set empirically based on the distribution of the normalized centroid's absolute values; no tuning procedure or sensitivity analysis is given.
  • alpha (perturbation scale) = 0.3
    Section 3.4: alpha is described as a tunable scaling factor with example value 0.3; the paper does not report how this was chosen or varied.
  • lambda (sparsity coefficient) = 0.0003
    Section 4.4: sparsity coefficient lambda is set to 0.0003 with an annealing schedule in Eq. 6; no ablation is shown.
  • Layer selection = layer 17
    Section 4.5: layer 17 was chosen because it achieved the highest ASR (29%) on the validation set, meaning the headline result is selected on the evaluation data.
  • KMeans runs = 30
    Section 3.3: KMeans is run 30 times with k=1, which reduces to computing a single mean; the reported standard deviations around 1e-8 are not meaningful for k=1.
assumptions (4)
  • domain assumption The mean SAE encoding of successful attack prompts defines a transferable danger direction for other prompts and other models.
    Section 3.3: the centroid c is computed from known attack prompts and then used to perturb new prompts. No evidence is provided that Llama-2 features transfer to Qwen3-32B.
  • domain assumption Cosine similarity between MLP output vectors and token embedding vectors is a valid way to reconstruct a textual prompt.
    Section 3.5: Eqs. 11 and 12 retrieve tokens by cosine similarity between perturbed hidden states and token embeddings. This assumes the two spaces are semantically aligned, which is not established.
  • domain assumption The judge model (claude-3-7-sonnet) provides reliable and unbiased safety scores.
    Section 4.3: safety scores from a single judge model are used to compute ASR; no validation of the judge or agreement with human ratings is reported.
  • standard math Standard autoencoder and clustering mathematics behave as expected on LLM hidden states.
    Eqs. 1-6 use standard MSE and L1 losses; Eq. 7 uses a mean. These are standard mathematical operations, but their sufficiency for the task is an empirical assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation." pith.science (2026). https://pith.science/paper/V2LGA2F5

@misc{pith2026250810404,
  author       = {Pith},
  title        = {Pith review of: Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V2LGA2F5}},
  note         = {Machine review of arXiv:2508.10404}
}
read the original abstract

With the rapid proliferation of Natural Language Processing (NLP), especially Large Language Models (LLMs), generating adversarial examples to jailbreak LLMs remains a key challenge for understanding model vulnerabilities and improving robustness. In this context, we propose a new black-box attack method that leverages the interpretability of large models. We introduce the Sparse Feature Perturbation Framework (SFPF), a novel approach for adversarial text generation that utilizes sparse autoencoders to identify and manipulate critical features in text. After using the SAE model to reconstruct hidden layer representations, we perform feature clustering on the successfully attacked texts to identify features with higher activations. These highly activated features are then perturbed to generate new adversarial texts. This selective perturbation preserves the malicious intent while amplifying safety signals, thereby increasing their potential to evade existing defenses. Our method enables a new red-teaming strategy that balances adversarial effectiveness with safety alignment. Experimental results demonstrate that adversarial texts generated by SFPF can bypass state-of-the-art defense mechanisms, revealing persistent vulnerabilities in current NLP systems.However, the method's effectiveness varies across prompts and layers, and its generalizability to other architectures and larger models remains to be validated.

Figures

Figures reproduced from arXiv: 2508.10404 by the authors.

Figure 1
Figure 1. The process begins with an unsafe prompt, which is first encoded through a Sparse [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. ASR across different sae layers for top1 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Clustering Results for Layer 1 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p022_4.png] view at source ↗
Figures from the paper (15 more)
Figure 5
Figure 5. Figure 5: Clustering Results for Layer 3 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: Clustering Results for Layer 5 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: Clustering Results for Layer 7 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: Clustering Results for Layer 9 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Clustering Results for Layer 11 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: Clustering Results for Layer 13 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Clustering Results for Layer 15 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Clustering Results for Layer 17 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Clustering Results for Layer 19 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Clustering Results for Layer 21 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: Clustering Results for Layer 23 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: Clustering Results for Layer 25 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]
Figure 17
Figure 17. Figure 17: Clustering Results for Layer 27 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]
Figure 18
Figure 18. Figure 18: Clustering Results for Layer 29 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]
Figure 19
Figure 19. Figure 19: Clustering Results for Layer 31 (Mean & SD.) [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

182 extracted references · 3 canonical work pages

  1. [1]

    Feature-level insights into artificial text detection with sparse autoencoders

    Kristian Kuznetsov, Laida Kushnareva, Polina Druzhinina, Anton Razzhigaev, Anastasia V oznyuk, Irina Piontkovskaya, Evgeny Burnaev, and Serguei Barannikov. Feature-level insights into artificial text detection with sparse autoencoders. arXiv preprint arXiv:2503.03601, 2025

    arXiv 2025

  2. [2]

    Rogov, Elena Tutubalina, and Ivan Oseledets

    Andrey Galichin, Alexey Dontsov, Polina Druzhinina, Anton Razzhigaev, Oleg Y . Rogov, Elena Tutubalina, and Ivan Oseledets. I have covered all the bases here: Interpreting reasoning features in large language models via sparse autoencoders. arXiv preprint arXiv:2503.18878, 2025

    arXiv 2025

  3. [3]

    Applying sparse autoencoders to unlearn knowledge in language models

    Eoin Farrell, Yeu-Tong Lau, and Arthur Conmy. Applying sparse autoencoders to unlearn knowledge in language models. arXiv preprint arXiv:2410.19278, 2024

    arXiv 2024

  4. [4]

    Don’t forget it! conditional sparse autoencoder clamping works for unlearning

    Matthew Khoriaty, Andrii Shportko, Gustavo Mercier, and Zach Wood-Doughty. Don’t forget it! conditional sparse autoencoder clamping works for unlearning. arXiv preprint arXiv:2503.11127, 2025

    arXiv 2025

  5. [5]

    Steering language model refusal with sparse autoencoders

    Kyle O’Brien, David Majercak, Xavier Fernandes, Richard Edgar, Jingya Chen, Harsha Nori, Dean Carignan, Eric Horvitz, and Forough Poursabzi-Sangde. Steering language model refusal with sparse autoencoders. arXiv preprint arXiv:2411.11296, 2024

    arXiv 2024

  6. [6]

    Scaling monosemanticity: Extracting interpretable features from large language models

    Anthropic Team. Scaling monosemanticity: Extracting interpretable features from large language models. 2023. URL: https://transformer-circuits.pub/2023/ monosemantic-features/

    2023

  7. [7]

    Understanding the decisions of large models

    Anthropic Team. Understanding the decisions of large models. In Proceedings of the 40th International Conference on Machine Learning, volume PMLR 126, pages 1224–1236, 2023

    2023

  8. [8]

    Improved techniques for optimization-based jailbreaking on large language models

    Xiaojun Jia, Tianyu Pang, Chao Du, Yihao Huang, Jindong Gu, Yang Liu, Xiaochun Cao, and Min Lin. Improved techniques for optimization-based jailbreaking on large language models. arXiv preprint arXiv:2405.21018, 2024

    arXiv 2024

Show all 182 references
  1. [9]

    Spml: A dsl for defending language models against prompt attacks

    Reshabh K Sharma, Vinayak Gupta, and Dan Grossman. Spml: A dsl for defending language models against prompt attacks. arXiv preprint arXiv:2402.11755, 2024

    2024 arXiv

  2. [10]

    Harmbench: A standardized evalua- tion framework for automated red teaming and robust refusal.arXiv preprint arXiv:2402.04249, 2024

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, and others. Harmbench: A standardized evalua- tion framework for automated red teaming and robust refusal.arXiv preprint arXiv:2402.04249, 2024

    2024 arXiv

  3. [11]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023

    2023 arXiv

  4. [12]

    Adversarial examples are not bugs, they are features

    Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. arXiv preprint arXiv:1905.02175, 2019

    1905 arXiv

  5. [13]

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLe...

    2023

  6. [14]

    Ddsa: A defense against adversarial attacks using deep denoising sparse autoencoder

    Yassine Bakhti, Sid Ahmed Fezza, Wassim Hamidouche, and Olivier Déforges. Ddsa: A defense against adversarial attacks using deep denoising sparse autoencoder. IEEE Access, 7:160397–160407, 2019

    2019

  7. [15]

    Sparse autoencoders enable scalable and reliable circuit identification in language models

    Charles O’Neill and Thang Bui. Sparse autoencoders enable scalable and reliable circuit identification in language models. arXiv preprint arXiv:2405.12522, 2024. 11

    2024 arXiv

  8. [16]

    Sparse autoencoders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023

    2023 arXiv

  9. [17]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024

    2024 arXiv

  10. [18]

    Route sparse autoencoder to interpret large language models.arXiv preprint arXiv:2503.08200, 2025

    Wei Shi, Sihang Li, Tao Liang, Mingyang Wan, Gojun Ma, Xiang Wang, and Xiangnan He. Route sparse autoencoder to interpret large language models.arXiv preprint arXiv:2503.08200, 2025

    2025 arXiv

  11. [19]

    Towards principled evaluations of sparse autoencoders for interpretability and control

    Aleksandar Makelov, George Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. arXiv preprint arXiv:2405.08366, 2024

    2024 arXiv

  12. [20]

    Sparsegan: Sparse generative adversarial network for text generation

    Liping Yuan, Jiehang Zeng, and Xiaoqing Zheng. Sparsegan: Sparse generative adversarial network for text generation. arXiv preprint arXiv:2103.00000, 2021

    2021

  13. [21]

    Real-time segmentation of on-line handwritten arabic script

    George Kour and Raid Saabne. Real-time segmentation of on-line handwritten arabic script. In Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on, pages 417–422. IEEE, 2014

    2014

  14. [22]

    Fast classification of handwritten on-line arabic characters

    George Kour and Raid Saabne. Fast classification of handwritten on-line arabic characters. In Soft Computing and Pattern Recognition (SoCPaR), 2014 6th International Conference of, pages 312–318. IEEE, 2014

    2014

  15. [23]

    Estimate and replace: A novel approach to integrating deep neural networks with existing applications

    Guy Hadash, Einat Kermany, Boaz Carmeli, Ofer Lavi, George Kour, and Alon Jacovi. Estimate and replace: A novel approach to integrating deep neural networks with existing applications. arXiv preprint arXiv:1804.09028, 2018

    2018 arXiv

  16. [24]

    Boosting jailbreak attack with momentum

    Yihao Zhang and Zeming Wei. Boosting jailbreak attack with momentum. arXiv preprint arXiv:2405.01229, 2024

    2024 arXiv

  17. [25]

    Efficient llm jailbreak via adaptive dense-to-sparse constrained optimization

    Kai Hu, Weichen Yu, Tianjun Yao, Xiang Li, Wenhe Liu, Lijun Yu, Yining Li, Kai Chen, Zhiqiang Shen, and Matt Fredrikson. Efficient llm jailbreak via adaptive dense-to-sparse constrained optimization. arXiv preprint arXiv:2405.09113, 2024

    2024 arXiv

  18. [26]

    Simon Geisler, Tom Wollschläger, M. H. I. Abdalla, Johannes Gasteiger, and Stephan Gün- nemann. Attacking large language models with projected gradient descent. arXiv preprint arXiv:2402.09154, 2024

    2024 arXiv

  19. [27]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451, 2024

    2024 arXiv

  20. [28]

    Hijacking large language models via adversarial in-context learning

    Yao Qiang, Xiangyu Zhou, and Dongxiao Zhu. Hijacking large language models via adversarial in-context learning. arXiv preprint arXiv:2311.09948, 2024

    2024 arXiv

  21. [29]

    Prp: Propagating universal perturbations to attack large language model guard-rails

    Neal Mangaokar, Ashish Hooda, Jihye Choi, Shreyas Chandrashekaran, Kassem Fawaz, Somesh Jha, and Atul Prakash. Prp: Propagating universal perturbations to attack large language model guard-rails. arXiv preprint arXiv:2402.15911, 2024

    2024 arXiv

  22. [30]

    Asetf: A novel method for jailbreak attack on llms through translate suffix embeddings

    Hao Wang, Hao Li, Minlie Huang, and Lei Sha. Asetf: A novel method for jailbreak attack on llms through translate suffix embeddings. arXiv preprint arXiv:2402.16006, 2024

    2024 arXiv

  23. [31]

    Automatic and universal prompt injection attacks against large language models

    Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. Automatic and universal prompt injection attacks against large language models. arXiv preprint arXiv:2403.04957, 2024

    2024 arXiv

  24. [32]

    Improved generation of adversarial examples against safety-aligned llms

    Qizhang Li, Yiwen Guo, Wangmeng Zuo, and Hao Chen. Improved generation of adversarial examples against safety-aligned llms. arXiv preprint arXiv:2405.20778, 2024

    2024 arXiv

  25. [33]

    Weak-to-strong jailbreaking on large language models

    Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du, Lei Li, Yu-Xiang Wang, and William Yang Wang. Weak-to-strong jailbreaking on large language models. arXiv preprint arXiv:2401.17256, 2024. 12

    2024 arXiv

  26. [34]

    Analyzing the inher- ent response tendency of llms: Real-world instructions-driven jailbreak

    Yanrui Du, Sendong Zhao, Ming Ma, Yuhan Chen, and Bing Qin. Analyzing the inher- ent response tendency of llms: Real-world instructions-driven jailbreak. arXiv preprint arXiv:2312.04127, 2024

    2024 arXiv

  27. [35]

    Don’t say no: Jailbreaking llm by suppressing refusal

    Yukai Zhou and Wenjie Wang. Don’t say no: Jailbreaking llm by suppressing refusal. arXiv preprint arXiv:2404.16369, 2024

    2024 arXiv

  28. [36]

    Lockpicking llms: A logit-based jailbreak using token-level manipulation

    Yuxi Li, Yi Liu, Yuekang Li, Ling Shi, Gelei Deng, Shengquan Chen, and Kailong Wang. Lockpicking llms: A logit-based jailbreak using token-level manipulation. arXiv preprint arXiv:2405.13068, 2024

    2024 arXiv

  29. [37]

    Make them spill the beans! coercive knowledge extraction from (production) llms

    Zhuo Zhang, Guangyu Shen, Guanhong Tao, Siyuan Cheng, and Xiangyu Zhang. Make them spill the beans! coercive knowledge extraction from (production) llms. arXiv preprint arXiv:2312.04782, 2023

    2023 arXiv

  30. [38]

    Cold-attack: Jailbreaking llms with stealthiness and controllability

    Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. Cold-attack: Jailbreaking llms with stealthiness and controllability. arXiv preprint arXiv:2402.08679, 2024

    2024 arXiv

  31. [39]

    Fast adversarial attacks on language models in one gpu minute

    Vinu Sankar Sadasivan, Shoumik Saha, Gaurang Sriramanan, Priyatham Kattakinda, Atoosa Chegini, and Soheil Feizi. Fast adversarial attacks on language models in one gpu minute. arXiv preprint arXiv:2402.15570, 2024

    2024 arXiv

  32. [40]

    Jailbreaking leading safety-aligned llms with simple adaptive attacks

    Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety-aligned llms with simple adaptive attacks. arXiv preprint arXiv:2404.02151, 2024

    2024 arXiv

  33. [41]

    Shadow alignment: The ease of subverting safely-aligned language models

    Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02949, 2023

    2023 arXiv

  34. [42]

    Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023

    2023 arXiv

  35. [43]

    Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b

    Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. arXiv preprint arXiv:2310.20624, 2024

    2024 arXiv

  36. [44]

    Removing rlhf protections in gpt-4 via fine-tuning

    Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Removing rlhf protections in gpt-4 via fine-tuning. arXiv preprint arXiv:2311.05553, 2024

    2024 arXiv

  37. [45]

    Learning diverse attacks on large language models for robust red-teaming and safety tuning

    Seanie Lee, Minsu Kim, Lynn Cherif, David Dobre, Juho Lee, Sung Ju Hwang, Kenji Kawaguchi, Gauthier Gidel, Yoshua Bengio, Nikolay Malkin, and Moksh Jain. Learning diverse attacks on large language models for robust red-teaming and safety tuning. arXiv preprint arXiv:2405.18540, 2024

    2024 arXiv

  38. [46]

    Exploiting novel gpt-4 apis

    Kellin Pelrine, Mohammad Taufeeque, Michał Zaj ˛ ac, Euan McLean, and Adam Gleave. Exploiting novel gpt-4 apis. arXiv preprint arXiv:2312.14302, 2024

    2024 arXiv

  39. [47]

    Vazquez, Ulisse Mini, and Monte MacDiarmid

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248, 2024

    2024 arXiv

  40. [48]

    Trojan activation attack: Red-teaming large language models using activation steering for safety-alignment

    Haoran Wang and Kai Shu. Trojan activation attack: Red-teaming large language models using activation steering for safety-alignment. arXiv preprint arXiv:2311.09433, 2024

    2024 arXiv

  41. [49]

    Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space

    Leo Schwinn, David Dobre, Sophie Xhonneux, Gauthier Gidel, and Stephan Gunnemann. Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space. arXiv preprint arXiv:2402.09063, 2024

    2024 arXiv

  42. [50]

    Loft: Local proxy fine-tuning for improving transfer- ability of adversarial attacks against large language model

    Muhammad Ahmed Shah, Roshan Sharma, Hira Dhamyal, Raphael Olivier, Ankit Shah, Joseph Konan, Dareen Alharthi, Hazim T Bukhari, Massa Baali, Soham Deshmukh, Michael Kuhlmann, Bhiksha Raj, and Rita Singh. Loft: Local proxy fine-tuning for improving transfer- ability of adversari...

    2023 arXiv

  43. [51]

    McCabe, Tanay Wakhare, Yejin Kim, H

    Rimon Melamed, Lucas H. McCabe, Tanay Wakhare, Yejin Kim, H. Howie Huang, and Enric Boix-Adsera. Prompts have evil twins. arXiv preprint arXiv:2311.07064, 2024

    2024 arXiv

  44. [52]

    Pal: Proxy-guided black-box attack on large language models

    Chawin Sitawarin, Norman Mu, David Wagner, and Alexandre Araujo. Pal: Proxy-guided black-box attack on large language models. arXiv preprint arXiv:2402.09674, 2024

    2024 arXiv

  45. [53]

    Advprompter: Fast adaptive adversarial prompting for llms

    Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. Advprompter: Fast adaptive adversarial prompting for llms. arXiv preprint arXiv:2404.16873, 2024

    2024 arXiv

  46. [54]

    Uncovering safety risks of large language models through concept activation vector

    Zhihao Xu, Ruixuan Huang, Changyu Chen, Shuai Wang, and Xiting Wang. Uncovering safety risks of large language models through concept activation vector. arXiv preprint arXiv:2404.12038, 2024

    2024 arXiv

  47. [55]

    arXiv preprint arXiv:2403.16432, 2024

    Yue Xu and Wenjie Wang.linkprompt: Natural and universal adversarial attacks on prompt- based language models. arXiv preprint arXiv:2403.16432, 2024

    2024 arXiv

  48. [56]

    Zheng-Xin Yong, Cristina Menghini, and Stephen H. Bach. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446, 2024

    2024 arXiv

  49. [57]

    Multilingual jailbreak challenges in large language models

    Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual jailbreak challenges in large language models. arXiv preprint arXiv:2310.06474, 2024

    2024 arXiv

  50. [58]

    Cognitive overload: Jailbreaking large language models with overloaded logical thinking

    Nan Xu, Fei Wang, Ben Zhou, Bang Zheng Li, Chaowei Xiao, and Muhao Chen. Cognitive overload: Jailbreaking large language models with overloaded logical thinking. arXiv preprint arXiv:2311.09827, 2024

    2024 arXiv

  51. [59]

    A cross-language investigation into jailbreak attacks in large language models

    Jie Li, Yi Liu, Chongyang Liu, Ling Shi, Xiaoning Ren, Yaowen Zheng, Yang Liu, and Yinxing Xue. A cross-language investigation into jailbreak attacks in large language models. arXiv preprint arXiv:2401.16765, 2024

    2024 arXiv

  52. [60]

    Does refusal training in llms generalize to the past tense? arXiv preprint arXiv:2407.11969, 2024

    Maksym Andriushchenko and Nicolas Flammarion. Does refusal training in llms generalize to the past tense? arXiv preprint arXiv:2407.11969, 2024

    2024 arXiv

  53. [61]

    Pappas, and Eric Wong

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2024

    2024 arXiv

  54. [62]

    Tree of attacks: Jailbreaking black-box llms automatically

    Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically. arXiv preprint arXiv:2312.02119, 2024

    2024 arXiv

  55. [63]

    Distract large language models for automatic jailbreak attack

    Zeguan Xiao, Yan Yang, Guanhua Chen, and Yun Chen. Distract large language models for automatic jailbreak attack. arXiv preprint arXiv:2403.08424, 2024

    2024 arXiv

  56. [64]

    Gpt-4 jailbreaks itself with near-perfect success using self-explanation

    Govind Ramesh, Yao Dou, and Wei Xu. Gpt-4 jailbreaks itself with near-perfect success using self-explanation. arXiv preprint arXiv:2405.13077, 2024

    2024 arXiv

  57. [65]

    Yixin Cheng, Markos Georgopoulos, V olkan Cevher, and Grigorios G. Chrysos. Leverag- ing the context through multi-round interactions for jailbreaking attacks. arXiv preprint arXiv:2402.09177, 2024

    2024 arXiv

  58. [66]

    Large language models are vulnerable to bait-and-switch attacks for generating harmful content

    Federico Bianchi and James Zou. Large language models are vulnerable to bait-and-switch attacks for generating harmful content. arXiv preprint arXiv:2402.13926, 2024

    2024 arXiv

  59. [67]

    Foot in the door: Understanding large language model jailbreaking via cognitive psychology

    Zhenhua Wang, Wei Xie, Baosheng Wang, Enze Wang, Zhiwen Gui, Shuoyoucheng Ma, and Kai Chen. Foot in the door: Understanding large language model jailbreaking via cognitive psychology. arXiv preprint arXiv:2402.15690, 2024

    2024 arXiv

  60. [68]

    Speak out of turn: Safety vulnerability of large language models in multi-turn dialogue

    Zhenhong Zhou, Jiuyang Xiang, Haopeng Chen, Quan Liu, Zherui Li, and Sen Su. Speak out of turn: Safety vulnerability of large language models in multi-turn dialogue. arXiv preprint arXiv:2402.17262, 2024

    2024 arXiv

  61. [69]

    Great, now write an article about that: The crescendo multi-turn llm jailbreak attack

    Mark Russinovich, Ahmed Salem, and Ronen Eldan. Great, now write an article about that: The crescendo multi-turn llm jailbreak attack. arXiv preprint arXiv:2404.01833, 2024. 14

    2024 arXiv

  62. [70]

    Chain of attack: a semantic-driven contextual multi-turn attacker for llm

    Xikang Yang, Xuehai Tang, Songlin Hu, and Jizhong Han. Chain of attack: a semantic-driven contextual multi-turn attacker for llm. arXiv preprint arXiv:2405.05610, 2024

    2024 arXiv

  63. [71]

    How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms

    Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373, 2024

    2024 arXiv

  64. [72]

    Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher

    Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463, 2024

    2024 arXiv

  65. [73]

    Jailbreaking proprietary large language models using word substitution cipher

    Divij Handa, Advait Chirmule, Bimal Gajera, and Chitta Baral. Jailbreaking proprietary large language models using word substitution cipher. arXiv preprint arXiv:2402.10601, 2024

    2024

  66. [74]

    Brian R. Y . Huang, Maximilian Li, and Leonard Tang. Endless jailbreaks with bijection learning. arXiv preprint arXiv:2410.01294, 2024

    2024 arXiv

  67. [75]

    Artprompt: Ascii art-based jailbreak attacks against aligned llms

    Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran. Artprompt: Ascii art-based jailbreak attacks against aligned llms. arXiv preprint arXiv:2402.11753, 2024

    2024 arXiv

  68. [76]

    Enhancing jailbreak attack against large language models through silent tokens

    Jiahao Yu, Haozheng Luo, Jerry Yao-Chieh Hu, Wenbo Guo, Han Liu, and Xinyu Xing. Enhancing jailbreak attack against large language models through silent tokens. arXiv preprint arXiv:2405.20653, 2024

    2024 arXiv

  69. [77]

    Improved few-shot jailbreaking can circumvent aligned language models and their defenses

    Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Jing Jiang, and Min Lin. Improved few-shot jailbreaking can circumvent aligned language models and their defenses. arXiv preprint arXiv:2406.01288, 2024

    2024 arXiv

  70. [78]

    Autobreach: Universal and adaptive jailbreaking with efficient wordplay-guided optimization

    Jiawei Chen, Xiao Yang, Zhengwei Fang, Yu Tian, Yinpeng Dong, Zhaoxia Yin, and Hang Su. Autobreach: Universal and adaptive jailbreaking with efficient wordplay-guided optimization. arXiv preprint arXiv:2405.19668, 2024

    2024 arXiv

  71. [79]

    Play guessing game with llm: Indirect jailbreak attack with implicit clues

    Zhiyuan Chang, Mingyang Li, Yi Liu, Junjie Wang, Qing Wang, and Yang Liu. Play guessing game with llm: Indirect jailbreak attack with implicit clues. arXiv preprint arXiv:2402.09091, 2024

    2024 arXiv

  72. [80]

    Drattack: Prompt decomposition and reconstruction makes powerful llm jailbreakers

    Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. Drattack: Prompt decomposition and reconstruction makes powerful llm jailbreakers. arXiv preprint arXiv:2402.16914, 2024

    2024 arXiv

  73. [81]

    Making them ask and answer: Jailbreaking large language models in few queries via disguise and reconstruction

    Tong Liu, Yingjie Zhang, Zhe Zhao, Yinpeng Dong, Guozhu Meng, and Kai Chen. Making them ask and answer: Jailbreaking large language models in few queries via disguise and reconstruction. arXiv preprint arXiv:2402.18104, 2024

    2024 arXiv

  74. [82]

    Codeattack: Revealing safety generalization challenges of large language models via code completion

    Qibing Ren, Chang Gao, Jing Shao, Junchi Yan, Xin Tan, Wai Lam, and Lizhuang Ma. Codeattack: Revealing safety generalization challenges of large language models via code completion. arXiv preprint arXiv:2403.07865, 2024

    2024 arXiv

  75. [83]

    Codechameleon: Personalized encryption framework for jailbreaking large language models

    Huijie Lv, Xiao Wang, Yuansen Zhang, Caishuang Huang, Shihan Dou, Junjie Ye, Tao Gui, Qi Zhang, and Xuanjing Huang. Codechameleon: Personalized encryption framework for jailbreaking large language models. arXiv preprint arXiv:2402.16717, 2024

    2024 arXiv

  76. [84]

    A wolf in sheep’s clothing: Generalized nested jailbreak prompts can fool large language models easily

    Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, and Shujian Huang. A wolf in sheep’s clothing: Generalized nested jailbreak prompts can fool large language models easily. arXiv preprint arXiv:2311.08268, 2024

    2024 arXiv

  77. [85]

    Deepin- ception: Hypnotize large language model to be jailbreaker

    Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. Deepin- ception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191, 2024

    2024 arXiv

  78. [86]

    Fig- ure it out: Analyzing-based jailbreak attack on large language models

    Shi Lin, Rongchang Li, Xun Wang, Changting Lin, Wenpeng Xing, and Meng Han. Fig- ure it out: Analyzing-based jailbreak attack on large language models. arXiv preprint arXiv:2407.16205, 2024. 15

    2024 arXiv

  79. [87]

    Exploiting programmatic behavior of llms: Dual-use through standard secu- rity attacks

    Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. Exploiting programmatic behavior of llms: Dual-use through standard secu- rity attacks. arXiv preprint arXiv:2302.05733, 2023

    2023 arXiv

  80. [88]

    Harris, and Marcel Carlsson

    Dongyu Yao, Jianshu Zhang, Ian G. Harris, and Marcel Carlsson. Fuzzllm: A novel and universal fuzzing framework for proactively discovering jailbreak vulnerabilities in large language models. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal P...

    2024

  81. [89]

    Hidden you malicious goal into benign narratives: Jail- break large language models through logic chain injection

    Zhilong Wang, Yebo Cao, and Peng Liu. Hidden you malicious goal into benign narratives: Jail- break large language models through logic chain injection. arXiv preprint arXiv:2404.04849, 2024

    2024 arXiv

  82. [90]

    do anything now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.03825, 2024

    2024 arXiv

  83. [91]

    Jailbreaking chatgpt via prompt engineering: An empirical study

    Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, Kailong Wang, and Yang Liu. Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860, 2024

    2024 arXiv

  84. [92]

    Jailbreak and guard aligned language models with only few in-context demonstrations

    Zeming Wei, Yifei Wang, Ang Li, Yichuan Mo, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387, 2024

    2024 arXiv

  85. [93]

    Masterkey: Automated jailbreaking of large language model chatbots

    Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. Masterkey: Automated jailbreaking of large language model chatbots. In Proceedings 2024 Network and Distributed System Security Symposium, 2024

    2024

  86. [94]

    Scalable and transferable black-box jailbreaks for language models via persona modulation

    Rusheb Shah, Quentin Feuillade–Montixi, Soroush Pour, Arush Tagade, Stephen Casper, and Javier Rando. Scalable and transferable black-box jailbreaks for language models via persona modulation. arXiv preprint arXiv:2311.03348, 2023

    2023 arXiv

  87. [95]

    Guard: Role-playing to generate natural-language jailbreakings to test guideline adherence of large language models

    Haibo Jin, Ruoxi Chen, Andy Zhou, Yang Zhang, and Haohan Wang. Guard: Role-playing to generate natural-language jailbreakings to test guideline adherence of large language models. arXiv preprint arXiv:2402.03299, 2024

    2024

  88. [96]

    Autodan: Interpretable gradient-based adversarial attacks on large language models

    Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. Autodan: Interpretable gradient-based adversarial attacks on large language models. arXiv preprint arXiv:2310.15140, 2023

    2023 arXiv

  89. [97]

    All in how you ask for it: Simple black-box method for jailbreak attacks

    Kazuhiro Takemoto. All in how you ask for it: Simple black-box method for jailbreak attacks. Applied Sciences, 14(9):3558, April 2024

    2024

  90. [98]

    Open sesame! universal black box jailbreaking of large language models

    Raz Lapid, Ron Langberg, and Moshe Sipper. Open sesame! universal black box jailbreaking of large language models. arXiv preprint arXiv:2309.01446, 2024

    2024 arXiv

  91. [99]

    Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts

    Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253, 2024

    2024 arXiv

  92. [100]

    Query-efficient black-box red teaming via bayesian optimization

    Deokjae Lee, JunYeong Lee, Jung-Woo Ha, Jin-Hwa Kim, Sang-Woo Lee, Hwaran Lee, and Hyun Oh Song. Query-efficient black-box red teaming via bayesian optimization. arXiv preprint arXiv:2305.17444, 2023

    2023 arXiv

  93. [101]

    Semantic mirror jailbreak: Genetic algorithm based jailbreak prompts against open-source llms

    Xiaoxia Li, Siyuan Liang, Jiyi Zhang, Han Fang, Aishan Liu, and Ee-Chien Chang. Semantic mirror jailbreak: Genetic algorithm based jailbreak prompts against open-source llms. arXiv preprint arXiv:2402.14872, 2024

    2024 arXiv

  94. [102]

    Jailbreaking prompt attack: A controllable adversarial attack against diffusion models

    Jiachen Ma, Anda Cao, Zhiqing Xiao, Yijiang Li, Jie Zhang, Chao Ye, and Junbo Zhao. Jailbreaking prompt attack: A controllable adversarial attack against diffusion models. arXiv preprint arXiv:2404.02928, 2024

    2024 arXiv

  95. [103]

    Surrogateprompt: Bypassing the safety filter of text-to-image models via substitution

    Zhongjie Ba, Jieming Zhong, Jiachen Lei, Peng Cheng, Qinglong Wang, Zhan Qin, Zhibo Wang, and Kui Ren. Surrogateprompt: Bypassing the safety filter of text-to-image models via substitution. arXiv preprint arXiv:2309.14122, 2024. 16

    2024 arXiv

  96. [104]

    Sneakyprompt: Jailbreaking text-to-image generative models

    Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao. Sneakyprompt: Jailbreaking text-to-image generative models. arXiv preprint arXiv:2305.12082, 2023

    2023 arXiv

  97. [105]

    Ring-a-bell! how reliable are concept removal methods for diffusion models? arXiv preprint arXiv:2310.10012, 2024

    Yu-Lin Tsai, Chia-Yi Hsu, Chulin Xie, Chih-Hsun Lin, Jia-You Chen, Bo Li, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang. Ring-a-bell! how reliable are concept removal methods for diffusion models? arXiv preprint arXiv:2310.10012, 2024

    2024 arXiv

  98. [106]

    Divide-and-conquer attack: Harnessing the power of llm to bypass safety filters of text-to-image models

    Yimo Deng and Huangxun Chen. Divide-and-conquer attack: Harnessing the power of llm to bypass safety filters of text-to-image models. arXiv preprint arXiv:2312.07130, 2024

    2024 arXiv

  99. [107]

    Bspa: Exploring black-box stealthy prompt attacks against image generators

    Yu Tian, Xiao Yang, Yinpeng Dong, Heming Yang, Hang Su, and Jun Zhu. Bspa: Exploring black-box stealthy prompt attacks against image generators. arXiv preprint arXiv:2402.15218, 2024

    2024 arXiv

  100. [108]

    Flirt: Feedback loop in-context red teaming

    Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan, and Rahul Gupta. Flirt: Feedback loop in-context red teaming. arXiv preprint arXiv:2308.04265, 2023

    2023 arXiv

  101. [109]

    Visual adversarial examples jailbreak aligned large language models

    Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. arXiv preprint arXiv:2306.13213, 2023

    2023 arXiv

  102. [110]

    Figstep: Jailbreaking large vision-language models via typographic visual prompts

    Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. Figstep: Jailbreaking large vision-language models via typographic visual prompts. arXiv preprint arXiv:2311.05608, 2023

    2023 arXiv

  103. [111]

    Abusing images and sounds for indirect instruction injection in multi-modal llms

    Eugene Bagdasaryan, Tsung-Yin Hsieh, Ben Nassi, and Vitaly Shmatikov. Abusing images and sounds for indirect instruction injection in multi-modal llms. arXiv preprint arXiv:2307.10490, 2023

    2023 arXiv

  104. [112]

    Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models

    Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh. Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models. arXiv preprint arXiv:2307.14539, 2023

    2023 arXiv

  105. [113]

    Adversarial illusions in multi-modal embeddings

    Tingwei Zhang, Rishi Jha, Eugene Bagdasaryan, and Vitaly Shmatikov. Adversarial illusions in multi-modal embeddings. arXiv preprint arXiv:2308.11804, 2024

    2024 arXiv

  106. [114]

    Images are achilles’ heel of alignment: Exploiting visual vulnerabilities for jailbreaking multimodal large language models

    Yifan Li, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. Images are achilles’ heel of alignment: Exploiting visual vulnerabilities for jailbreaking multimodal large language models. arXiv preprint arXiv:2403.09792, 2024

    2024 arXiv

  107. [115]

    Jailbreak vision language models via bi-modal adversarial prompt

    Zonghao Ying, Aishan Liu, Tianyuan Zhang, Zhengmin Yu, Siyuan Liang, Xianglong Liu, and Dacheng Tao. Jailbreak vision language models via bi-modal adversarial prompt. arXiv preprint arXiv:2406.04031, 2024

    2024 arXiv

  108. [116]

    Jailguard: A universal detection framework for llm prompt-based attacks

    Xiaoyu Zhang, Cen Zhang, Tianlin Li, Yihao Huang, Xiaojun Jia, Ming Hu, Jie Zhang, Yang Liu, Shiqing Ma, and Chao Shen. Jailguard: A universal detection framework for llm prompt-based attacks. arXiv preprint arXiv:2312.10766, 2024

    2024 arXiv

  109. [117]

    Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Itay Yona, Eric Wallace, David Rolnick, and Florian Tramèr

    Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Itay Yona, Eric Wallace, David Rolnick, and Florian Tramèr. Stealing part of a production language model. ...

    2024 arXiv

  110. [118]

    Extracting training data from large language models

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-V oss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. arXiv preprint arXiv:2012.07805, 2021

    2012 arXiv

  111. [119]

    Quantifying association capabilities of large language models and its implications on privacy leakage

    Hanyin Shao, Jie Huang, Shen Zheng, and Kevin Chen-Chuan Chang. Quantifying association capabilities of large language models and its implications on privacy leakage. arXiv preprint arXiv:2305.12707, 2024

    2024 arXiv

  112. [120]

    Do language models plagiarize? In Proceedings of the ACM Web Conference 2023, April 2023

    Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. Do language models plagiarize? In Proceedings of the ACM Web Conference 2023, April 2023. 17

    2023

  113. [121]

    Counterfactual memorization in neural language models

    Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938, 2023

    2023 arXiv

  114. [122]

    Analyzing leakage of personally identifiable information in language models

    Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella- Béguelin. Analyzing leakage of personally identifiable information in language models. arXiv preprint arXiv:2302.00539, 2023

    2023 arXiv

  115. [123]

    Propile: Probing privacy leakage in large language models

    Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, and Seong Joon Oh. Propile: Probing privacy leakage in large language models. arXiv preprint arXiv:2307.01881, 2023

    2023 arXiv

  116. [124]

    Feder Cooper, Daphne Ippolito, Christopher A

    Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311...

    2023 arXiv

  117. [125]

    Multi-step jailbreaking privacy attacks on chatgpt

    Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197, 2023

    2023 arXiv

  118. [126]

    Bag of tricks for training data extraction from language models

    Weichen Yu, Tianyu Pang, Qian Liu, Chao Du, Bingyi Kang, Yan Huang, Min Lin, and Shuicheng Yan. Bag of tricks for training data extraction from language models. arXiv preprint arXiv:2302.04460, 2023

    2023 arXiv

  119. [127]

    Unveiling memorization in code models

    Zhou Yang, Zhipeng Zhao, Chenyu Wang, Jieke Shi, Dongsun Kim, Donggyun Han, and David Lo. Unveiling memorization in code models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, volume 33, pages 1–13, April 2024

    2024

  120. [128]

    Prompt stealing attacks against large language models

    Zeyang Sha and Yang Zhang. Prompt stealing attacks against large language models. arXiv preprint arXiv:2402.12959, 2024

    2024 arXiv

  121. [129]

    Pleak: Prompt leaking attacks against large language model applications

    Bo Hui, Haolin Yuan, Neil Gong, Philippe Burlina, and Yinzhi Cao. Pleak: Prompt leaking attacks against large language model applications. arXiv preprint arXiv:2405.06823, 2024

    2024 arXiv

  122. [130]

    Blind baselines beat membership inference attacks for foundation models

    Debeshee Das, Jie Zhang, and Florian Tramèr. Blind baselines beat membership inference attacks for foundation models. arXiv preprint arXiv:2406.16201, 2024

    2024 arXiv

  123. [131]

    Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841, 2024

    Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettle- moyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841, 2024

    2024 arXiv

  124. [132]

    Sampling-based pseudo- likelihood for membership inference attacks

    Masahiro Kaneko, Youmi Ma, Yuki Wata, and Naoaki Okazaki. Sampling-based pseudo- likelihood for membership inference attacks. arXiv preprint arXiv:2404.11262, 2024

    2024 arXiv

  125. [133]

    Membership inference attacks against language models via neighbourhood comparison

    Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. Membership inference attacks against language models via neighbourhood comparison. arXiv preprint arXiv:2305.18462, 2023

    2023 arXiv

  126. [134]

    Jailbreak attacks and defenses against large language models: A survey

    Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. Jailbreak attacks and defenses against large language models: A survey. arXiv preprint arXiv:2407.04295, 2024

    2024 arXiv

  127. [135]

    Comprehensive assessment of jailbreak attacks against llms

    Junjie Chu, Yugeng Liu, Ziqing Yang, Xinyue Shen, Michael Backes, and Yang Zhang. Comprehensive assessment of jailbreak attacks against llms. arXiv preprint arXiv:2402.05668, 2024

    2024 arXiv

  128. [136]

    Attacks, defenses and evaluations for llm conversation safety: A survey

    Zhichen Dong, Zhanhui Zhou, Chao Yang, Jing Shao, and Yu Qiao. Attacks, defenses and evaluations for llm conversation safety: A survey. arXiv preprint arXiv:2402.09283, 2024

    2024 arXiv

  129. [137]

    A comprehensive study of jailbreak attack versus defense for large language models

    Zihao Xu, Yi Liu, Gelei Deng, Yuekang Li, and Stjepan Picek. A comprehensive study of jailbreak attack versus defense for large language models. arXiv preprint arXiv:2402.13457, 2024

    2024 arXiv

  130. [138]

    Recent advancements in llm red-teaming: Techniques, defenses, and ethical considerations

    Tarun Raheja and Nilay Pochhi. Recent advancements in llm red-teaming: Techniques, defenses, and ethical considerations. arXiv preprint arXiv:2410.09097, 2024. 18

    2024 arXiv

  131. [139]

    Survey of vulnerabilities in large language models revealed by adversarial attacks

    Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael Abu- Ghazaleh. Survey of vulnerabilities in large language models revealed by adversarial attacks. arXiv preprint arXiv:2310.10844, 2023

    2023 arXiv

  132. [140]

    Breaking down the defenses: A comparative survey of attacks on large language models

    Arijit Ghosh Chowdhury, Md Mofijul Islam, Vaibhav Kumar, Faysal Hossain Shezan, Vaibhav Kumar, Vinija Jain, and Aman Chadha. Breaking down the defenses: A comparative survey of attacks on large language models. arXiv preprint arXiv:2403.04786, 2024

    2024

  133. [141]

    Opera- tionalizing a threat model for red-teaming large language models (llms)

    Apurv Verma, Satyapriya Krishna, Sebastian Gehrmann, Madhavan Seshadri, Anu Pradhan, Tom Ault, Leslie Barrett, David Rabinowitz, John Doucette, and NhatHai Phan. Opera- tionalizing a threat model for red-teaming large language models (llms). arXiv preprint arXiv:2407.14937, 2024

    2024

  134. [142]

    A survey on large language model (llm) security and privacy: The good, the bad, and the ugly

    Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, 4(2):100211, June 2024

    2024

  135. [143]

    A survey of attacks on large vision-language models: Resources, advances, and future trends

    Daizong Liu, Mingyu Yang, Xiaoye Qu, Pan Zhou, Yu Cheng, and Wei Hu. A survey of attacks on large vision-language models: Resources, advances, and future trends. arXiv preprint arXiv:2407.07403, 2024

    2024 arXiv

  136. [144]

    From llms to mllms: Exploring the landscape of multimodal jailbreaking

    Siyuan Wang, Zhuohan Long, Zhihao Fan, and Zhongyu Wei. From llms to mllms: Exploring the landscape of multimodal jailbreaking. arXiv preprint arXiv:2406.14859, 2024

    2024 arXiv

  137. [145]

    Jailbreakzoo: Survey, landscapes, and horizons in jailbreaking large language and vision-language models

    Haibo Jin, Leyang Hu, Xinuo Li, Peiyan Zhang, Chonghan Chen, Jun Zhuang, and Haohan Wang. Jailbreakzoo: Survey, landscapes, and horizons in jailbreaking large language and vision-language models. arXiv preprint arXiv:2407.01599, 2024

    2024

  138. [146]

    Safety of multimodal large language models on images and texts

    Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, and Yu Qiao. Safety of multimodal large language models on images and texts. arXiv preprint arXiv:2402.00357, 2024

    2024 arXiv

  139. [147]

    Against the achilles’ heel: A survey on red teaming for generative models

    Lizhi Lin, Honglin Mu, Zenan Zhai, Minghan Wang, Yuxia Wang, Renxi Wang, Jun- jie Gao, Yixuan Zhang, Wanxiang Che, Timothy Baldwin, Xudong Han, and Haonan Li. Against the achilles’ heel: A survey on red teaming for generative models. arXiv preprint arXiv:2404.00629, 2024

    2024 arXiv

  140. [148]

    Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast

    Xiangming Gu, Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Ye Wang, Jing Jiang, and Min Lin. Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast. arXiv preprint arXiv:2402.08567, 2024

    2024 arXiv

  141. [149]

    Cheatagent: Attacking llm-empowered recommender systems via llm agent

    Liang-bo Ning, Shijie Wang, Wenqi Fan, Qing Li, Xin Xu, Hao Chen, and Feiran Huang. Cheatagent: Attacking llm-empowered recommender systems via llm agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2284–2295, 2024

    2024

  142. [150]

    Badagent: Inserting and activating backdoor attacks in llm agents

    Yifei Wang, Dizhan Xue, Shengjie Zhang, and Shengsheng Qian. Badagent: Inserting and activating backdoor attacks in llm agents. arXiv preprint arXiv:2406.03007, 2024

    2024 arXiv

  143. [151]

    Watch out for your agents! investigating backdoor threats to llm-based agents

    Wenkai Yang, Xiaohan Bi, Yankai Lin, Sishuo Chen, Jie Zhou, and Xu Sun. Watch out for your agents! investigating backdoor threats to llm-based agents. arXiv preprint arXiv:2402.11208, 2024

    2024 arXiv

  144. [152]

    Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases

    Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases. arXiv preprint arXiv:2407.12784, 2024

    2024 arXiv

  145. [153]

    Flooding spread of manipulated knowl- edge in llm-based multi-agent communities

    Tianjie Ju, Yiting Wang, Xinbei Ma, Pengzhou Cheng, Haodong Zhao, Yulong Wang, Lifeng Liu, Jian Xie, Zhuosheng Zhang, and Gongshen Liu. Flooding spread of manipulated knowl- edge in llm-based multi-agent communities. arXiv preprint arXiv:2407.07791, 2024

    2024 arXiv

  146. [154]

    Jailbreaking text-to- image models with llm-based agents

    Yingkai Dong, Zheng Li, Xiangtao Meng, Ning Yu, and Shanqing Guo. Jailbreaking text-to- image models with llm-based agents. arXiv preprint arXiv:2408.00523, 2024

    2024 arXiv

  147. [155]

    Reinforcement learning-driven llm agent for automated attacks on llms

    Xiangwen Wang, Jie Peng, Kaidi Xu, Huaxiu Yao, and Tianlong Chen. Reinforcement learning-driven llm agent for automated attacks on llms. In Proceedings of the Fifth Workshop on Privacy in Natural Language Processing, pages 170–177, August 2024. 19

    2024

  148. [156]

    Breaking agents: Compromising autonomous llm agents through malfunction amplification

    Boyang Zhang, Yicong Tan, Yun Shen, Ahmed Salem, Michael Backes, Savvas Zannettou, and Yang Zhang. Breaking agents: Compromising autonomous llm agents through malfunction amplification. arXiv preprint arXiv:2407.20859, 2024

    2024 arXiv

  149. [157]

    Sukmin Cho, Soyeong Jeong, Jeongyeon Seo, Taeho Hwang, and Jong C. Park. Typos that broke the rag’s back: Genetic attack on rag pipeline by simulating documents in the wild via low-level perturbations. arXiv preprint arXiv:2404.13948, 2024

    2024 arXiv

  150. [158]

    Machine against the rag: Jamming retrieval-augmented generation with blocker documents

    Avital Shafran, Roei Schuster, and Vitaly Shmatikov. Machine against the rag: Jamming retrieval-augmented generation with blocker documents. arXiv preprint arXiv:2406.05870, 2024

    2024 arXiv

  151. [159]

    Human- imperceptible retrieval poisoning attacks in llm-powered applications

    Quan Zhang, Binqi Zeng, Chijin Zhou, Gwihwan Go, Heyuan Shi, and Yu Jiang. Human- imperceptible retrieval poisoning attacks in llm-powered applications. arXiv preprint arXiv:2404.17196, 2024

    2024 arXiv

  152. [160]

    glue pizza and eat rocks

    Zhen Tan, Chengshuai Zhao, Raha Moraffah, Yifan Li, Song Wang, Jundong Li, Tianlong Chen, and Huan Liu. "glue pizza and eat rocks" – exploiting vulnerabilities in retrieval-augmented generative models. arXiv preprint arXiv:2406.19417, 2024

    2024 arXiv

  153. [161]

    Poisonedrag: Knowledge cor- ruption attacks to retrieval-augmented generation of large language models

    Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. Poisonedrag: Knowledge cor- ruption attacks to retrieval-augmented generation of large language models. arXiv preprint arXiv:2402.07867, 2024

    2024 arXiv

  154. [162]

    Pandora: Jailbreak gpts by retrieval augmented generation poisoning

    Gelei Deng, Yi Liu, Kailong Wang, Yuekang Li, Tianwei Zhang, and Yang Liu. Pandora: Jailbreak gpts by retrieval augmented generation poisoning. arXiv preprint arXiv:2402.08416, 2024

    2024 arXiv

  155. [163]

    Is my data in your retrieval database? membership inference attacks against retrieval augmented generation

    Maya Anderson, Guy Amit, and Abigail Goldsteen. Is my data in your retrieval database? membership inference attacks against retrieval augmented generation. arXiv preprint arXiv:2405.20446, 2024

    2024 arXiv

  156. [164]

    Generating is believing: Membership inference attacks against retrieval-augmented generation

    Yuying Li, Gaoyang Liu, Chen Wang, and Yang Yang. Generating is believing: Membership inference attacks against retrieval-augmented generation. arXiv preprint arXiv:2406.19234, 2024

    2024 arXiv

  157. [165]

    Black-box opinion manipulation attacks to retrieval-augmented generation of large language models

    Zhuo Chen, Jiawei Liu, Haotan Liu, Qikai Cheng, Fan Zhang, Wei Lu, and Xiaozhong Liu. Black-box opinion manipulation attacks to retrieval-augmented generation of large language models. arXiv preprint arXiv:2407.13757, 2024

    2024 arXiv

  158. [166]

    Badrag: Identifying vulnerabilities in retrieval augmented generation of large language models

    Jiaqi Xue, Mengxin Zheng, Yebowen Hu, Fei Liu, Xun Chen, and Qian Lou. Badrag: Identifying vulnerabilities in retrieval augmented generation of large language models. arXiv preprint arXiv:2406.00083, 2024

    2024 arXiv

  159. [167]

    Follow my instruction and spill the beans: Scalable data extraction from retrieval-augmented generation systems

    Zhenting Qi, Hanlin Zhang, Eric Xing, Sham Kakade, and Himabindu Lakkaraju. Follow my instruction and spill the beans: Scalable data extraction from retrieval-augmented generation systems. arXiv preprint arXiv:2402.17840, 2024

    2024 arXiv

  160. [168]

    Unleashing worms and extracting data: Escalating the outcome of attacks against rag-based inference in scale and severity using jailbreaking

    Stav Cohen, Ron Bitton, and Ben Nassi. Unleashing worms and extracting data: Escalating the outcome of attacks against rag-based inference in scale and severity using jailbreaking. arXiv preprint arXiv:2409.08045, 2024

    2024 arXiv

  161. [169]

    Data stealing attacks against large language models via backdooring

    Jiaming He, Guanyu Hou, Xinyue Jia, Yangyang Chen, Wenqi Liao, Yinhang Zhou, and Rang Zhou. Data stealing attacks against large language models via backdooring. Electronics, 13(14):2858, 2024

    2024

  162. [170]

    An embarrassingly simple method to compromise language models

    Jingwei Wang. An embarrassingly simple method to compromise language models. Preprints, June 2024

    2024

  163. [171]

    Sos! soft prompt attack against open-source large language models

    Ziqing Yang, Michael Backes, Yang Zhang, and Ahmed Salem. Sos! soft prompt attack against open-source large language models. arXiv preprint arXiv:2407.03160, 2024

    2024 arXiv

  164. [172]

    Competition report: Finding universal jailbreak backdoors in aligned llms

    Javier Rando, Francesco Croce, Kryštof Mitka, Stepan Shabalin, Maksym Andriushchenko, Nicolas Flammarion, and Florian Tramèr. Competition report: Finding universal jailbreak backdoors in aligned llms. arXiv preprint arXiv:2404.14461, 2024. 20

    2024 arXiv

  165. [173]

    Xuanli He, Jun Wang, Qiongkai Xu, Pasquale Minervini, Pontus Stenetorp, Benjamin I. P. Rubinstein, and Trevor Cohn. Tuba: Cross-lingual transferability of backdoor attacks in llms with instruction tuning. arXiv preprint arXiv:2404.19597, 2024

    2024 arXiv

  166. [174]

    Backdoorllm: A com- prehensive benchmark for backdoor attacks on large language models

    Yige Li, Hanxun Huang, Yunhan Zhao, Xingjun Ma, and Jun Sun. Backdoorllm: A com- prehensive benchmark for backdoor attacks on large language models. arXiv preprint arXiv:2408.12798, 2024

    2024 arXiv

  167. [175]

    Turning generative models degenerate: The power of data poisoning attacks

    Shuli Jiang, Swanand Ravindra Kadhe, Yi Zhou, Farhan Ahmed, Ling Cai, and Nathalie Baracaldo. Turning generative models degenerate: The power of data poisoning attacks. arXiv preprint arXiv:2407.12281, 2024

    2024 arXiv

  168. [176]

    Watch out for your guidance on generation! exploring conditional backdoor attacks against large language models

    Jiaming He, Wenbo Jiang, Guanyu Hou, Wenshu Fan, Rui Zhang, and Hongwei Li. Watch out for your guidance on generation! exploring conditional backdoor attacks against large language models. arXiv preprint arXiv:2404.14795, 2024

    2024 arXiv

  169. [177]

    Poisonprompt: Backdoor attack on prompt-based large language models

    Hongwei Yao, Jian Lou, and Zhan Qin. Poisonprompt: Backdoor attack on prompt-based large language models. arXiv preprint arXiv:2310.12439, 2023

    2023 arXiv

  170. [178]

    Learning to poison large language models during instruction tuning

    Yao Qiang, Xiangyu Zhou, Saleh Zare Zade, Mohammad Amin Roshani, Prashant Khan- duri, Douglas Zytko, and Dongxiao Zhu. Learning to poison large language models during instruction tuning. arXiv preprint arXiv:2402.13459, 2024

    2024 arXiv

  171. [179]

    Qwen3: Think deeper, act faster

    Qwen Team. Qwen3: Think deeper, act faster. 2025. URL: https://qwenlm.github.io/ blog/qwen3/

    2025

  172. [180]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Nicolas Dadoun, Laurent Besacier, Morgane Pauli, Camille Couprie, Alexandre Diffloth, Baptiste Gabriel, Armand Joulin, Edouard Grave, Sylvain Gugger, Jakob...

    2023 arXiv

  173. [181]

    Making them ask and answer: Jailbreaking large language models in few queries via disguise and reconstruction

    Tong Liu, Zhe Zhao, Yinpeng Dong, Guozhu Meng, and Kai Chen. Making them ask and answer: Jailbreaking large language models in few queries via disguise and reconstruction. In 33rd USENIX Security Symposium (USENIX Security 24), pages 4711–4728, Philadelphia, PA, 2024

    2024

  174. [182]

    I". </rule> <rule> Never ever use phrases like

    Anthropic. Claude 3.7 sonnet and claude code. 2025. URL: https://www.anthropic.com/ news/claude-3-7-sonnet . 21 A Appendix A.1 Layer-wise Clustering Analysis Our analysis demonstrates that KMeans clustering introduces minimal randomness across Transformer layers, as evidenced ...

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.