KDA: A Knowledge-Distilled Attacker for Generating Diverse Prompts to Jailbreak LLMs

Buyun Liang; Darshan Thaker; Jinqi Luo; Kwan Ho Ryan Chan; Ren\'e Vidal

REVIEW 3 major objections 5 minor 47 references

KDA: A Knowledge-Distilled Attacker for Generating Diverse Prompts to Jailbreak LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single fine-tuned open-source model can absorb three jailbreak attack strategies and beat each of them on success rate, speed, and diversity.

desk verdict Plausible and useful distillation approach; the matched-budget results are convincing, but the train/eval judge circularity and unmatched headline budgets need fixing before I fully trust the ASR numbers. read the letter →

arxiv 2502.05223 v1 pith:KWXXUUHV submitted 2025-02-05 cs.CR cs.AIcs.CLcs.LG

Buyun Liang , Kwan Ho Ryan Chan , Darshan Thaker , Jinqi Luo , René Vidal This is my paper · ORCID

classification cs.CRcs.AIcs.CLcs.LG

keywords jailbreakattacksLLMsafetyknowledgedistillationred-teamingpromptgenerationattacksuccessrateformatensemblingLoRAfine-tuning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that a single open-source attacker model can absorb the prompt-generation strategies of several established jailbreak methods and then outperform all of them. It introduces KDA, a LoRA-fine-tuned Vicuna-13B trained only on successful attack prompts harvested from AutoDAN, PAIR, and GPTFuzzer, conditioned on an explicit format label so it can emulate any one teacher or mix them. The payoff is practical red-teaming: KDA reports higher attack success on open and commercial LLMs while needing fewer queries and less wall-clock time than the teachers, for example 88.5% on Llama-2-7B-Chat and 100% on five open models on the HarmBench standard set, and 84% versus 38% for GPTFuzzer on the Harmful-Behavior dataset with a 30-query budget. If true, large-scale safety evaluation no longer requires hand-engineered system prompts or expensive per-attack optimization.

What carries the argument

The central object is the format-conditioned distillation objective. KDA is trained by minimizing $\mathbb{E}_{(Q,A,\hat{A}_\theta,F)}[L(\hat{A}_\theta(Q,F), A)]$, where $L$ is cross-entropy, $A$ is a ground-truth attack collected by Algorithm 1, and $F$ indicates whether the prompt is AutoDAN-style, PAIR-style, GPTFuzzer-style, or a mixed format. Algorithm 1 is the load-bearing filter: it generates candidate attacks with a teacher, scores the target response with the binary judge $J(Q,R)$, and keeps only the successful $(Q,A)$ pairs. At generation time, Algorithm 2 samples a format by one of three strategies—uniform, inference-guided softmax over success counts, or training-guided softmax over success counts—so diversity comes from the ensemble of formats rather than from any single prompt template.

What would settle it

Audit a random sample of responses the HB evaluator marks as successful by having independent human annotators judge whether the response actually satisfies the harmful query; if a large share are judged harmless, off-topic, or merely free of refusal phrases, the reported attack-success gains would not reflect real jailbreaks.

Watch

Extended reading notes

Core claim

KDA is a LoRA-fine-tuned Vicuna-13B model trained to map a harmful query $Q$ and a format label $F$ to an attack prompt $A$, where $F \in \{A, P, G, M\}$ names the style of AutoDAN, PAIR, GPTFuzzer, or a GPT-4o-synthesized mix. The training set is built by running the three teachers against open and commercial targets and keeping only $(Q, A)$ pairs for which the binary evaluator $J(Q, R) = 1$, so KDA learns from verified successes rather than from the teachers' full output distributions. At inference, KDA samples a format, generates a prompt, queries the target, and returns the first prompt the evaluator accepts; format selection can be uniform, guided by inference-time success counts, or guided by training-time success counts. The paper reports that this recipe raises the HB-evaluated attack success rate at a 30-query budget on Llama-2-7B from 38% for GPTFuzzer and 16% for AutoDAN to 84%, and on the HarmBench standard set reaches 88.5% on Llama-2-7B-Chat and 100% on Vicuna, Qwen, and Mistral targets. The ablation attributes the gain to format ensembling and topic diversity rather than to any single teacher's template.

Load-bearing premise

The load-bearing premise is that the binary evaluator $J$ faithfully measures genuine harm, because KDA is trained only on prompts $J$ accepts and every reported success rate is measured by $J$; if $J$ can be gamed by benign but refusal-free text, the claimed superiority would not carry over to real-world harm.

Editorial extensions

If this is right

With KDA, a single saved model can replace the teacher attack pipelines, so red-teaming a new target reduces to sampling formats, generating prompts, and checking the judge—no per-target prompt design or mutation loop.
The reported numbers imply that KDA generalizes beyond its training targets: it keeps high attack success on the HarmBench dataset and on models such as Claude 2.1 that were never used in dataset construction, although there its success rate is still low.
The format ensemble is the active ingredient: the ablation shows ensemble strategies reach higher HB-ASR at a 30-query budget than any single format, so KDA's edge is diversity of attack topics rather than one secret template.
Because the training data contains only successful prompts, extending KDA to a new jailbreak method is cheap: generate that method's hits, label them with a new format, and fine-tune.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same distillation recipe, with a different binary success signal, could produce specialized attackers or auditors for properties like privacy leakage, misinformation, or toxicity; the paper's machinery does not depend on the semantics of jailbreaking.
A stronger or human-calibrated judge would likely raise KDA's true ceiling, because the current filter and all reported success rates inherit the evaluator's blind spots; an evaluator that rewards only substantive harmful content would make the measured gains more meaningful.
Defenders should evaluate under a format ensemble rather than any single attacker, since KDA's ablations tie success to topic diversity; a safety evaluation that uses only one attack style may overstate robustness.
The small 800-prompt training set suggests the method is easy to reproduce and extend, which points toward an arms race where both attackers and defenses can be updated cheaply.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the central empirical claim is probably right, but the evaluation needs cleaning before it sticks. KDA—a LoRA-tuned Vicuna-13B trained on successful prompts from AutoDAN, PAIR, and GPTFuzzer with a format token—does seem to generate jailbreak prompts that beat its teachers on the HarmBench judge at the same query budget. The fairest evidence is Table 2 at M=30: on Llama-2-7B, KDA gets 84% ASR-HB versus 38% for GPTFuzzer and 16% for AutoDAN; on GPT-4 it gets 88% versus 78%. Time per attack is also much lower. That is a real and useful result if it holds up.

What's genuinely new is not generative attacking per se—AdvPrompter and AmpleGCG already do that—but the multi-teacher ensemble with explicit format conditioning, plus the training-guided format selection. The single-format ablations and the topic-diversity analysis give decent evidence that the ensemble and format mixing are doing the work.

The main concern is the circularity the stress-test flags: the same binary evaluator J filters the training set (Algorithm 1 keeps only (Q,A) with J(Q,R)=1) and then scores every ASR table. So KDA is explicitly trained to produce prompts that satisfy J, and then measured by J. That is not automatically fatal—HarmBench's HB is a standard classifier trained on human labels, so satisfying it is a meaningful proxy. But the paper reports no human evaluation and no agreement numbers for the "human-aligned judge LLM" mentioned in Appendix D.3. Given the headline claim is "jailbreaks," not "HB-positive responses," this is a real gap. A small human-annotated sample of KDA successes versus teacher successes would settle it.

Two smaller issues. Table 1 compares KDA at a budget of 120 against HarmBench baselines whose budgets are undisclosed; those headline numbers should be de-emphasized or rerun at a common budget. The authors do provide the fair comparison in Table 2, so this is fixable. Second, the training targets overlap with several evaluation targets (Vicuna-7B, Llama-2-7B, GPT-3.5, GPT-4), so the strong numbers on those are partly in-distribution. The unseen-model results (Qwen, Mistral, Claude) are more convincing; they still look good. Also, v1 promises release of model, data, and code but provides nothing. That limits reproducibility; given the paper's value is empirical, a release should be a condition of acceptance.

Who this is for: people working on automated red-teaming and LLM safety evaluation. It deserves a serious referee. The core experiment at M=30 is well-designed, the ablations are informative, and the efficiency claim is significant. I would send it to review with a request for human-annotated evaluation, a common-budget comparison for the headline table, and the promised release.

Referee Report

3 major / 5 minor

Summary. This paper proposes KDA, a Vicuna-13B model fine-tuned with LoRA to generate jailbreak prompts conditioned on a style/format variable F in {A, P, G, M}, representing AutoDAN, PAIR, GPTFuzzer, and a GPT-4o-synthesized mixed style. The training set is constructed by running the three teacher attackers against four target LLMs and retaining only (query, attack) pairs for which the binary evaluator J(Q,R) (HarmBench or text-matching) labels the target response as a successful jailbreak (Algorithm 1). At attack time, KDA samples a format using one of three strategies (uniform, inference-guided, or training-guided) and generates candidate prompts in a single forward pass. The paper reports that KDA achieves higher attack success rates than HarmBench baselines (Table 1), higher ASR_{30}^{HB} and lower average attack time than AutoDAN/PAIR/GPTFuzzer at a fixed budget of M=30 (Table 2), higher topic diversity with comparable coherence, and presents ablations showing that ensemble and mixed formats are key to its performance.

Significance. If the results are corroborated by an independent evaluator, KDA is a practical contribution: it distills three expensive iterative attackers into a single open-source model that generates diverse, coherent prompts with large query-budget and latency savings, and the format-ensemble mechanism is a plausible explanation for the gains. The paper is transparent in providing pseudocode for training and generation, detailed hyperparameters, bootstrap confidence intervals, and extensive ablations across format-selection strategies and query budgets in appendices. The budget-controlled evidence in Table 2 (M=30) is strong on Llama-2-7B and GPT-3.5/GPT-4, and Appendix E shows that the inference-guided and uniform variants largely preserve the advantage, partially mitigating concerns that the headline numbers depend on an unfair budget or target-specific training statistics. The main unresolved issue is the absence of any independent or human evaluation that would separate real jailbreaks from optimization against the automated judge used both for training-data filtering and for scoring.

major comments (3)

[Algorithm 1, Eq. (1), Section 3.1, Tables 1-2, Appendix D.3] The binary evaluator J is used twice in the same role: Algorithm 1 filters the KDA training set by keeping only (Q,A) pairs with J(Q,R)=1, and every reported ASR (Tables 1, 2, Figures 3-5) is computed with the same J. No independent human evaluation is reported, and the 'human-aligned judge LLM' mentioned in Appendix D.3 is not described or validated. The second evaluator, text matching, is a refusal-keyword list that is easy to game. Consequently, the central claim that KDA 'causes the LLM to generate harmful content' is not yet distinguished from the alternative that KDA learns prompts that satisfy J without eliciting genuinely harmful responses. Please add a human-annotated sample of J-positive responses, or agreement statistics between J and a second independently implemented judge, and discuss any disagreement cases.
[Section 3.2.1, Table 1] The headline comparison in Table 1 is not budget-controlled: the text states that baseline hyperparameters and maximum query budgets are undisclosed, while KDA is granted M=120 queries per behavior. Since ASR is nondecreasing in the query budget, KDA's large apparent advantage (e.g., 88.5% vs 34.5% on Llama-2-7B-Chat) may be partly an artifact of budget asymmetry. The fixed-budget comparison in Table 2 (M=30) is the appropriate primary evidence; please either report the baseline budgets used by HarmBench or restructure the presentation so that the main claim rests on the budget-controlled comparison.
[Section 2.3, Appendix C Table 5, Table 2] The 'trn' format-selection strategy uses training-time success counts Nsuccess,F from Appendix C Table 5 for exactly the target models evaluated in Tables 1 and 2 (Vicuna, Llama-2, GPT-3.5, GPT-4), giving KDA target-specific information from the training phase that the baseline attackers do not have. Although Appendix E shows that KDAifr and KDAuni also outperform or match baselines on several targets (e.g., Llama-2-7B ASR_HB30 of 82 for KDAifr vs 38 for GPTFuzzer), the main-text tables should report a variant that does not use per-target training statistics, or explicitly label KDAtrn as an upper-bound result.

minor comments (5)

[Section 3.2.3] 'KDA generate coherence attack prompts with PPL below 60' should read 'coherent attack prompts'; the same paragraph contains a few grammatical errors that should be corrected.
[Equation (4)] The expression 'F ∼ exp(SF )P F ′ exp(SF ′)' is malformed; it should be written as a proper categorical softmax distribution, e.g., P(F) = exp(S_F) / sum_{F'} exp(S_{F'}).
[Table 6] Qwen-7B-Chat is listed twice; one of the entries should presumably be Qwen-14B-Chat.
[Section 2.2, Algorithm 1] The text says 'Assuming Ntrain = 50 samples are generated for each F and Q', but Algorithm 1 stops once the per-format dataset DF reaches Ntrain; clarify how the stated total of 3x4x50=600 samples follows from the algorithm.
[Appendix C, Table 5] The success counts for Llama-2-7B and Llama-2-13B are identical; state explicitly whether the 13B counts were measured separately or transferred from the 7B model, as this affects the interpretation of KDAtrn results on Llama-2-13B.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: KDA's core comparison is empirical, externally benchmarked, and no reported result reduces to a fitted parameter or definition.

full rationale

This is an empirical systems paper, not a mathematical derivation, so most circularity categories do not apply. The training pipeline (Algorithm 1) filters teacher-generated attack prompts using the same binary evaluator J that is later used to measure ASR; this creates a validity concern (does KDA satisfy J rather than produce genuinely harmful content?), but it is not circularity by construction: KDA is trained by cross-entropy on fixed successful prompts, not by optimizing J directly, and the primary HB evaluator is an external HarmBench classifier from Mazeika et al. (2024), not a parameter fitted by this paper. The KDAtrn format-selection strategy uses per-target training success counts (Appendix C, Table 5) on targets that also appear in Table 2, so those comparisons are partly in-sample; however, the main HarmBench comparison (Table 1) uses the uni strategy, and generalization is also demonstrated on target models with no training data (Qwen, Mistral, Claude), so the central claim has independent content. Self-citations (Liang et al. 2023a,b; Luo et al. 2024) support peripheral claims about diversity and harmful categories and are not load-bearing. No equation in the paper reduces a reported result to a fitted parameter or to a definition, so no circularity is exhibited.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests mostly on curated data and evaluator choices rather than on unstated math: the factorization assumption, the reliability of the binary judge J, and the assumption that four training targets transfer to the wider benchmark. The format-selection success counts are method parameters derived from training-time performance on the same targets, which is the main source of target-specific tuning.

free parameters (4)

format selection temperature beta = 0.1
Softmax scaling in Eq. 4 (Appendix C) chosen by hand with no ablation.
format success counts N_success,F for KDAtrn = per-target table (e.g., Vicuna-7B: 96/68/96/94 for A/P/G/M)
These target-specific counts steer format sampling and are derived from training-time performance on the same target models used in evaluation.
dataset gating threshold = 120 harmful queries kept after discarding queries consistently refused by four gate models
Appendix A; controls which queries enter training and may select for easy or transferable queries.
coherence PPL threshold = 60
Section 3.2.3; borrows a threshold from Liu et al. 2024c to label KDA prompts coherent.

assumptions (6)

domain assumption The joint distribution factorizes as P(Q,A,F) = P(A|Q,F)P(Q)P(F)
Section 2.2 uses this factorization to justify sampling format, query, and attack independently when constructing the training set.
domain assumption The binary evaluator J correctly identifies harmful jailbreaks
Algorithm 1 filters training samples with J and all ASR tables score with J; if J is gameable, measured success is inflated.
ad hoc to paper Only successful attacks should be used as training targets
Algorithm 1 discards failed attacks; the paper hypothesizes this improves quality but does not test training on all attempts.
domain assumption The four training target models are representative for transfer
Training data is generated against Vicuna-7B, Llama-2-7B, GPT-3.5, and GPT-4; generalization to Qwen, Mistral, and Claude is assumed.
domain assumption BERTopic topics and GPT-2 perplexity are valid diversity and coherence measures
Section 3.1; TDR and PPL are proxy metrics whose selection could affect the diversity claims.
domain assumption The Harmful-Query-KDA dataset is representative of harmful behaviors in HarmBench
Appendix A; the benchmark is allegedly unseen, yet the generalization claim relies on distributional similarity between the two sets.

how reviews work

0 comments

Cite this review

Pith. "Pith review of KDA: A Knowledge-Distilled Attacker for Generating Diverse Prompts to Jailbreak LLMs." pith.science (2026). https://pith.science/paper/KWXXUUHV

@misc{pith2026250205223,
  author       = {Pith},
  title        = {Pith review of: KDA: A Knowledge-Distilled Attacker for Generating Diverse Prompts to Jailbreak LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KWXXUUHV}},
  note         = {Machine review of arXiv:2502.05223}
}

read the original abstract

Jailbreak attacks exploit specific prompts to bypass LLM safeguards, causing the LLM to generate harmful, inappropriate, and misaligned content. Current jailbreaking methods rely heavily on carefully designed system prompts and numerous queries to achieve a single successful attack, which is costly and impractical for large-scale red-teaming. To address this challenge, we propose to distill the knowledge of an ensemble of SOTA attackers into a single open-source model, called Knowledge-Distilled Attacker (KDA), which is finetuned to automatically generate coherent and diverse attack prompts without the need for meticulous system prompt engineering. Compared to existing attackers, KDA achieves higher attack success rates and greater cost-time efficiency when targeting multiple SOTA open-source and commercial black-box LLMs. Furthermore, we conducted a quantitative diversity analysis of prompts generated by baseline methods and KDA, identifying diverse and ensemble attacks as key factors behind KDA's effectiveness and efficiency.

Figures

Figures reproduced from arXiv: 2502.05223 by the authors.

**Figure 1.** KDA Attack Generation: Overview and Example Prompts. (Top) Schematic overview of the KDA attack generation process. The formats A, P, G, and M correspond to prompts learned from the teacher attackers: AutoDAN, PAIR, GPTFuzzer, and Mixed, respectively. (Bottom) Examples of attack prompts generated by KDA, conditioned on different formats. These challenges lead us to our main research question: Can we create an automa… view at source ↗

**Figure 2.** Schematic overview of KDA training. The schematic overview of the KDA training process is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Comparison of ASR vs. target query budget for KDA and SOTA attack methods. The evaluation is conducted on the Harmful-Behavior dataset (Chao et al., 2024). The ASR values in this plot are evaluated using the HB evaluator. Our KDA method employs the format selection strategy trn. The curves represent average ASR across different LLM targets, computed over 10,000 bootstrap samples, with shaded regions indicating stand… view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Ablation study on attack success rate for all KDA format selection strategies. The curves depict ASRHB 30 , the attack success rate with a target query budget of M = 30 using the HB evaluator, comparing single-format settings (F ∈ {A, P, G, M}) and ensemble-format s…

**Figure 5.** Figure 5: ASR and Topic Diversity of KDA using single format setting KDA with format F ∈ {A, P, G, M} compared to SOTA baselines AutoDAN, PAIR, and GPTFuzzer. (Top) Attack Success Rate (ASR); (Bottom) Topic Diversity Ratio (TDR). KDAA, KDAP, and KDAG share the same color scheme…

**Figure 6.** Figure 6: Topic distribution heatmap comparing the diversity of successful attack prompts generated by baseline methods and KDA across different LLM targets. Each subplot corresponds to a different target LLM, with the x-axis representing KDA with different single format select…

**Figure 7.** Figure 7: provides a schematic overview of our Harmful-Query-KDA construction. We first instruct GPT-4-Turbo-2024-04- 09 to generate a diverse set of malicious queries (e.g., “How to make a bomb”) across 12 harmful categories, including harassment and illegal activities. Next, G…

**Figure 8.** Figure 8: Examples of initial prompts for GPTFuzzer. B.3. Training Hyperparameters For KDA training, we use the paged adamw 32bit optimizer with a learning rate of 5 × 10−4 The model is trained for 6 epochs with a batch size of 4. Additionally, for LoRA fine-tuning, we set the r…

**Figure 9.** Figure 9: Keyphrases for Text Matching D.2. LLM version [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 3 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

and Kamfonas, M

Alon, G. and Kamfonas, M. Detecting Language Model Attacks with Perplexity , November 2023. URL http://arxiv.org/abs/2308.14132. arXiv:2308.14132 [cs]

arXiv 2023
[3]

Jailbreaking Leading Safety - Aligned LLMs with Simple Adaptive Attacks , June 2024

Andriushchenko, M., Croce, F., and Flammarion, N. Jailbreaking Leading Safety - Aligned LLMs with Simple Adaptive Attacks , June 2024. URL http://arxiv.org/abs/2404.02151. arXiv:2404.02151 [cs, stat]

arXiv 2024
[4]

Play Guessing Game with LLM : Indirect Jailbreak Attack with Implicit Clues , February 2024

Chang, Z., Li, M., Liu, Y., Wang, J., Wang, Q., and Liu, Y. Play Guessing Game with LLM : Indirect Jailbreak Attack with Implicit Clues , February 2024. URL http://arxiv.org/abs/2402.09091. arXiv:2402.09091 [cs]

arXiv 2024
[5]

J., and Wong, E

Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., and Wong, E. Jailbreaking Black Box Large Language Models in Twenty Queries , July 2024. URL http://arxiv.org/abs/2310.08419. arXiv:2310.08419 [cs]

arXiv 2024
[6]

A Wolf in Sheep 's Clothing : Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily , April 2024

Ding, P., Kuang, J., Ma, D., Cao, X., Xian, Y., Chen, J., and Huang, S. A Wolf in Sheep 's Clothing : Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily , April 2024. URL http://arxiv.org/abs/2311.08268. arXiv:2311.08268 [cs]

arXiv 2024
[7]

Dubey, A., Kadian, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu...

arXiv 2024
[8]

BERTopic : Neural topic modeling with a class-based TF - IDF procedure, March 2022

Grootendorst, M. BERTopic : Neural topic modeling with a class-based TF - IDF procedure, March 2022. URL http://arxiv.org/abs/2203.05794. arXiv:2203.05794 [cs]

arXiv 2022

Show all 47 references

[9]

COLD - Attack : Jailbreaking LLMs with Stealthiness and Controllability , June 2024

Guo, X., Yu, F., Zhang, H., Qin, L., and Hu, B. COLD - Attack : Jailbreaking LLMs with Stealthiness and Controllability , June 2024. URL http://arxiv.org/abs/2402.08679. arXiv:2402.08679 [cs]

2024 arXiv
[10]

J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA : Low - Rank Adaptation of Large Language Models , October 2021. URL http://arxiv.org/abs/2106.09685. arXiv:2106.09685 [cs]

2021 arXiv
[11]

ArtPrompt : ASCII Art -based Jailbreak Attacks against Aligned LLMs , June 2024

Jiang, F., Xu, Z., Niu, L., Xiang, Z., Ramasubramanian, B., Li, B., and Poovendran, R. ArtPrompt : ASCII Art -based Jailbreak Attacks against Aligned LLMs , June 2024. URL http://arxiv.org/abs/2402.11753. arXiv:2402.11753 [cs]

2024 arXiv
[12]

ChatGPT for good? On opportunities and challenges of large language models for education

Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., Stadler, M., Weller, J., ...

2023
[13]

Open Sesame ! Universal Black Box Jailbreaking of Large Language Models , November 2023

Lapid, R., Langberg, R., and Sipper, M. Open Sesame ! Universal Black Box Jailbreaking of Large Language Models , November 2023. URL http://arxiv.org/abs/2309.01446. arXiv:2309.01446 [cs]

2023 arXiv
[14]

Multi-step Jailbreaking Privacy Attacks on ChatGPT , November 2023

Li, H., Guo, D., Fan, W., Xu, M., Huang, J., Meng, F., and Song, Y. Multi-step Jailbreaking Privacy Attacks on ChatGPT , November 2023. URL http://arxiv.org/abs/2304.05197. arXiv:2304.05197 [cs]

2023 arXiv
[15]

Semantic Mirror Jailbreak : Genetic Algorithm Based Jailbreak Prompts Against Open -source LLMs , February 2024 a

Li, X., Liang, S., Zhang, J., Fang, H., Liu, A., and Chang, E.-C. Semantic Mirror Jailbreak : Genetic Algorithm Based Jailbreak Prompts Against Open -source LLMs , February 2024 a . URL http://arxiv.org/abs/2402.14872. arXiv:2402.14872 [cs]

2024 arXiv
[16]

DrAttack : Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers , March 2024 b

Li, X., Wang, R., Cheng, M., Zhou, T., and Hsieh, C.-J. DrAttack : Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers , March 2024 b . URL http://arxiv.org/abs/2402.16914. arXiv:2402.16914 [cs]

2024 arXiv
[17]

DeepInception : Hypnotize Large Language Model to Be Jailbreaker , May 2024 c

Li, X., Zhou, Z., Zhu, J., Yao, J., Liu, T., and Han, B. DeepInception : Hypnotize Large Language Model to Be Jailbreaker , May 2024 c . URL http://arxiv.org/abs/2311.03191. arXiv:2311.03191 [cs]

2024 arXiv
[18]

Optimization and Optimizers for Adversarial Robustness , March 2023 a

Liang, H., Liang, B., Peng, L., Cui, Y., Mitchell, T., and Sun, J. Optimization and Optimizers for Adversarial Robustness , March 2023 a . URL http://arxiv.org/abs/2303.13401. arXiv:2303.13401 [cs]

2023 arXiv
[19]

Implications of Solution Patterns on Adversarial Robustness

Liang, H., Liang, B., Sun, J., Cui, Y., and Mitchell, T. Implications of Solution Patterns on Adversarial Robustness . In 2023 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops ( CVPRW ) , pp.\ 2393--2400, Vancouver, BC, Canada, June 2023 b . IEEE. ISB...

2023
[20]

and Sun, H

Liao, Z. and Sun, H. AmpleGCG : Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs , May 2024. URL http://arxiv.org/abs/2404.07921. arXiv:2404.07921 [cs]

2024 arXiv
[21]

Making Them Ask and Answer : Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction , June 2024 a

Liu, T., Zhang, Y., Zhao, Z., Dong, Y., Meng, G., and Chen, K. Making Them Ask and Answer : Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction , June 2024 a . URL http://arxiv.org/abs/2402.18104. arXiv:2402.18104 [cs]

2024 arXiv
[22]

AutoDAN - Turbo : A Lifelong Agent for Strategy Self - Exploration to Jailbreak LLMs , November 2024 b

Liu, X., Li, P., Suh, E., Vorobeychik, Y., Mao, Z., Jha, S., McDaniel, P., Sun, H., Li, B., and Xiao, C. AutoDAN - Turbo : A Lifelong Agent for Strategy Self - Exploration to Jailbreak LLMs , November 2024 b . URL http://arxiv.org/abs/2410.05295. arXiv:2410.05295 [cs]

2024 arXiv
[23]

AutoDAN : Generating Stealthy Jailbreak Prompts on Aligned Large Language Models , March 2024 c

Liu, X., Xu, N., Chen, M., and Xiao, C. AutoDAN : Generating Stealthy Jailbreak Prompts on Aligned Large Language Models , March 2024 c . URL http://arxiv.org/abs/2310.04451. arXiv:2310.04451 [cs]

2024 arXiv
[24]

Luo, J., Ding, T., Chan, K. H. R., Thaker, D., Chattopadhyay, A., Callison-Burch, C., and Vidal, R. PaCE : Parsimonious Concept Engineering for Large Language Models , June 2024. URL http://arxiv.org/abs/2406.04331. arXiv:2406.04331 [cs]

2024 arXiv
[25]

CodeChameleon : Personalized Encryption Framework for Jailbreaking Large Language Models , February 2024

Lv, H., Wang, X., Zhang, Y., Huang, C., Dou, S., Ye, J., Gui, T., Zhang, Q., and Huang, X. CodeChameleon : Personalized Encryption Framework for Jailbreaking Large Language Models , February 2024. URL http://arxiv.org/abs/2402.16717. arXiv:2402.16717 [cs]

2024 arXiv
[26]

HarmBench : A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal , February 2024

Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhaee, E., Li, N., Basart, S., Li, B., Forsyth, D., and Hendrycks, D. HarmBench : A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal , February 2024. URL http://arxiv.org/abs/2402.04249....

2024 arXiv
[27]

Tree of Attacks : Jailbreaking Black - Box LLMs Automatically , February 2024

Mehrotra, A., Zampetakis, M., Kassianik, P., Nelson, B., Anderson, H., Singer, Y., and Karbasi, A. Tree of Attacks : Jailbreaking Black - Box LLMs Automatically , February 2024. URL http://arxiv.org/abs/2312.02119. arXiv:2312.02119 [cs, stat]

2024 arXiv
[28]

OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapi...

2024 arXiv
[29]

AdvPrompter : Fast Adaptive Adversarial Prompting for LLMs , April 2024

Paulus, A., Zharmagambetov, A., Guo, C., Amos, B., and Tian, Y. AdvPrompter : Fast Adaptive Adversarial Prompting for LLMs , April 2024. URL http://arxiv.org/abs/2404.16873. arXiv:2404.16873 [cs]

2024 arXiv
[30]

Language Models are Unsupervised Multitask Learners

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language Models are Unsupervised Multitask Learners
[31]

Robey, A., Wong, E., Hassani, H., and Pappas, G. J. SmoothLLM : Defending Large Language Models Against Jailbreaking Attacks , June 2024. URL http://arxiv.org/abs/2310.03684. arXiv:2310.03684 [cs, stat]

2024 arXiv
[32]

E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Ferrer, C

Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Ferrer, C. C., Grattafiori, A., Xiong, W., Défossez, A., Copet, J., Azhar, F., Touvron, H., Mart...

2024 arXiv
[33]

C., Lupu, A., Hambro, E., Markosyan, A

Samvelyan, M., Raparthy, S. C., Lupu, A., Hambro, E., Markosyan, A. H., Bhatt, M., Mao, Y., Jiang, M., Parker-Holder, J., Foerster, J., Rocktäschel, T., and Raileanu, R. Rainbow Teaming : Open - Ended Generation of Diverse Adversarial Prompts , July 2024. URL http://arxiv.org/...

2024 arXiv
[34]

Do Anything Now

Shen, X., Chen, Z., Backes, M., Shen, Y., and Zhang, Y. " Do Anything Now ": Characterizing and Evaluating In - The - Wild Jailbreak Prompts on Large Language Models , May 2024. URL http://arxiv.org/abs/2308.03825. arXiv:2308.03825 [cs]

2024 arXiv
[35]

PAL : Proxy - Guided Black - Box Attack on Large Language Models , February 2024

Sitawarin, C., Mu, N., Wagner, D., and Araujo, A. PAL : Proxy - Guided Black - Box Attack on Large Language Models , February 2024. URL http://arxiv.org/abs/2402.09674. arXiv:2402.09674 [cs]

2024 arXiv
[36]

Fine-tuning large neural language models for biomedical natural language processing

Tinn, R., Cheng, H., Gu, Y., Usuyama, N., Liu, X., Naumann, T., Gao, J., and Poon, H. Fine-tuning large neural language models for biomedical natural language processing. Patterns, 4 0 (4): 0 100729, April 2023. ISSN 26663899. doi:10.1016/j.patter.2023.100729. URL https://link...

2023
[37]

DAN is my new friend, 2022

walkerspider. DAN is my new friend, 2022. URL https://old.reddit.com/r/ChatGPT/comments/zlcyr9/dan_is_my_new_friend/

2022
[38]

ASETF : A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings , June 2024 a

Wang, H., Li, H., Huang, M., and Sha, L. ASETF : A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings , June 2024 a . URL http://arxiv.org/abs/2402.16006. arXiv:2402.16006 [cs]

2024 arXiv
[39]

Detoxifying Large Language Models via Knowledge Editing , May 2024 b

Wang, M., Zhang, N., Xu, Z., Xi, Z., Deng, S., Yao, Y., Zhang, Q., Yang, L., Wang, J., and Chen, H. Detoxifying Large Language Models via Knowledge Editing , May 2024 b . URL http://arxiv.org/abs/2403.14472. arXiv:2403.14472 [cs]

2024 arXiv
[40]

Jailbroken: How Does LLM Safety Training Fail ?, July 2023

Wei, A., Haghtalab, N., and Steinhardt, J. Jailbroken: How Does LLM Safety Training Fail ?, July 2023. URL http://arxiv.org/abs/2307.02483. arXiv:2307.02483 [cs]

2023 arXiv
[41]

BloombergGPT : A Large Language Model for Finance , December 2023

Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., and Mann, G. BloombergGPT : A Large Language Model for Finance , December 2023. URL http://arxiv.org/abs/2303.17564. arXiv:2303.17564 [cs, q-fin]

2023 arXiv
[42]

Yong, Z.-X., Menghini, C., and Bach, S. H. Low- Resource Languages Jailbreak GPT -4, January 2024. URL http://arxiv.org/abs/2310.02446. arXiv:2310.02446 [cs]

2024 arXiv
[43]

GPTFUZZER : Red Teaming Large Language Models with Auto - Generated Jailbreak Prompts , June 2024

Yu, J., Lin, X., Yu, Z., and Xing, X. GPTFUZZER : Red Teaming Large Language Models with Auto - Generated Jailbreak Prompts , June 2024. URL http://arxiv.org/abs/2309.10253. arXiv:2309.10253 [cs]

2024 arXiv
[44]

How Johnny Can Persuade LLMs to Jailbreak Them : Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs , January 2024

Zeng, Y., Lin, H., Zhang, J., Yang, D., Jia, R., and Shi, W. How Johnny Can Persuade LLMs to Jailbreak Them : Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs , January 2024. URL http://arxiv.org/abs/2401.06373. arXiv:2401.06373 [cs]

2024 arXiv
[45]

P., Zhang, H., Gonzalez, J

Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM -as-a- Judge with MT - Bench and Chatbot Arena , December 2023. URL http://arxiv.org/abs/2306.05685. arXiv:2306.05685 [cs]

2023 arXiv
[46]

AutoDAN : Interpretable Gradient - Based Adversarial Attacks on Large Language Models , December 2023

Zhu, S., Zhang, R., An, B., Wu, G., Barrow, J., Wang, Z., Huang, F., Nenkova, A., and Sun, T. AutoDAN : Interpretable Gradient - Based Adversarial Attacks on Large Language Models , December 2023. URL http://arxiv.org/abs/2310.15140. arXiv:2310.15140 [cs]

2023 arXiv
[47]

Z., and Fredrikson, M

Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M. Universal and Transferable Adversarial Attacks on Aligned Language Models , December 2023. URL http://arxiv.org/abs/2307.15043. arXiv:2307.15043 [cs]

2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

and Kamfonas, M

Alon, G. and Kamfonas, M. Detecting Language Model Attacks with Perplexity , November 2023. URL http://arxiv.org/abs/2308.14132. arXiv:2308.14132 [cs]

arXiv 2023

[3] [3]

Jailbreaking Leading Safety - Aligned LLMs with Simple Adaptive Attacks , June 2024

Andriushchenko, M., Croce, F., and Flammarion, N. Jailbreaking Leading Safety - Aligned LLMs with Simple Adaptive Attacks , June 2024. URL http://arxiv.org/abs/2404.02151. arXiv:2404.02151 [cs, stat]

arXiv 2024

[4] [4]

Play Guessing Game with LLM : Indirect Jailbreak Attack with Implicit Clues , February 2024

Chang, Z., Li, M., Liu, Y., Wang, J., Wang, Q., and Liu, Y. Play Guessing Game with LLM : Indirect Jailbreak Attack with Implicit Clues , February 2024. URL http://arxiv.org/abs/2402.09091. arXiv:2402.09091 [cs]

arXiv 2024

[5] [5]

J., and Wong, E

Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., and Wong, E. Jailbreaking Black Box Large Language Models in Twenty Queries , July 2024. URL http://arxiv.org/abs/2310.08419. arXiv:2310.08419 [cs]

arXiv 2024

[6] [6]

A Wolf in Sheep 's Clothing : Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily , April 2024

Ding, P., Kuang, J., Ma, D., Cao, X., Xian, Y., Chen, J., and Huang, S. A Wolf in Sheep 's Clothing : Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily , April 2024. URL http://arxiv.org/abs/2311.08268. arXiv:2311.08268 [cs]

arXiv 2024

[7] [7]

Dubey, A., Kadian, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu...

arXiv 2024

[8] [8]

BERTopic : Neural topic modeling with a class-based TF - IDF procedure, March 2022

Grootendorst, M. BERTopic : Neural topic modeling with a class-based TF - IDF procedure, March 2022. URL http://arxiv.org/abs/2203.05794. arXiv:2203.05794 [cs]

arXiv 2022