Pith. sign in

REVIEW 4 major objections 6 minor 68 references

SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Models

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims DeepSeek's jailbreak resistance is selective—it withstands optimization-driven attacks but yields much more than GPT-4 to prompt-based and human-crafted adversarial prompts.

desk verdict A useful DeepSeek-vs-GPT jailbreak benchmark whose central conclusion is an artifact of comparing raw ASR; under the paper's own DR-normalized metric, the susceptibility result reverses. read the letter →

arxiv 2506.18543 v2 pith:TZ4HSZUD submitted 2025-06-23 cs.CR cs.AI

Xiaodong Wu , Xiangman Li , Qi Li , Lingshuang Liu , Jianbing Ni This is my paper
classification cs.CRcs.AI
keywords jailbreakattackssafetyalignmentDeepSeekGPT-4attacksuccessrateopen-weightLLMrobustnessadversarialpromptingmodelscaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the DeepSeek family of open-weight language models has a real but uneven safety alignment: it partially resists optimization-driven jailbreaks, yet it is substantially easier to compromise through prompt rewrites, persuasion, and human-crafted adversarial inputs. The claim matters because open-weight models are being adopted rapidly, and this is one of the first systematic comparisons of DeepSeek against the GPT-3.5 and GPT-4 families under a single standardized protocol. On the paper's reading, GPT-4-family alignment generalizes across behavioral categories, whereas DeepSeek's safety constraints apply unevenly and become less reliable as model size grows. If the paper is right, safe deployment of open-source models requires targeted safety tuning rather than assuming base alignment carries over.

What carries the argument

The central object is the attack success rate (ASR) computed by a fixed evaluation protocol: a curated set of 510 harmful behaviors, seven attack methods spanning black-box and white-box categories, and a fine-tuned classifier that labels each model completion as a successful jailbreak or a refusal. Behavior-level breakdowns by functional category (standard, copyright, contextual, multimodal) and by six high-risk semantic domains let the authors separate where safety holds from where it fails. The direct-request baseline serves as the reference point, and the paper argues that reporting the margin over this baseline captures scaling-related risk better than the raw ASR.

What would settle it

Re-score a stratified sample of completions—balanced by model family, attack method, and behavior category—with independent human raters using the paper's stated success criteria, then recompute the attack success rates; if the DeepSeek-over-GPT gap on prompt-based attacks shrinks below statistical significance, the central trade-off claim fails.

Watch

Extended reading notes

Core claim

The discovery is a systematic asymmetry in jailbreak resilience. Under the same evaluation protocol, DeepSeek models show partial resistance to the tree-search attack TAP-T, but their attack success rates under prompt-based attacks are much higher than GPT-4's: DeepSeek reaches roughly 77% on standard behaviors and 74% on contextual behaviors under a zero-shot prompting attack, about double GPT-4's rates, while GPT-4 Turbo stays lower and more uniform across categories. Within the DeepSeek family, gradient-based and chain-of-thought attacks become more successful as parameters grow, with GCG rising from 33.44% at 1.5B to 55.31% at 32B. The paper explains the gap by alignment strategy: the GPT family benefits from reinforcement-learning-from-human-feedback pipelines and red-teaming, while DeepSeek's distilled variants rely more on supervised fine-tuning. It concludes that scaling capability without scaling alignment enlarges the attack surface.

Load-bearing premise

The load-bearing premise is that the fine-tuned classifier correctly identifies jailbreak success for both model families; if that judge systematically favors or penalizes one family's style of completions, the reported attack-success-rate gaps are not trustworthy.

Editorial extensions

If this is right

  • Larger DeepSeek variants (7B to 32B) are more, not less, vulnerable to gradient-based and chain-of-thought attacks, so model scaling without alignment scaling widens the attack surface.
  • Defenses should prioritize gradient-based adversarial training and refusal mechanisms that monitor chain-of-thought reasoning, and evaluations should report ASR relative to the direct-request baseline.
  • The safety gap is category-specific: DeepSeek is stronger on chemical/biological and copyright behaviors, while GPT models are stronger on misinformation, cybercrime, harassment, and illegal content.
  • Open-weight releases need alignment pipelines with the same breadth of feedback and red-teaming as the GPT family, or deployment carries a measurable safety penalty.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The failure pattern suggests a testable hypothesis—fine-tuning a DeepSeek variant on adversarial paraphrases from prompt-rewriting and persuasion attacks should close most of the zero-shot gap, because the vulnerability appears to be surface-form generalization rather than missing safety knowledge.
  • Inference: Because the whole comparison depends on one classifier judge, a human-labeled replication on a few hundred outputs would settle whether the family-level gaps are real; the direction of the comparison could shift if the judge favors one style of completions.
  • Inference: The copyright result complicates the simple 'open models are less safe' narrative, since DeepSeek shows stricter verbatim-reproduction behavior than GPT-4 in this benchmark.
  • Inference: The scaling trend implies that every distilled variant of a reasoning model should be safety-tested individually, not only the base model, since capability distillation does not appear to preserve alignment strength.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents a SoK-style empirical evaluation of jailbreak resilience in DeepSeek distilled models (1.5B, 7B, 8B, 14B, 32B) and GPT-3.5/GPT-4 models. Using the HarmBench benchmark, it evaluates seven attack methods across 510 harmful behaviors, reporting raw Attack Success Rates (ASR), category-level breakdowns by Standard/Contextual/Copyright behaviors, and qualitative case studies. The main claimed findings are that DeepSeek models show partial resilience to optimization-driven attacks such as TAP-T but greater susceptibility to prompt-based and manually engineered adversarial inputs, while GPT-4 Turbo exhibits more consistent safety alignment. The paper also analyzes scaling effects within the DeepSeek family and discusses implications for alignment research.

Significance. The paper provides a timely and fairly large empirical comparison of a popular open-source model family against closed-source GPT models, using the standardized HarmBench protocol and its classifier-based evaluation. The inclusion of multiple DeepSeek sizes, seven attack methods, category-level breakdowns, and concrete case studies is valuable and makes the measurements reproducible in principle. If the raw ASR comparisons were reliable, the paper would offer a useful addition to the jailbreak-evaluation literature by focusing on a model family that is underrepresented in existing studies. However, the central comparative claim is undercut by the paper's own preferred metric: the text explicitly recommends assessing robustness by margin over Direct Request (DR) rather than absolute ASR, yet the headline cross-family conclusions are drawn from raw ASR. Under DR-normalized margins, the ordering for most prompt-based attacks reverses, which means the abstract's core claim is not supported by the paper's own recommended analysis. The treatment of multimodal behaviors and the lack of variance estimates further weaken confidence in the quantitative conclusions.

major comments (4)
  1. [3.4.1, Table 2] The central cross-family comparison uses raw ASR even though Section 3.4.1 states that 'robustness is better assessed by margin over DR rather than absolute success rates.' Applying the paper's own normalized margin, (ASR - DR)/(1 - DR), to Table 2 reverses the headline. For DeepSeek-32B versus GPT-4 Turbo: PAIR gives -5.43% versus +13.95%, ZeroShot gives -9.38% versus +3.32%, GCG-T gives -1.25% versus +13.95%, TAP-T gives +21.35% versus +53.16%, and only Human shows DeepSeek higher (+1.98% versus -7.86%). Thus the abstract's claim that DeepSeek results in 'greater susceptibility to prompt-based and manually engineered adversarial inputs' is an artifact of comparing raw ASR across models with very different Direct Request baselines. The paper should either revise the central claim or report DR-normalized margins for the cross-family comparison, especially since Section 4.2 itself acknowledges that raw ASR values can mask incremental risks and that normalized indicators better capture safety behavior under attack.
  2. [3.1, Tables 2-3] The paper states that HarmBench contains 510 behaviors, of which 110 are multimodal, but the treatment of these multimodal behaviors is never stated. The evaluated DeepSeek distilled models are text-only language models, so either the multimodal behaviors were excluded from the ASR computations or some visual input was synthesized or omitted. If they were excluded, the effective dataset is 400 behaviors and the aggregate numbers in Tables 2 and 3 are not directly comparable to HarmBench's standard 510-behavior protocol; if they were included, the evaluation protocol for text-only models is unspecified. This ambiguity affects the denominators of all reported ASRs and therefore the comparability of the central claims; the paper should clarify exactly which behaviors were used and report the per-category denominators.
  3. [3.4.2, Appendix B] Section 3.4.2 states that in the chemical and biological domain 'all DeepSeek variants successfully prevent jailbreaks,' but Appendix B reports that under GCG-T, DeepSeek-8B provided a detailed multi-step chemical synthesis plan for the banned pesticide parathion, including suggested reagents and conditions. This is a direct contradiction. If the claim is scoped only to specific attacks (e.g., the red-teaming attacks in Figure 3(a)) or to a particular aggregation, that scope should be stated explicitly; as written, the claim is falsified by the paper's own case study.
  4. [3.4, Tables 2-3] All ASRs are reported as point estimates with no confidence intervals, no repeated runs, and no discussion of sampling variance. Several cross-model differences that support the paper's narrative are small (e.g., DeepSeek 7B vs 8B on DR: 40.94 vs 41.56; DeepSeek 14B vs 32B on TAP-T: 50.00 vs 52.81). Without repeated evaluations or at least a statement that decoding was greedy and deterministic, it is unclear whether these differences are within noise. The paper should report variance or otherwise justify the stability of the rankings that drive its conclusions.
minor comments (6)
  1. [Table 2] The Human column for GPT-3.5 Turbo 1106 shows a value of '3.1', which appears to be a typo; all other Human values are in the 12-42 range, and 3.1 would imply near-perfect refusal, contradicting the surrounding analysis.
  2. [2.1.1, Table 3] TAP is classified as a gradient-based white-box attack in Table 3 and in the white-box evaluation description, but the cited TAP method (ref. [34]) is a black-box tree-of-attacks method; this misclassification should be corrected.
  3. [Acknowledgments/Conflicts of Interest] The manuscript retains journal template boilerplate in these sections (e.g., 'In this section, you can acknowledge...' and 'Any interest or relationship...'); these placeholders should be replaced with actual statements or removed.
  4. [Figures 2-3] The figures show Standard, Contextual, and Copyright categories but omit the Multimodal category, despite Section 3.1 listing it as one of the four HarmBench categories; the captions should state explicitly whether multimodal behaviors were excluded and why.
  5. [Data Availability Statement] For a SoK-style empirical paper, stating that 'additional processed results are available from the corresponding author upon reasonable request' is weak; providing the evaluation scripts and raw per-behavior results would substantially strengthen reproducibility.
  6. [3.4.1, Table 2] ZeroShot is listed under LLM-based attacks in Table 2, but the attack is not described or cited in Section 2.2.3; the paper should define the ZeroShot prompt construction or provide a reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's claims are empirical benchmark measurements against the externally maintained HarmBench protocol, with no fitted parameter or self-citation used to produce its conclusions.

full rationale

The paper is an empirical evaluation, not a derivation. Its central comparisons are Attack Success Rates obtained by running seven attack methods on GPT and DeepSeek models and labeling completions with HarmBench's pretrained classifier, an external benchmark artifact from other authors. There is no equation in the paper whose output is defined by its own inputs, no parameter fitted to a subset of data and then renamed as a prediction, and no load-bearing self-citation: the authors do not justify any core premise by citing their own prior work. The only step that resembles a circularity concern is the paper's own observation that robustness is better assessed by margin over the Direct Request baseline rather than absolute ASR, followed by cross-family headline claims based on raw ASR. That inconsistency is a methodological validity or normalization issue, not circularity, because the raw ASR comparison is neither definitionally equivalent to the DR-margin result nor produced by fitting a parameter to that result. Similarly, dependence on HarmBench's Llama-2-13B-Chat judge may bias the comparison, but the judge is external, pretrained, and not constructed to match this paper's conclusions, so this is an external-validity caveat rather than a self-referential reduction. No circular step can be quoted and exhibited, so the honest finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

This is an empirical benchmark study, so the ledger records the experimental assumptions the comparison rests on rather than fitted constants. The only hand-set quantities are unreported attack hyperparameters; no derivation or fitted model underlies the claims.

free parameters (1)
  • Attack hyperparameters (query budgets, iterations, temperature) = not reported
    ASR values for PAIR, TAP/TAP-T, GCG, and PAP are sensitive to these settings; the paper does not report them, so the point estimates are conditional on unreported choices and may not transfer.
assumptions (4)
  • domain assumption HarmBench's fine-tuned Llama-2-13B-Chat classifier correctly labels jailbreak success for both GPT and DeepSeek completions.
    Invoked in Section 3.3; if the judge is biased toward one family, all cross-model ASR comparisons are affected.
  • domain assumption The OpenAI API returns raw completions without additional moderation or post-processing.
    Stated in Section 3.2 but not independently verifiable; any hidden safety layer would reduce reported GPT ASR.
  • domain assumption The five DeepSeek distilled variants are representative of the DeepSeek model family.
    Section 3.2 does not give exact checkpoint identifiers, so reproducibility and generalization of the family-level conclusion rest on this.
  • domain assumption Attack implementations in the HarmBench harness faithfully reproduce the original attacks.
    Section 3.3 relies on HarmBench protocols; deviations would change ASR levels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Models." pith.science (2026). https://pith.science/paper/TZ4HSZUD

@misc{pith2026250618543,
  author       = {Pith},
  title        = {Pith review of: SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TZ4HSZUD}},
  note         = {Machine review of arXiv:2506.18543}
}
read the original abstract

The rapid proliferation of Large Language Models (LLMs) has heightened concerns regarding their exposure to jailbreak attacks, which craft adversarial inputs designed to elicit unsafe content. Although proprietary models such as GPT-4 have been extensively evaluated, the robustness of emerging open-source systems like DeepSeek remains insufficiently examined, despite their growing use in LLM applications. In this paper, we conduct the first comprehensive jailbreak analysis of the DeepSeek model family, comparing it with GPT-3.5 and GPT-4 through the HarmBench benchmark. We investigate seven representative attack methods across 510 harmful behaviors, organized along both functional and semantic dimensions. Findings indicate that DeepSeek provides partial resilience against optimization-driven attacks such as TAP-T, but also results in greater susceptibility to prompt-based and manually engineered adversarial inputs. In contrast, GPT-4 Turbo demonstrates more robust and consistent safety alignment across a wide range of behaviors, likely due to stronger safety optimization and reinforcement learning from human feedback. In addition, fine-grained behavioral analysis and case studies reveal that DeepSeek often fails to consistently apply safety constraints to adversarial prompts, leading to uneven refusal behaviors. Overall, our results highlight an inherent trade-off between model efficiency and alignment generalization, underscoring the importance of targeted safety tuning and robust alignment strategies to ensure secure deployment of open-source LLMs.

Figures

Figures reproduced from arXiv: 2506.18543 by the authors.

Figure 1
Figure 1. Taxonomy of jailbreak attacks. instantiated through adversarially engineered prompts, informed by model knowledge or training data, or refined via iterative probing. A detailed taxonomy and overview of jailbreak methods is shown in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. ASR of DeepSeek and GPT under all behaviors dataset. +15.31 at 32B, and H-CoT from 1.12 to 11.74. TAP remains at or below DR across all sizes, while AutoDan stays close to DR, dipping slightly at 1.5B and 14B. Thus, GCG and H-CoT show the largest gains with scale, while TAP improves only modestly. These trends suggest defenses should prioritize GCG-style adversarial training and refusal mechanisms sensitive to chain… view at source ↗
Figure 3
Figure 3. ASRs of GPT and DeepSeek for different categories. We further examined vulnerabilities across six high-risk domains exploited in adversarial prompting. As shown in [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 25 canonical work pages

  1. [1]

    Improving language understanding by generative pre-training2018

    Radford, A.; Narasimhan, K.; Salimans, T.; Sutskever, I.; others. Improving language understanding by generative pre-training2018

    work page

  2. [2]

    GPT-4 Technical Report, 2023, [arXiv:cs.CL/2303.08774]

    OpenAI. GPT-4 Technical Report, 2023, [arXiv:cs.CL/2303.08774]

    arXiv 2023

  3. [3]

    Deepseek llm: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.029542024

    Bi, X.; Chen, D.; Chen, G.; Chen, S.; Dai, D.; Deng, C.; Ding, H.; Dong, K.; Du, Q.; Fu, Z.; others. Deepseek llm: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.029542024

    arXiv

  4. [4]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.129482025

    Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; Zhu, Q.; Ma, S.; Wang, P.; Bi, X.; others. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.129482025

    arXiv

  5. [5]

    Pal: Proxy-guided black-box attack on large language models.arXiv preprint arXiv:2402.096742024

    Sitawarin, C.; Mu, N.; Wagner, D.; Araujo, A. Pal: Proxy-guided black-box attack on large language models.arXiv preprint arXiv:2402.096742024

    arXiv

  6. [6]

    Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.150432023

    Zou, A.; Wang, Z.; Carlini, N.; Nasr, M.; Kolter, J.Z.; Fredrikson, M. Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.150432023

    arXiv

  7. [7]

    Jailbroken: How Does LLM Safety Training Fail?arXiv preprint arXiv:2311.066072023

    Wei, J.; Chilton, L.; Borgeaud, S.; others. Jailbroken: How Does LLM Safety Training Fail?arXiv preprint arXiv:2311.066072023

    arXiv

  8. [8]

    A survey of backdoor attacks and defenses on large language models: Implications for security measures.Authorea Preprints2024

    Zhao, S.; Jia, M.; Guo, Z.; Gan, L.; Xu, X.; Wu, X.; Fu, J.; Feng, Y .; Pan, F.; Tuan, L.A. A survey of backdoor attacks and defenses on large language models: Implications for security measures.Authorea Preprints2024

    work page

Show all 68 references
  1. [9]

    Training language models to follow instructions with human feedback.NeurIPS2022,35, 27730–27744

    Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; others. Training language models to follow instructions with human feedback.NeurIPS2022,35, 27730–27744

  2. [10]

    Defending Large Language Models Against Jailbreak Attacks Through Chain of Thought Prompting

    Cao, Y .; Gu, N.; Shen, X.; Yang, D.; Zhang, X. Defending Large Language Models Against Jailbreak Attacks Through Chain of Thought Prompting. NaNA, 2024, pp. 125–130

    2024

  3. [11]

    Many-shot jailbreaking.NeurIPS2024,37, 129696–129742

    Anil, C.; Durmus, E.; Panickssery, N.; Sharma, M.; Benton, J.; Kundu, S.; Batson, J.; Tong, M.; Mu, J.; Ford, D.; others. Many-shot jailbreaking.NeurIPS2024,37, 129696–129742

  4. [12]

    Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoning.CoRR abs/2402.084162024

    Deng, G.; Liu, Y .; Wang, K.; Li, Y .; Zhang, T.; Liu, Y . Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoning.CoRR abs/2402.084162024

    arXiv

  5. [13]

    Multi-step jailbreaking privacy attacks on chatgpt.arXiv preprint arXiv:2304.051972023

    Li, H.; Guo, D.; Fan, W.; Xu, M.; Huang, J.; Meng, F.; Song, Y . Multi-step jailbreaking privacy attacks on chatgpt.arXiv preprint arXiv:2304.051972023

    arXiv

  6. [14]

    Adversarial demonstration attacks on large language models.arXiv preprint arXiv:2305.149502023

    Wang, J.; Liu, Z.; Park, K.H.; Jiang, Z.; Zheng, Z.; Wu, Z.; Chen, M.; Xiao, C. Adversarial demonstration attacks on large language models.arXiv preprint arXiv:2305.149502023. https://doi.org/10.xxxx/xxx 14 of 19 Wu et al.Pragmatic Cybersecur .2025,V olume(Issue), Page Number

    arXiv

  7. [15]

    Jailbreak and guard aligned language models with only few in-context demonstrations.arXiv preprint arXiv:2310.063872023

    Wei, Z.; Wang, Y .; Li, A.; Mo, Y .; Wang, Y . Jailbreak and guard aligned language models with only few in-context demonstrations.arXiv preprint arXiv:2310.063872023

    arXiv

  8. [16]

    Improved few-shot jailbreaking can circumvent aligned language models and their defenses.NeurIPS2024,37, 32856–32887

    Zheng, X.; Pang, T.; Du, C.; Liu, Q.; Jiang, J.; Lin, M. Improved few-shot jailbreaking can circumvent aligned language models and their defenses.NeurIPS2024,37, 32856–32887

  9. [17]

    Play guessing game with llm: Indirect jailbreak attack with implicit clues.arXiv preprint arXiv:2402.090912024

    Chang, Z.; Li, M.; Liu, Y .; Wang, J.; Wang, Q.; Liu, Y . Play guessing game with llm: Indirect jailbreak attack with implicit clues.arXiv preprint arXiv:2402.090912024

    arXiv

  10. [18]

    Artprompt: Ascii art-based jailbreak attacks against aligned llms

    Jiang, F.; Xu, Z.; Niu, L.; Xiang, Z.; Ramasubramanian, B.; Li, B.; Poovendran, R. Artprompt: Ascii art-based jailbreak attacks against aligned llms. ACL (V olume 1: Long Papers), 2024, pp. 15157–15173

    2024

  11. [19]

    Semantic mirror jailbreak: Genetic algorithm based jailbreak prompts against open-source llms.arXiv preprint arXiv:2402.148722024

    Li, X.; Liang, S.; Zhang, J.; Fang, H.; Liu, A.; Chang, E.C. Semantic mirror jailbreak: Genetic algorithm based jailbreak prompts against open-source llms.arXiv preprint arXiv:2402.148722024

    arXiv

  12. [20]

    Understanding and enhancing the transferability of jailbreaking attacks.arXiv preprint arXiv:2502.030522025

    Lin, R.; Han, B.; Li, F.; Liu, T. Understanding and enhancing the transferability of jailbreaking attacks.arXiv preprint arXiv:2502.030522025

    arXiv

  13. [21]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models

    Liu, X.; Xu, N.; Chen, M.; Xiao, C. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.044512023

    arXiv

  14. [22]

    Flipattack: Jailbreak llms via flipping

    Liu, Y .; He, X.; Xiong, M.; Fu, J.; Deng, S.; Hooi, B. Flipattack: Jailbreak llms via flipping. ICML, 2025

    2025

  15. [23]

    All in how you ask for it: Simple black-box method for jailbreak attacks.Applied Sciences2024,14, 3558

    Takemoto, K. All in how you ask for it: Simple black-box method for jailbreak attacks.Applied Sciences2024,14, 3558

  16. [24]

    Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection

    Wei, Z.; Liu, Y .; Erichson, N.B. Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection. ICML, 2025

    2025

  17. [25]

    The Dark Side of Trust: Authority Citation-Driven Jailbreak Attacks on Large Language Models.arXiv preprint arXiv:2411.114072024

    Yang, X.; Tang, X.; Han, J.; Hu, S. The Dark Side of Trust: Authority Citation-Driven Jailbreak Attacks on Large Language Models.arXiv preprint arXiv:2411.114072024

    arXiv

  18. [26]

    Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts

    Yu, J.; Lin, X.; Yu, Z.; Xing, X. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.102532023

    arXiv

  19. [27]

    Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher2024

    Yuan, Y .; Jiao, W.; Wang, W.; Huang, J.t.; He, P.; Shi, S.; Tu, Z. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher2024

  20. [28]

    Explore, establish, exploit: Red teaming language models from scratch.arXiv preprint arXiv:2306.094422023

    Casper, S.; Lin, J.; Kwon, J.; Culp, G.; Hadfield-Menell, D. Explore, establish, exploit: Red teaming language models from scratch.arXiv preprint arXiv:2306.094422023

    arXiv

  21. [29]

    Jailbreaking black box large language models in twenty queries

    Chao, P.; Robey, A.; Dobriban, E.; Hassani, H.; Pappas, G.J.; Wong, E. Jailbreaking black box large language models in twenty queries. SaTML, 2025, pp. 23–42

    2025

  22. [30]

    Masterkey: Automated jailbreak across multiple large language model chatbots.arXiv preprint arXiv:2307.087152023

    Deng, G.; Liu, Y .; Li, Y .; Wang, K.; Zhang, Y .; Li, Z.; Wang, H.; Zhang, T.; Liu, Y . Masterkey: Automated jailbreak across multiple large language model chatbots.arXiv preprint arXiv:2307.087152023

    arXiv

  23. [31]

    MART: Improving LLM Safety with Multi-round Automatic Red-Teaming

    Ge, S.; Zhou, C.; Hou, R.; Khabsa, M.; Wang, Y .C.; Wang, Q.; Han, J.; Mao, Y . MART: Improving LLM Safety with Multi-round Automatic Red-Teaming. NAACL (V olume 1: Long Papers), 2024, pp. 1927–1937

    2024

  24. [32]

    Guard: Role-playing to generate natural-language jailbreakings to test guideline adherence of large language models.arXiv preprint arXiv:2402.032992024

    Jin, H.; Chen, R.; Zhang, P.; Zhou, A.; Zhang, Y .; Wang, H. Guard: Role-playing to generate natural-language jailbreakings to test guideline adherence of large language models.arXiv preprint arXiv:2402.032992024

  25. [33]

    Goal-oriented prompt attack and safety evaluation for llms.arXiv preprint arXiv:2309.118302023

    Liu, C.; Zhao, F.; Qing, L.; Kang, Y .; Sun, C.; Kuang, K.; Wu, F. Goal-oriented prompt attack and safety evaluation for llms.arXiv preprint arXiv:2309.118302023

    arXiv

  26. [34]

    Tree of attacks: Jailbreaking black-box llms automatically.NeurIPS2024,37, 61065–61105

    Mehrotra, A.; Zampetakis, M.; Kassianik, P.; Nelson, B.; Anderson, H.; Singer, Y .; Karbasi, A. Tree of attacks: Jailbreaking black-box llms automatically.NeurIPS2024,37, 61065–61105

  27. [35]

    Scalable and transferable black-box jailbreaks for language models via persona modulation.arXiv preprint arXiv:2311.033482023

    Shah, R.; Pour, S.; Tagade, A.; Casper, S.; Rando, J.; others. Scalable and transferable black-box jailbreaks for language models via persona modulation.arXiv preprint arXiv:2311.033482023

    arXiv

  28. [36]

    Evil geniuses: Delving into the safety of llm-based agents.arXiv preprint arXiv:2311.118552023

    Tian, Y .; Yang, X.; Zhang, J.; Dong, Y .; Su, H. Evil geniuses: Delving into the safety of llm-based agents.arXiv preprint arXiv:2311.118552023

    arXiv

  29. [37]

    How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms

    Zeng, Y .; Lin, H.; Zhang, J.; Yang, D.; Jia, R.; Shi, W. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. ACL (V olume 1: Long Papers), 2024, pp. 14322–14350

    2024

  30. [38]

    Attacking large language models with projected gradient descent.arXiv preprint arXiv:2402.091542024

    Geisler, S.; Wollschläger, T.; Abdalla, M.H.I.; Gasteiger, J.; Günnemann, S. Attacking large language models with projected gradient descent.arXiv preprint arXiv:2402.091542024

    arXiv

  31. [39]

    Query-based adversarial prompt generation.NeurIPS2024, 37, 128260–128279

    Hayase, J.; Borevkovi´c, E.; Carlini, N.; Tramèr, F.; Nasr, M. Query-based adversarial prompt generation.NeurIPS2024, 37, 128260–128279

  32. [40]

    Improved techniques for optimization-based jailbreaking on large language models.arXiv preprint arXiv:2405.210182024

    Jia, X.; Pang, T.; Du, C.; Huang, Y .; Gu, J.; Liu, Y .; Cao, X.; Lin, M. Improved techniques for optimization-based jailbreaking on large language models.arXiv preprint arXiv:2405.210182024

    arXiv

  33. [41]

    Iterative self-tuning llms for enhanced jailbreaking capabilities.arXiv preprint arXiv:2410.184692024

    Sun, C.E.; Liu, X.; Yang, W.; Weng, T.W.; Cheng, H.; San, A.; Galley, M.; Gao, J. Iterative self-tuning llms for enhanced jailbreaking capabilities.arXiv preprint arXiv:2410.184692024

  34. [42]

    From noise to clarity: Unraveling the adversarial suffix of large language model attacks via translation of text embeddings.CoRR2024

    Wang, H.; Li, H.; Huang, M.; Sha, L. From noise to clarity: Unraveling the adversarial suffix of large language model attacks via translation of text embeddings.CoRR2024. https://doi.org/10.xxxx/xxx 15 of 19 Wu et al.Pragmatic Cybersecur .2025,V olume(Issue), Page Number

  35. [43]

    Guiding not forcing: Enhancing the transferability of jailbreaking attacks on llms via removing superfluous constraints.arXiv preprint arXiv:2503.018652025

    Yang, J.; Zhang, Z.; Cui, S.; Wang, H.; Huang, M. Guiding not forcing: Enhancing the transferability of jailbreaking attacks on llms via removing superfluous constraints.arXiv preprint arXiv:2503.018652025

    arXiv

  36. [44]

    AutoDAN: interpretable gradient-based adversarial attacks on large language models.arXiv preprint arXiv:2310.151402023

    Zhu, S.; Zhang, R.; An, B.; Wu, G.; Barrow, J.; Wang, Z.; Huang, F.; Nenkova, A.; Sun, T. AutoDAN: interpretable gradient-based adversarial attacks on large language models.arXiv preprint arXiv:2310.151402023

    arXiv

  37. [45]

    Analyzing the inherent response tendency of llms: Real-world instructions- driven jailbreak.arXiv preprint arXiv:2312.041272023

    Du, Y .; Zhao, S.; Ma, M.; Chen, Y .; Qin, B. Analyzing the inherent response tendency of llms: Real-world instructions- driven jailbreak.arXiv preprint arXiv:2312.041272023

    arXiv

  38. [46]

    COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability

    Guo, X.; Yu, F.; Zhang, H.; Qin, L.; Hu, B. COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability. PMLR2024,235, 16974–17002

  39. [47]

    Droj: A prompt-driven attack against large language models.arXiv preprint arXiv:2411.091252024

    Hu, L.; Wang, B. Droj: A prompt-driven attack against large language models.arXiv preprint arXiv:2411.091252024

    arXiv

  40. [48]

    Catastrophic jailbreak of open-source llms via exploiting generation

    Huang, Y .; Gupta, S.; Xia, M.; Li, K.; Chen, D. Catastrophic jailbreak of open-source llms via exploiting generation. ICLR, 2024

    2024

  41. [49]

    Make them spill the beans! coercive knowledge extraction from (production) llms.arXiv preprint arXiv:2312.047822023

    Zhang, Z.; Shen, G.; Tao, G.; Cheng, S.; Zhang, X. Make them spill the beans! coercive knowledge extraction from (production) llms.arXiv preprint arXiv:2312.047822023

    arXiv

  42. [50]

    Weak-to-strong jailbreaking on large language models.arXiv preprint arXiv:2401.172562024

    Zhao, X.; Yang, X.; Pang, T.; Du, C.; Li, L.; Wang, Y .X.; Wang, W.Y . Weak-to-strong jailbreaking on large language models.arXiv preprint arXiv:2401.172562024

    arXiv

  43. [51]

    Don’t say no: Jailbreaking llm by suppressing refusal.arXiv preprint arXiv:2404.163692024

    Zhou, Y .; Lou, J.; Huang, Z.; Qin, Z.; Yang, Y .; Wang, W. Don’t say no: Jailbreaking llm by suppressing refusal.arXiv preprint arXiv:2404.163692024

    arXiv

  44. [52]

    Fine-tuning aligned language models compromises safety, even when users do not intend to!arXiv preprint arXiv:2310.036932023

    Qi, X.; Zeng, Y .; Xie, T.; Chen, P.Y .; Jia, R.; Mittal, P.; Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to!arXiv preprint arXiv:2310.036932023

    arXiv

  45. [53]

    Shadow alignment: The ease of subverting safely-aligned language models.arXiv preprint arXiv:2310.029492023

    Yang, X.; Wang, X.; Zhang, Q.; Petzold, L.; Wang, W.Y .; Zhao, X.; Lin, D. Shadow alignment: The ease of subverting safely-aligned language models.arXiv preprint arXiv:2310.029492023

    arXiv

  46. [54]

    Removing rlhf protections in gpt-4 via fine-tuning

    Zhan, Q.; Fang, R.; Bindu, R.; Gupta, A.; Hashimoto, T.; Kang, D. Removing rlhf protections in gpt-4 via fine-tuning. arXiv preprint arXiv:2311.055532023

    arXiv

  47. [55]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.arXiv preprint arXiv:2402.042492024

    Mazeika, M.; Phan, L.; Yin, X.; Zou, A.; Wang, Z.; Mu, N.; Sakhaee, E.; Li, N.; Basart, S.; Li, B.; others. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.arXiv preprint arXiv:2402.042492024

    arXiv

  48. [56]

    Easyjailbreak: A unified framework for jailbreaking large language models.arXiv preprint arXiv:2403.121712024

    Zhou, W.; Wang, X.; Xiong, L.; Xia, H.; Gu, Y .; Chai, M.; Zhu, F.; Huang, C.; Dou, S.; Xi, Z.; others. Easyjailbreak: A unified framework for jailbreaking large language models.arXiv preprint arXiv:2403.121712024

    arXiv

  49. [57]

    Playing language game with llms leads to jailbreaking.arXiv preprint arXiv:2411.127622024

    Peng, Y .; Long, Z.; Dong, F.; Li, C.; Wu, S.; Chen, K. Playing language game with llms leads to jailbreaking.arXiv preprint arXiv:2411.127622024

    arXiv

  50. [58]

    do anything now

    Shen, X.; Chen, Z.; Backes, M.; Shen, Y .; Zhang, Y . " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. CCS, 2024, pp. 1671–1685

    2024

  51. [59]

    Chain-of-Lure: A Synthetic Narrative-Driven Approach to Compromise Large Language Models.arXiv preprint arXiv:2505.175192025

    Chang, W.; Zhu, T.; Zhao, Y .; Song, S.; Xiong, P.; Zhou, W.; Li, Y . Chain-of-Lure: A Synthetic Narrative-Driven Approach to Compromise Large Language Models.arXiv preprint arXiv:2505.175192025

  52. [60]

    Reasoning-augmented conversation for multi-turn jailbreak attacks on large language models.arXiv preprint arXiv:2502.110542025

    Ying, Z.; Zhang, D.; Jing, Z.; Xiao, Y .; Zou, Q.; Liu, A.; Liang, S.; Zhang, X.; Liu, X.; Tao, D. Reasoning-augmented conversation for multi-turn jailbreak attacks on large language models.arXiv preprint arXiv:2502.110542025

    arXiv

  53. [61]

    Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debate.arXiv preprint arXiv:2504.164892025

    Qi, S.; Zou, Y .; Li, P.; Lin, Z.; Cheng, X.; Yu, D. Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debate.arXiv preprint arXiv:2504.164892025

    arXiv

  54. [62]

    Kuo, M.; Zhang, J.; Ding, A.; Wang, Q.; DiValentin, L.; Bao, Y .; Wei, W.; Li, H.; Chen, Y . H-cot: Hijacking the chain-of-thought safety reasoning mechanism to jailbreak large reasoning models, including openai o1/o3, deepseek-r1, and gemini 2.0 flash thinking.arXiv preprint ...

    arXiv

  55. [63]

    Advancing Jailbreak Strategies: A Hybrid Approach to Exploiting LLM Vulnerabilities and Bypassing Modern Defenses.arXiv preprint arXiv:2506.219722025

    Ahmed, M.; Abdelmouty, M.; Kim, M.; Kandula, G.; Park, A.; Davis, J.C. Advancing Jailbreak Strategies: A Hybrid Approach to Exploiting LLM Vulnerabilities and Bypassing Modern Defenses.arXiv preprint arXiv:2506.219722025

    arXiv

  56. [64]

    Gradient cuff: Detecting jailbreak attacks on large language models by exploring refusal loss landscapes.NeurIPS2024,37, 126265–126296

    Hu, X.; Chen, P.Y .; Ho, T.Y . Gradient cuff: Detecting jailbreak attacks on large language models by exploring refusal loss landscapes.NeurIPS2024,37, 126265–126296

  57. [65]

    JBShield: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulation

    Zhang, S.; Zhai, Y .; Guo, K.; Hu, H.; Guo, S.; Fang, Z.; Zhao, L.; Shen, C.; Wang, C.; Wang, Q. JBShield: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulation. USENIX, 2025, pp. 8215–8234

    2025

  58. [66]

    The hidden risks of large reasoning models: A safety assessment of r1.arXiv preprint arXiv:2502.126592025

    Zhou, K.; Liu, C.; Zhao, X.; Jangam, S.; Srinivasa, J.; Liu, G.; Song, D.; Wang, X.E. The hidden risks of large reasoning models: A safety assessment of r1.arXiv preprint arXiv:2502.126592025

  59. [67]

    Scaling Trends in Language Model Robustness

    Howe, N.H.R.; Mckenzie, I.R.; Hollinsworth, O.J.; Zaj ˛ ac, M.; Tseng, T.; Tucker, A.D.; Bacon, P.L.; Gleave, A. Scaling Trends in Language Model Robustness. ICML. PMLR, 2025, V ol. 267, pp. 24080–24138

    2025

  60. [68]

    deepseek-distill-8b

    Sun, Z.; Miceli-Barone, A.V . Scaling behavior of machine translation with large language models under prompt injection attacks.arXiv preprint arXiv:2403.098322024. https://doi.org/10.xxxx/xxx 16 of 19 Wu et al.Pragmatic Cybersecur .2025,V olume(Issue), Page Number A. Dataset ...

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.