pith. machine review for the scientific record. sign in

arxiv: 2305.13860 · v2 · submitted 2023-05-23 · 💻 cs.SE · cs.AI· cs.CL

Recognition: 1 theorem link

· Lean Theorem

Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study

Yi Liu , Gelei Deng , Zhengzi Xu , Yuekang Li , Yaowen Zheng , Ying Zhang , Lida Zhao , Tianwei Zhang
show 2 more authors
Kailong Wang Yang Liu
Authors on Pith no claims yet

Pith reviewed 2026-05-16 22:33 UTC · model grok-4.3

classification 💻 cs.SE cs.AIcs.CL
keywords jailbreakingprompt engineeringChatGPTlarge language modelscontent filtersadversarial promptssafety evaluationempirical study
0
0 comments X

The pith

Jailbreak prompts classified into ten patterns can consistently evade ChatGPT's content restrictions in 40 use-case scenarios.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests how specific prompt structures can bypass the built-in safeguards in ChatGPT. It first groups existing jailbreak prompts into ten patterns that fall into three broader categories. Then it runs 3,120 questions across eight prohibited topics on both ChatGPT 3.5 and 4.0 to measure how often the prompts succeed. The central result is that these prompts reliably overcome the model's restrictions in forty different practical situations. This matters because it shows that current safety layers can be defeated by carefully worded inputs and that prompt design itself is a key variable in whether those layers hold.

Core claim

The study develops a classification model that divides jailbreak prompts into ten distinct patterns and three categories. When these prompts are applied to a dataset of 3,120 questions spanning eight prohibited scenarios, they succeed in circumventing the restrictions of ChatGPT versions 3.5 and 4.0. The evaluation further shows that the same prompts maintain consistent effectiveness across forty separate use-case scenarios.

What carries the argument

A classification model that organizes jailbreak prompts into ten patterns and three categories, then measures their success rate against ChatGPT's built-in content filters.

If this is right

  • Prompt structure is a decisive factor in whether safety constraints are respected.
  • Current defense mechanisms leave exploitable gaps that can be mapped systematically.
  • Evaluation of LLM safety must include testing across varied prompt patterns rather than single examples.
  • Prevention strategies will need to address the full range of the ten identified patterns.
  • The effectiveness observed on both 3.5 and 4.0 indicates the vulnerability persists across model updates.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same classification approach could be used to test safety in other large language models.
  • Training data for future models might need to include adversarial examples from all ten patterns.
  • Detection systems could be built that scan incoming prompts for the identified structural signatures.
  • Repeated testing on new model releases would reveal whether the patterns remain effective over time.

Load-bearing premise

The 3,120 questions and forty use-case scenarios are representative of real attempts to bypass restrictions and the ten-pattern classification covers the main ways prompts can succeed.

What would settle it

A new collection of questions drawn from a wider set of topics or a later ChatGPT version that the identified prompt patterns fail to bypass would show the claimed consistency does not hold.

read the original abstract

Large Language Models (LLMs), like ChatGPT, have demonstrated vast potential but also introduce challenges related to content constraints and potential misuse. Our study investigates three key research questions: (1) the number of different prompt types that can jailbreak LLMs, (2) the effectiveness of jailbreak prompts in circumventing LLM constraints, and (3) the resilience of ChatGPT against these jailbreak prompts. Initially, we develop a classification model to analyze the distribution of existing prompts, identifying ten distinct patterns and three categories of jailbreak prompts. Subsequently, we assess the jailbreak capability of prompts with ChatGPT versions 3.5 and 4.0, utilizing a dataset of 3,120 jailbreak questions across eight prohibited scenarios. Finally, we evaluate the resistance of ChatGPT against jailbreak prompts, finding that the prompts can consistently evade the restrictions in 40 use-case scenarios. The study underscores the importance of prompt structures in jailbreaking LLMs and discusses the challenges of robust jailbreak prompt generation and prevention.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 0 minor

Summary. The manuscript investigates jailbreaking of ChatGPT using prompt engineering. It develops a classification model to identify ten distinct patterns and three categories of jailbreak prompts from existing examples. The study then evaluates the effectiveness of these prompts on ChatGPT versions 3.5 and 4.0 using a dataset of 3,120 jailbreak questions across eight prohibited scenarios. Finally, it assesses the resilience of ChatGPT, concluding that the prompts can consistently evade restrictions in 40 use-case scenarios.

Significance. This empirical study provides insights into the vulnerabilities of large language models to prompt-based jailbreaks, which is timely and relevant for improving AI safety. The use of a substantial dataset of 3,120 questions is a positive aspect. However, the significance is limited by the lack of detailed methodology and success metrics, which if addressed could make this a valuable contribution to the field of LLM security.

major comments (3)
  1. Abstract: the claim that 'the prompts can consistently evade the restrictions in 40 use-case scenarios' provides no definition of success (e.g., exact prohibited output vs. refusal), no per-scenario success rates, no trial counts, and no explanation of how the 40 scenarios relate to the eight prohibited ones, rendering the central effectiveness claim unverifiable.
  2. Abstract/Methods: no details are given on the training, validation, or features of the classification model used to derive the ten patterns and three categories, which is load-bearing for the distribution analysis and subsequent effectiveness claims.
  3. Results: the evaluation on 3,120 questions reports no quantitative outcomes, statistical controls, or per-pattern/per-scenario breakdowns, so it is impossible to assess whether the 'consistent' evasion finding holds or is reproducible.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We agree that the abstract and results sections require greater precision in definitions, methodology details, and quantitative reporting to make the claims verifiable and reproducible. We will revise the manuscript accordingly to address each major comment.

read point-by-point responses

  1. Referee: Abstract: the claim that 'the prompts can consistently evade the restrictions in 40 use-case scenarios' provides no definition of success (e.g., exact prohibited output vs. refusal), no per-scenario success rates, no trial counts, and no explanation of how the 40 scenarios relate to the eight prohibited ones, rendering the central effectiveness claim unverifiable.

    Authors: We agree that the abstract lacks sufficient detail on these elements. Success is defined as the jailbreak prompt eliciting a direct response to the prohibited query rather than a refusal or deflection by ChatGPT. The 40 use-case scenarios are specific instantiations derived from the eight prohibited categories through variations in phrasing and context. Each combination was evaluated over multiple independent trials. In the revised manuscript we will update the abstract to include this definition, report aggregate and per-scenario success rates, specify the number of trials, and clarify the mapping between the eight categories and 40 scenarios. revision: yes

  2. Referee: Abstract/Methods: no details are given on the training, validation, or features of the classification model used to derive the ten patterns and three categories, which is load-bearing for the distribution analysis and subsequent effectiveness claims.

    Authors: The ten patterns and three categories were derived through manual expert analysis of collected jailbreak examples rather than a trained machine-learning classifier; no training or validation splits were used. The features include linguistic structures such as role-playing instructions, hypothetical framing, and direct constraint overrides. We will expand the methods section to fully document the categorization criteria, provide representative examples for each pattern, and describe the process used to arrive at the taxonomy. revision: yes

  3. Referee: Results: the evaluation on 3,120 questions reports no quantitative outcomes, statistical controls, or per-pattern/per-scenario breakdowns, so it is impossible to assess whether the 'consistent' evasion finding holds or is reproducible.

    Authors: We acknowledge that the current results presentation is primarily qualitative and lacks the requested quantitative detail. The 3,120 questions were generated by crossing the jailbreak patterns with the prohibited scenarios, and we observed high rates of successful evasion. In the revision we will add tables and text reporting success rates broken down by pattern and scenario, the number of trials per item, and any statistical summaries to support reproducibility. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical results rest on external model behavior

full rationale

The paper performs an empirical study: it classifies existing prompts into ten patterns via data analysis, constructs a dataset of 3,120 questions, and measures ChatGPT's responses to those prompts. No equations, fitted parameters, or predictions are defined in terms of the target success rates. The reported evasion rates are direct observations of an external black-box model, not reductions of the input data by construction. Self-citations, if present, are not load-bearing for the central empirical claims. The study is therefore self-contained against external benchmarks and receives the default non-circularity finding.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that the selected prohibited scenarios and prompt collection accurately represent the space of jailbreak attempts and that success is measured consistently across model versions.

axioms (1)
  • domain assumption Certain topics are prohibited by LLM safety policies and can be reliably identified as such.
    The study defines eight prohibited scenarios without providing an independent justification or reference for why these particular topics are treated as the relevant test cases.

pith-pipeline@v0.9.0 · 5503 in / 1222 out tokens · 61219 ms · 2026-05-16T22:33:57.809333+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 21 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Catching the Infection Before It Spreads: Foresight-Guided Defense in Multi-Agent Systems

    cs.AI 2026-05 unverdicted novelty 7.0

    A foresight-based local purification method using multi-persona simulations and recursive diagnosis reduces infectious jailbreak spread in multi-agent systems from over 95% to below 5.47% while matching benign perform...

  2. RACC: Representation-Aware Coverage Criteria for LLM Safety Testing

    cs.SE 2026-02 unverdicted novelty 7.0

    RACC defines six representation-aware coverage criteria that score jailbreak test suites by measuring activation of safety concepts extracted from LLM hidden states on a calibration set.

  3. Quantifying LLM Safety Degradation Under Repeated Attacks Using Survival Analysis

    cs.CR 2026-05 unverdicted novelty 6.0

    Survival analysis applied to repeated jailbreak attacks on three LLMs shows one model degrades rapidly while the others maintain moderate vulnerability on HarmBench prompts.

  4. Catching the Infection Before It Spreads: Foresight-Guided Defense in Multi-Agent Systems

    cs.AI 2026-05 unverdicted novelty 6.0

    A foresight-based local purification method simulates future agent interactions, detects infections via response diversity across personas, and applies targeted rollback or recursive diagnosis to cut maximum infection...

  5. Catching the Infection Before It Spreads: Foresight-Guided Defense in Multi-Agent Systems

    cs.AI 2026-05 unverdicted novelty 6.0

    FLP uses multi-persona foresight simulation to detect infections via response diversity and applies local purification to reduce maximum cumulative infection rates in multi-agent systems from over 95% to below 5.47%.

  6. Representation-Guided Parameter-Efficient LLM Unlearning

    cs.CL 2026-04 unverdicted novelty 6.0

    REGLU guides LoRA-based unlearning via representation subspaces and orthogonal regularization to outperform prior methods on forget-retain trade-off in LLM benchmarks.

  7. TEMPLATEFUZZ: Fine-Grained Chat Template Fuzzing for Jailbreaking and Red Teaming LLMs

    cs.CR 2026-04 unverdicted novelty 6.0

    TEMPLATEFUZZ mutates chat templates with element-level rules and heuristic search to reach 98.2% average jailbreak success rate on twelve open-source LLMs while degrading accuracy by only 1.1%.

  8. Exclusive Unlearning

    cs.CL 2026-04 unverdicted novelty 6.0

    Exclusive Unlearning makes LLMs safe by forgetting all but retained domain knowledge, protecting against jailbreaks while preserving useful responses in areas like medicine and math.

  9. CoopGuard: Stateful Cooperative Agents Safeguarding LLMs Against Evolving Multi-Round Attacks

    cs.CR 2026-04 unverdicted novelty 6.0

    CoopGuard deploys cooperative agents to track conversation history and counter evolving multi-round attacks on LLMs, achieving a 78.9% reduction in attack success rate on a new 5,200-sample benchmark.

  10. SoK: Agentic Skills -- Beyond Tool Use in LLM Agents

    cs.CR 2026-02 unverdicted novelty 6.0

    The paper systematizes agentic skills beyond tool use, providing design pattern and representation-scope taxonomies plus security analysis of malicious skill infiltration in agent marketplaces.

  11. Stop Tracking Me! Proactive Defense Against Attribute Inference Attack in LLMs

    cs.CR 2026-02 conditional novelty 6.0

    TRACE-RPS drops LLM attribute inference accuracy from around 50% to below 5% via fine-grained anonymization plus a two-stage rejection optimization.

  12. Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projection

    cs.LG 2026-02 conditional novelty 6.0

    OGPSA projects safety gradients orthogonal to a low-rank subspace from general capability gradients, improving safety-utility trade-offs in SFT and DPO pipelines on Qwen2.5-7B and Llama3.1-8B.

  13. From Rookie to Expert: Manipulating LLMs for Automated Vulnerability Exploitation in Enterprise Software

    cs.SE 2025-12 unverdicted novelty 6.0

    RSA prompting enables LLMs to automatically create functional exploits for CVEs in Odoo ERP, succeeding on all tested cases in 3-5 rounds and removing the need for manual effort.

  14. A StrongREJECT for Empty Jailbreaks

    cs.LG 2024-02 conditional novelty 6.0

    StrongREJECT provides a standardized benchmark and evaluator for jailbreak attacks that aligns better with human judgments than prior methods and reveals that successful jailbreaks often reduce model capabilities.

  15. Low-Resource Languages Jailbreak GPT-4

    cs.CL 2023-10 conditional novelty 6.0

    Translating unsafe inputs to low-resource languages jailbreaks GPT-4 at rates on par with or exceeding state-of-the-art attacks.

  16. GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts

    cs.AI 2023-09 unverdicted novelty 6.0

    GPTFuzz is a black-box fuzzing framework that mutates seed jailbreak templates to automatically generate effective attacks, achieving over 90% success rates on models including ChatGPT and Llama-2.

  17. "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models

    cs.CR 2023-08 unverdicted novelty 6.0

    Real-world jailbreak prompts collected from the wild achieve up to 0.95 attack success rates against major LLMs including GPT-4, with some persisting for over 240 days.

  18. Metaphor Is Not All Attention Needs

    cs.CL 2026-05 unverdicted novelty 5.0

    Poetic jailbreaks succeed because they induce distinct attention patterns in LLMs that are independent of harmful-content detection, not because models fail to recognize literary formatting.

  19. A Systematic Study of Training-Free Methods for Trustworthy Large Language Models

    cs.CL 2026-04 unverdicted novelty 5.0

    Training-free methods for LLM trustworthiness show inconsistent results across dimensions, with clear trade-offs in utility, robustness, and overhead depending on where they intervene during inference.

  20. Jailbreak Attacks and Defenses Against Large Language Models: A Survey

    cs.CR 2024-07 accept novelty 4.0

    A survey that creates taxonomies for jailbreak attacks and defenses on LLMs, subdivides them into sub-classes, and compares evaluation approaches.

  21. Beyond Context: Large Language Models' Failure to Grasp Users' Intent

    cs.AI 2025-12 unverdicted novelty 3.0

    LLMs fail to detect hidden harmful intent, allowing systematic bypass of safety mechanisms through framing techniques, with reasoning modes often worsening the issue.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages · cited by 19 Pith papers

  1. [1]

    Prompting large language model for machine translation: A case study,

    B. Zhang, B. Haddow, and A. Birch, “Prompting large language model for machine translation: A case study,” CoRR, vol. abs/2301.07069,

    work page arXiv

  2. [2]

    Prompting large language model for machine translation: A case study,

    [Online]. Available: https://doi .org/10.48550/arXiv.2301.07069

    work page doi:10.48550/arxiv.2301.07069

  3. [3]

    A complete survey on generative AI (AIGC): is chatgpt from GPT-4 to GPT-5 all you need?

    C. Zhang, C. Zhang, S. Zheng, Y . Qiao, C. Li, M. Zhang, S. K. Dam, C. M. Thwal, Y . L. Tun, L. L. Huy, D. U. Kim, S. Bae, L. Lee, Y . Yang, H. T. Shen, I. S. Kweon, and C. S. Hong, “A complete survey on generative AI (AIGC): is chatgpt from GPT-4 to GPT-5 all you need?” CoRR, vol. abs/2303.11717, 2023. [Online]. Available: https://doi.org/10.48550/arXiv....

    work page doi:10.48550/arxiv.2303.11717 2023

  4. [4]

    Recent advances in deep learning based dialogue systems: a systematic survey,

    J. Ni, T. Young, V . Pandelea, F. Xue, and E. Cambria, “Recent advances in deep learning based dialogue systems: a systematic survey,” Artif. Intell. Rev., vol. 56, no. 4, pp. 3055–3155, 2023. [Online]. Available: https://doi.org/10.1007/s10462-022-10248-8

    work page doi:10.1007/s10462-022-10248-8 2023

  5. [5]

    New chat,

    “New chat,” https://chat .openai.com/, (Accessed on 02/02/2023)

    work page 2023

  6. [6]

    Models - openai api,

    “Models - openai api,” https://platform .openai.com/docs/models/, (Ac- cessed on 02/02/2023)

    work page 2023

  7. [7]

    “Openai,” https://openai .com/, (Accessed on 02/02/2023)

    work page 2023

  8. [8]

    Multi-step jailbreaking privacy attacks on chatgpt,

    H. Li, D. Guo, W. Fan, M. Xu, and Y . Song, “Multi-step jailbreaking privacy attacks on chatgpt,” 2023

    work page 2023

  9. [9]

    A prompt pattern catalog to enhance prompt engineering with chatgpt,

    J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern catalog to enhance prompt engineering with chatgpt,” 2023

    work page 2023

  10. [10]

    Meet dan â C

    “Meet dan â C” the â C˜jailbreakâC™ version of chatgpt and how to use it â C” ai unchained and unfiltered | by michael king | medium,” https://medium.com/@neonforge/meet-dan-the-jailbreak-version-of- chatgpt-and-how-to-use-it-ai-unchained-and-unfiltered-f91bfa679024, (Accessed on 02/02/2023)

    work page 2023

  11. [11]

    Moderation - openai api,

    “Moderation - openai api,” https://platform .openai.com/docs/guides/ moderation, (Accessed on 02/02/2023)

    work page 2023

  12. [12]

    Llm jailbreak study,

    “Llm jailbreak study,” https://sites .google.com/view/llm-jailbreak-study, (Accessed on 05/06/2023)

    work page 2023

  13. [13]

    Alex albert,

    “Alex albert,” https://alexalbert .me/, (Accessed on 05/06/2023)

    work page 2023

  14. [14]

    Grounded theory in software engineering research: a critical review and guidelines,

    K. Stol, P. Ralph, and B. Fitzgerald, “Grounded theory in software engineering research: a critical review and guidelines,” in Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2016 , L. K. Dillon, W. Visser, and L. A. Williams, Eds. ACM, 2016, pp. 120–131. [Online]. Available: https://doi.org...

    work page doi:10.1145/2884781.2884833 2016

  15. [15]

    Api reference - openai api,

    “Api reference - openai api,” https://platform .openai.com/docs/ api-reference/completions/create#completions/create-temperature, (Accessed on 05/04/2023)

    work page 2023

  16. [16]

    NACDL - Computer Fraud and Abuse Act (CFAA),

    “NACDL - Computer Fraud and Abuse Act (CFAA),” https://www.govinfo.gov/app/details/USCODE-2010-title18/USCODE- 2010-title18-partI-chap47-sec1030, accessed: 2023-5-5

    work page 2010

  17. [17]

    Children’s online privacy protection rule (

    “Children’s online privacy protection rule ("coppa") | federal trade commission,” https://www .ftc.gov/legal-library/browse/rules/childrens- online-privacy-protection-rule-coppa, (Accessed on 05/04/2023)

    work page 2023

  18. [18]

    TITLE 47â C

    “TITLE 47â C”TELECOMMUNICATIONS,” https://www.govinfo.gov/ content/pkg/USCODE-2021-title47/pdf/USCODE-2021-title47-chap5- subchapII-partI-sec224 .pdf, accessed: 2023-5-5

    work page 2021

  19. [19]

    18 U.S.C. 2516 - Authorization for interception of wire, oral, or electronic communications

    “18 U.S.C. 2516 - Authorization for interception of wire, oral, or electronic communications.” https://www .govinfo.gov/app/details/ USCODE-2021-title18/USCODE-2021-title18-partI-chap119-sec2516, accessed: 2023-5-6

    work page 2021

  20. [20]

    18 U.S.C. 2251 - Sexual exploitation of children

    “18 U.S.C. 2251 - Sexual exploitation of children.” https://www.govinfo.gov/app/details/USCODE-2021-title18/USCODE- 2021-title18-partI-chap119-sec2516, accessed: 2023-5-6

    work page 2021

  21. [21]

    52 U.S.C. 30116 - Limitations on contributions and expenditures,

    “52 U.S.C. 30116 - Limitations on contributions and expenditures,” https://www.govinfo.gov/app/details/USCODE-2014-title52/USCODE- 2014-title52-subtitleIII-chap301-subchapI-sec30116, accessed: 2023-5- 6

    work page 2014

  22. [22]

    INVESTMENT ADVISERS ACT OF 1940 [AMENDED 2022],

    “INVESTMENT ADVISERS ACT OF 1940 [AMENDED 2022],” https: //www.govinfo.gov/content/pkg/COMPS-1878/pdf/COMPS-1878 .pdf, accessed: 2023-5-6

    work page 1940

  23. [23]

    Prompting ai art: An investigation into the creative skill of prompt engineering,

    J. Oppenlaender, R. Linder, and J. Silvennoinen, “Prompting ai art: An investigation into the creative skill of prompt engineering,” 2023

    work page 2023

  24. [24]

    Prompt programming for large language models: Beyond the few-shot paradigm,

    L. Reynolds and K. McDonell, “Prompt programming for large language models: Beyond the few-shot paradigm,” 2021

    work page 2021

  25. [25]

    Fundamental limitations of alignment in large language models,

    Y . Wolf, N. Wies, Y . Levine, and A. Shashua, “Fundamental limitations of alignment in large language models,” 2023

    work page 2023

  26. [26]

    MTTM: metamorphic testing for textual content moderation software,

    W. Wang, J. Huang, W. Wu, J. Zhang, Y . Huang, S. Li, P. He, and M. R. Lyu, “MTTM: metamorphic testing for textual content moderation software,” CoRR, vol. abs/2302.05706, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2302.05706 12

    work page doi:10.48550/arxiv.2302.05706 2023