Pith. sign in

REVIEW 3 major objections 8 minor 2 cited by

Large Language Model Adversarial Landscape Through the Lens of Attack Objectives

T0 review · 3 major / 8 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes the first objective-driven taxonomy of adversarial attacks on large language models, organized by four attacker objectives: privacy breach, integrity compromise, availability disruption, and misuse.

desk verdict A useful expository survey that reframes known LLM attacks under four attacker objectives, but the 'comprehensive partition' claim is weakened by overlapping categories and no systematic selection protocol. read the letter →

arxiv 2502.02960 v1 pith:F5VIRNTV submitted 2025-02-05 cs.CR

Nan Wang , Kane Walter , Yansong Gao , Alsharif Abuadbba This is my paper · ORCID
classification cs.CR
keywords largelanguagemodelsadversarialattackstaxonomyprivacybreachintegritycompromiseavailabilitydisruptionmisusedefenses
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes reorganizing the study of adversarial attacks on large language models around the attacker's goal rather than the attack technique. It claims to be the first objective-driven taxonomy, sorting the landscape into four objectives: privacy breach, integrity compromise, availability disruption, and misuse. The authors argue that this shift makes threat coverage more comprehensive and gives security practitioners a direct way to prioritize defenses, because each objective maps to a distinct class of harm. A sympathetic reader would take the paper's contribution to be a conceptual map that lets researchers ask what the adversary is trying to achieve before asking how the attack is done.

What carries the argument

The organizing device is the four-category objective taxonomy itself: privacy, integrity, availability, and misuse, defined by what the adversary wants to happen to the model or its outputs. Each attack is mapped to one or more of these categories through the four key components the attacker can touch (data, prompts, weights, gradients) and the techniques used to touch them (data poisoning, weight poisoning, model editing, prompt injection, indirect prompt injection, backdoors, adversarial examples, side channels). The taxonomy does the work of the argument: it is the lens through which all surveyed attacks and defenses are presented, and it is what distinguishes this paper from prior surveys.

What would settle it

Locate one published LLM attack whose stated adversary goal is none of privacy breach, integrity compromise, availability disruption, or misuse—for example, an attack whose aim is to evade accountability or to waste a specific user's compute without denying service—and the claimed comprehensive coverage fails. The paper itself shows the boundary pressure: indirect prompt injection is classified under integrity, availability, and misuse in different sections.

Watch

Extended reading notes

Core claim

The paper's central claim is that the adversarial landscape of LLMs is best organized by four attacker objectives—privacy breach, integrity compromise, availability disruption, and misuse—and that this objective-driven taxonomy is the first to do so. Under privacy breach it groups membership inference, model inversion, gradient leakage, model extraction, prompt extraction, and personal information identification; under integrity compromise it groups data poisoning, weight poisoning, model editing, contrastive-learning backdoors, adversarial examples, and indirect prompt injection; under availability disruption it groups denial-of-service techniques such as sponge examples, indirect prompt injection, and backdoored retrieval; under misuse it groups jailbreaking, fraud, and malware. The paper argues that existing taxonomies (risk-based, lifecycle-based, modality-based, technique-based, system-based) capture how attacks work or where they strike, but miss the strategic intent, and that intent is the better axis for anticipating and mitigating threats.

Load-bearing premise

The taxonomy's completeness rests on the assumption that every meaningful LLM attack objective falls into one of exactly four buckets—privacy, integrity, availability, or misuse—and that the handful of attacks surveyed stands in for each bucket, with no fifth objective and no overlap that changes the defense priority.

Editorial extensions

If this is right

  • Defenders can prioritize mitigations by the harm an attacker wants to cause rather than by the technique used, so a privacy breach objective points to differential privacy and audit methods while a misuse objective points to alignment and guardrails.
  • The taxonomy predicts that new attack techniques will continue to fit the four objectives even as models change, making the map stable across model generations.
  • Security teams can map incidents to objectives and correlate trends, improving threat intelligence and situational awareness.
  • The framework positions defenses against each objective, including red teaming, unlearning, prompt optimization, randomized smoothing, differential privacy, and alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the objective lens is right, then defense research should be reorganized by objective: measuring success by whether the adversary's goal is prevented, not by whether a specific attack fails.
  • The overlap the paper itself exhibits—indirect prompt injection appears under integrity, availability, and misuse—suggests the four categories are not disjoint; a useful development would be to treat objectives as a vector of harms rather than a single label per attack.
  • A testable extension is to build a corpus of published LLM attacks annotated by stated objective and measure inter-annotator agreement; strong agreement would support the taxonomy's claim, while weak agreement would expose the overlap.
  • The taxonomy could be extended to defense evaluation: benchmark suites that sample attacks uniformly from each objective would give a more complete robustness score than today's jailbreak-centric benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper proposes an objective-driven taxonomy of adversarial attacks on large language models, organizing the landscape under four attacker objectives: privacy breach, integrity compromise, availability disruption, and misuse. It surveys representative attacks in each category (membership inference, model inversion, extraction, data poisoning, backdoors, jailbreaks, DoS, etc.), discusses defense families (red teaming, optimization-based mitigations, response reformulation, randomized smoothing, differential privacy, alignment), and suggests future research directions. The central claim, stated in the abstract and Section 1.1, is that this is the first objective-driven taxonomy and that it provides a comprehensive, actionable map that moves beyond technique-based, lifecycle-based, and system-based taxonomies. The paper is organized as a structured survey with a conceptual framework rather than a formal classification study, and no empirical or systematic validation is offered for the taxonomy's coverage.

Significance. If the organizational claim is accepted, the paper's main value is as a pedagogical and strategic scaffold: grouping attacks by attacker intent can help practitioners reason about defense priorities and spot under-studied objectives such as availability disruption. The paper brings together a large number of recent attacks and defenses with citations, and the component/technique breakdown in Table 2 and Figure 2 provides a useful quick reference. Credit is due for explicitly including availability (an often-neglected category), for the concise technical overview of LLM components, and for discussing defenses alongside attacks rather than treating them as separate literatures. However, the significance is conditional on the taxonomy's boundaries being well-defined; the paper currently does not supply a systematic method for selecting attacks or for assigning them to objectives, and internal evidence shows that the same technique (e.g., indirect prompt injection or backdoors) appears under multiple objectives.

major comments (3)
  1. [§2.2 and Sections 3–6] The paper's 'comprehensive' claim is not supported by a documented selection methodology. Section 2.2 says 'we will systematically explore representative attacks,' and Sections 4.1 and 6.1 announce 'seven representative attacks' and 'three representative attack types,' respectively, but there is no corpus, search protocol, inclusion/exclusion criteria, or coding rule to justify why these attacks and not others were chosen. As a result, the claim in Section 1.2 that the objective-based approach 'encompasses a broader range of scenarios' cannot be evaluated. I recommend adding a short methodology paragraph describing how attacks were collected and selected, or revising the central claims from 'comprehensive' to 'structured'.
  2. [§2.4, §4.2.6, §5.2.2, §6.2.2, §3.2.1, §4.2.1, §5.2.3] The relationship between the four objectives and attack techniques is ambiguous. Backdoor attacks appear under Privacy Breach (§3.2.1), Integrity Compromise (§4.2.1 and §4.2.3), and Availability Disruption (§5.2.3); indirect prompt injection appears under Integrity (§4.2.6), Availability (§5.2.2), and Misuse (§6.2.2). The paper itself acknowledges in §2.4 that backdoors 'can be used for various objectives,' which is reasonable, but this sits uneasily with the four-way partition implied by Figure 2, Table 2, and the defense-prioritization argument in §1.2. The authors should state explicitly whether the taxonomy is a partition or a multi-label mapping; if the latter, they should discuss primary versus secondary objectives for each attack so that the map can guide defense choices.
  3. [§7.1 and §4.2.6] There are citation and attribution errors that affect the reliability of the survey. In §7.1, the text says 'Similar to PAIR, Perez et al. [55] proposed an automated red teaming approach' — reference [55] is Chao et al. (PAIR), so the sentence appears to describe the same work twice with an incorrect attribution. In §4.2.6, the text says 'This aligns with prior findings by Perez et al. [40], who observed that LLMs are susceptible to reward hacking,' but reference [40] is 'Discovering language model behaviors with model-written evaluations,' which does not specifically establish the reward-hacking claim as described. Please correct the citations or rephrase the attributions.
minor comments (8)
  1. [§6.2.1, Mismatched Generalization] The sentence 'Yong et al., [13] demonstrated that translating objectionable English model inputs into low-resource languages can get the users towards their harmful goals 79% of the time.' appears twice verbatim in consecutive sentences; please remove the duplicate.
  2. [§6.2.1, Mismatched Generalization] The sentence 'Mismatched generalization arises when input samples are out-of-distribution for a model's safety training data but within the scope of its broad pretraining corpus. [11].' places the citation as if it were a standalone sentence; please integrate the citation into the preceding sentence.
  3. [§1.1] The claim of being 'the first objective-driven taxonomy' is hard to verify and should be qualified with a comparison against the 'risk-based' taxonomy in [15], which already separates security and privacy goals; if the contribution is the specific four-objective split, that should be stated directly.
  4. [§5.2] The sentence 'There are three major techniques to mount DoS attacks against LLMs:' introduces Sponge Examples, Indirect Prompt Injection, and Backdoor; 'techniques' is slightly misleading for what are better described as attack vectors, and the numbering of the following subsections (5.2.1–5.2.3) does not match the announced list in a helpful way.
  5. [§5.1–§5.2] Section 5 uses the heading 'Attacks' as an empty-level divider before 'Denial of Service (DoS)'; consider reorganizing so that the section title matches the content, or removing the intermediate heading.
  6. [Table 2] In Table 2, the 'Attack Technique' column repeats entries (e.g., 'Data Poisoning' appears twice under Integrity Compromise with the same component 'Data'), which is confusing; consider listing each technique once and using a matrix or checkmarks to show which components/objectives apply.
  7. [§7.4] The description of SmoothLLM could briefly note that the perturbations are applied to the input prompt copies, not to the model, since the current phrasing 'fragility of suffixes to character-level perturbations' does not make the mechanism fully clear.
  8. [References] The reference list is inconsistent in formatting—some entries have 'Available' URL lines, others do not, and arXiv identifiers are not uniformly presented; a consistent bibliography style would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the taxonomy is definitional but makes no quantitative derivation, and the few author-overlap citations are non-load-bearing.

full rationale

This paper is a survey/taxonomy, not a derivation. Its central contribution is a classification of LLM attacks under four attacker objectives (privacy, integrity, availability, misuse), with each attack summarized and mapped to a category. No fitted parameters, quantitative predictions, or equations are involved, so there is no fitted-input-called-prediction or self-definitional reduction. The only author-overlap citations are incidental: [73] is cited in Section 8 for the general importance of LLM security, [74] is a standard backdoor-detection reference co-authored by one of the present authors, and [91] is cited as an example of fault-injection availability attacks; none carries the taxonomy's weight. Concerns that the four categories may not be exhaustive or mutually exclusive (e.g., indirect prompt injection appears under integrity, availability, and misuse; backdoors appear under privacy, integrity, and availability) are substantive correctness/rigor issues about the taxonomy's assumptions, not circularity: the paper does not derive a conclusion from those assumptions in a way that is equivalent to its inputs. The grouping is true by construction, but that is inherent to a taxonomy and does not generate a prediction that reduces to its own input. No circular step exists.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The survey has no free parameters or invented entities. Its load-bearing assumptions are that the four objective categories are exhaustive and disjoint, that the cited papers are summarized accurately, and that the hand-picked attacks are representative. The lack of a systematic search protocol makes the first and third assumptions difficult to verify.

assumptions (3)
  • ad hoc to paper The four objectives (privacy breach, integrity compromise, availability disruption, misuse) form a complete and non-overlapping partition of adversarial goals.
    Introduced in Section 2.2 and used throughout Sections 3-6; no systematic corpus analysis or formal argument establishes exhaustiveness or mutual exclusivity, and some attacks (backdoor, indirect prompt injection) appear under multiple objectives.
  • domain assumption The cited primary papers are accurately summarized.
    The taxonomy is a secondary source; if any attack description misrepresents the original work, the assigned objective and the survey's conclusions are affected. The authors do not reproduce experiments.
  • ad hoc to paper The hand-selected attacks in each category are representative of the full literature.
    Sections 3-6 present representative attacks without stating inclusion criteria, search databases, or selection rules, so coverage cannot be independently checked.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large Language Model Adversarial Landscape Through the Lens of Attack Objectives." pith.science (2026). https://pith.science/paper/F5VIRNTV

@misc{pith2026250202960,
  author       = {Pith},
  title        = {Pith review of: Large Language Model Adversarial Landscape Through the Lens of Attack Objectives},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F5VIRNTV}},
  note         = {Machine review of arXiv:2502.02960}
}
read the original abstract

Large Language Models (LLMs) represent a transformative leap in artificial intelligence, enabling the comprehension, generation, and nuanced interaction with human language on an unparalleled scale. However, LLMs are increasingly vulnerable to a range of adversarial attacks that threaten their privacy, reliability, security, and trustworthiness. These attacks can distort outputs, inject biases, leak sensitive information, or disrupt the normal functioning of LLMs, posing significant challenges across various applications. In this paper, we provide a novel comprehensive analysis of the adversarial landscape of LLMs, framed through the lens of attack objectives. By concentrating on the core goals of adversarial actors, we offer a fresh perspective that examines threats from the angles of privacy, integrity, availability, and misuse, moving beyond conventional taxonomies that focus solely on attack techniques. This objective-driven adversarial landscape not only highlights the strategic intent behind different adversarial approaches but also sheds light on the evolving nature of these threats and the effectiveness of current defenses. Our analysis aims to guide researchers and practitioners in better understanding, anticipating, and mitigating these attacks, ultimately contributing to the development of more resilient and robust LLM systems.

Figures

Figures reproduced from arXiv: 2502.02960 by the authors.

Figure 1
Figure 1. A simplified architecture of LLM-based systems. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overview of the four categories of LLM [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A model extraction example with the privacy [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: A data poisoning example with the integrity [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: A DoS example with the availability disruption [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: A jailbreak example with the misuse objective. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When the Same Coefficients Reach Different Places: Asymmetric Realizability in Transplanting Tokenizers across Large Language Models

    cs.LG 2025-12 conditional novelty 7.0 of 10

    A 'breaker token' embedding can be inert in a donor LLM yet become a high-salience trigger after tokenizer transplant into a base LLM.

  2. From Promise to Peril: Rethinking Cybersecurity Red and Blue Teaming in the Age of LLMs

    cs.CR 2025-06 conditional novelty 3.0 of 10

    LLMs can assist both attackers and defenders in cybersecurity, but context limits, hallucinations, and weak reasoning make them unsafe to deploy without human oversight and real-world evaluation.

Reference graph

Works this paper leans on

95 extracted references · 52 canonical work pages · cited by 2 Pith papers

  1. [55]

    Jailbreaking black box large language models in twenty queries,

    P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,” 2023

    work page 2023

  2. [40]

    Discovering language model behaviors with model-written evaluations,

    E. Perez, S. Ringer, K. Luko ˇsi¯ut˙e, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, A. Jones, A. Chen, B. Mann, B. Israel, B. Seethor, C. McKinnon, C. Olah, D. Yan, D. Amodei, D. Amodei, D. Drain, D. Li, E. Tran-Johnson, G. Khundadze, J. Kernion, J. Landis, J. Kerr, J. Mueller, J. Hyun, J. Landau, K. Ndousse, L. Goldberg, L....

    arXiv 2022

  3. [1]

    Quantifying privacy risks of masked language models using membership inference attacks,

    F. Mireshghallah, K. Goyal, A. Uniyal, T. Berg-Kirkpatrick, and R. Shokri, “Quantifying privacy risks of masked language models using membership inference attacks,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, 2022, pp. 8332–8347

    2022

  4. [2]

    An empirical analysis of memorization in fine-tuned autoregressive language models,

    F. Mireshghallah, A. Uniyal, T. Wang, D. K. Evans, and T. Berg- Kirkpatrick, “An empirical analysis of memorization in fine-tuned autoregressive language models,” in Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing , 2022, pp. 1816–1826

    2022

  5. [3]

    Membership inference attacks against language models via neighbourhood comparison,

    J. Mattern, F. Mireshghallah, Z. Jin, B. Sch ¨olkopf, M. Sachan, and T. Berg-Kirkpatrick, “Membership inference attacks against language models via neighbourhood comparison,” arXiv preprint arXiv:2305.18462, 2023

    arXiv 2023

  6. [4]

    Practical membership inference attacks against fine-tuned large language models via self-prompt calibration,

    W. Fu, H. Wang, C. Gao, G. Liu, Y . Li, and T. Jiang, “Practical membership inference attacks against fine-tuned large language models via self-prompt calibration,” arXiv preprint arXiv:2311.06062, 2023

    arXiv 2023

  7. [5]

    Gradient-based adversarial attacks against text transformers,

    C. Guo, A. Sablayrolles, H. J ´egou, and D. Kiela, “Gradient-based adversarial attacks against text transformers,” 2021. [Online]. Available: https://arxiv.org/abs/2104.13733

    arXiv 2021

  8. [6]

    Black box adversarial prompting for foundation models,

    N. Maus, P. Chao, E. Wong, and J. Gardner, “Black box adversarial prompting for foundation models,” 2023. [Online]. Available: https://arxiv.org/abs/2302.04237

    arXiv 2023

Show all 95 references
  1. [7]

    Sponge examples: Energy-latency attacks on neural networks,

    I. Shumailov, Y . Zhao, D. Bates, N. Papernot, R. Mullins, and R. Anderson, “Sponge examples: Energy-latency attacks on neural networks,” in 2021 IEEE European symposium on security and privacy (EuroS&P). IEEE, 2021, pp. 212–231

    2021

  2. [8]

    The skipsponge attack: Sponge weight poisoning of deep neural networks,

    J. te Lintelo, S. Koffas, and S. Picek, “The skipsponge attack: Sponge weight poisoning of deep neural networks,” 2024. [Online]. Available: https://arxiv.org/abs/2402.06357

    2024

  3. [9]

    Badrag: Identifying vulnerabilities in retrieval augmented generation of large language models,

    J. Xue, M. Zheng, Y . Hu, F. Liu, X. Chen, and Q. Lou, “Badrag: Identifying vulnerabilities in retrieval augmented generation of large language models,” 2024. [Online]. Available: https: //arxiv.org/abs/2406.00083

    2024 arXiv

  4. [10]

    Not what you’ve signed up for: Compromising real- world llm-integrated applications with indirect prompt injection,

    K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real- world llm-integrated applications with indirect prompt injection,” in Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, ser. A...

    2023

  5. [11]

    Jailbroken: How does llm safety training fail?

    A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?” in Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 80 079–80 110

    2023

  6. [12]

    Masterkey: Automated jailbreaking of large language model chatbots,

    G. Deng, Y . Liu, Y . Li, K. Wang, Y . Zhang, Z. Li, H. Wang, T. Zhang, and Y . Liu, “Masterkey: Automated jailbreaking of large language model chatbots,” in Proceedings 2024 Network and Distributed System Security Symposium, ser. NDSS 2024. Internet Society, 2024

    2024

  7. [13]

    Low-resource lan- guages jailbreak gpt-4,

    Z.-X. Yong, C. Menghini, and S. H. Bach, “Low-resource lan- guages jailbreak gpt-4,” 2024

    2024

  8. [14]

    RRHF: rank responses to align language models with human feedback,

    H. Yuan, Z. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang, “RRHF: rank responses to align language models with human feedback,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, ...

    2023

  9. [15]

    Security and privacy challenges of large language models: A survey,

    B. C. Das, M. H. Amini, and Y . Wu, “Security and privacy challenges of large language models: A survey,” 2024. [Online]. Available: https://arxiv.org/abs/2402.00888

    2024 arXiv

  10. [16]

    Unique security and privacy threats of large language model: A comprehensive survey,

    S. Wang, T. Zhu, B. Liu, M. Ding, X. Guo, D. Ye, W. Zhou, and P. S. Yu, “Unique security and privacy threats of large language model: A comprehensive survey,” 2024. [Online]. Available: https://arxiv.org/abs/2406.07973

    2024

  11. [17]

    Survey of vulnerabilities in large language models revealed by adversarial attacks,

    E. Shayegani, M. A. A. Mamun, Y . Fu, P. Zaree, Y . Dong, and N. Abu-Ghazaleh, “Survey of vulnerabilities in large language models revealed by adversarial attacks,” 2023. [Online]. Available: https://arxiv.org/abs/2310.10844

    2023 arXiv

  12. [18]

    Breaking down the defenses: A comparative survey of attacks on large language models,

    A. Chowdhury, M. Islam, V . Kumar, H. Shezan, V . Jain, and A. Chadha, “Breaking down the defenses: A comparative survey of attacks on large language models,” 03 2024

    2024

  13. [19]

    A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,

    Y . Yao, J. Duan, K. Xu, Y . Cai, Z. Sun, and Y . Zhang, “A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,” High-Confidence Computing , vol. 4, no. 2, p. 100211, 2024. [Online]. Available: https: //www.sciencedirect.com/science/arti...

    2024

  14. [20]

    Privacy backdoors: Enhancing membership inference through poisoning pre-trained models,

    Y . Wen, L. Marchyok, S. Hong, J. Geiping, T. Goldstein, and N. Carlini, “Privacy backdoors: Enhancing membership inference through poisoning pre-trained models,” 2024. [Online]. Available: https://arxiv.org/abs/2404.01231

    2024 arXiv

  15. [21]

    Privacy side channels in machine learning systems,

    E. Debenedetti, G. Severi, N. Carlini, C. A. Choquette-Choo, M. Jagielski, M. Nasr, E. Wallace, and F. Tram `er, “Privacy side channels in machine learning systems,” 2024. [Online]. Available: https://arxiv.org/abs/2309.05610

    2024 arXiv

  16. [22]

    Extracting training data from large language models,

    N. Carlini, F. Tram `er, E. Wallace, M. Jagielski, A. Herbert- V oss, K. Lee, A. Roberts, T. Brown, D. Song, ´U. Erlingsson, A. Oprea, and C. Raffel, “Extracting training data from large language models,” in 30th USENIX Security Symposium (USENIX Security 21) . USENIX Associat...

    2021

  17. [23]

    Ethicist: Targeted training data extraction through loss smoothed soft prompting and calibrated confidence estimation,

    Z. Zhang, J. Wen, and M. Huang, “Ethicist: Targeted training data extraction through loss smoothed soft prompting and calibrated confidence estimation,” 2023. [Online]. Available: https://arxiv.org/abs/2307.04401

    2023 arXiv

  18. [24]

    Deep leakage from gradients,

    L. Zhu, Z. Liu, and S. Han, “Deep leakage from gradients,” 2019. [Online]. Available: https://arxiv.org/abs/1906.08935

    2019 arXiv

  19. [25]

    A theoretical insight into attack and defense of gradient leakage in transformer,

    C. Li, Z. Song, W. Wang, and C. Yang, “A theoretical insight into attack and defense of gradient leakage in transformer,” 2023. [Online]. Available: https://arxiv.org/abs/2311.13624

    2023 arXiv

  20. [26]

    Model leeching: An extraction attack targeting llms,

    L. Birch, W. Hackett, S. Trawicki, N. Suri, and P. Garraghan, “Model leeching: An extraction attack targeting llms,” 2023

    2023

  21. [27]

    Prompt injection attack against llm-integrated applications,

    Y . Liu, G. Deng, Y . Li, K. Wang, Z. Wang, X. Wang, T. Zhang, Y . Liu, H. Wang, Y . Zheng, and Y . Liu, “Prompt injection attack against llm-integrated applications,” 2024

    2024

  22. [28]

    Effective prompt extraction from language models,

    Y . Zhang, N. Carlini, and D. Ippolito, “Effective prompt extraction from language models,” 2024. [Online]. Available: https://arxiv.org/abs/2307.06865

    2024 arXiv

  23. [29]

    The janus interface: How fine-tuning in large language models amplifies the privacy risks,

    X. Chen, S. Tang, R. Zhu, S. Yan, L. Jin, Z. Wang, L. Su, X. Wang, and H. Tang, “The janus interface: How fine-tuning in large language models amplifies the privacy risks,” arXiv preprint arXiv:2310.15469, 2023

    2023 arXiv

  24. [30]

    Multi-step jailbreaking privacy attacks on chatgpt,

    H. Li, D. Guo, W. Fan, M. Xu, J. Huang, F. Meng, and Y . Song, “Multi-step jailbreaking privacy attacks on chatgpt,”

  25. [31]

    Badpre: Task-agnostic backdoor attacks to pre-trained NLP foun- dation models,

    K. Chen, Y . Meng, X. Sun, S. Guo, T. Zhang, J. Li, and C. Fan, “Badpre: Task-agnostic backdoor attacks to pre-trained NLP foun- dation models,” in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 , 2022

    2022

  26. [32]

    Composite backdoor attacks against large language models,

    H. Huang, Z. Zhao, M. Backes, Y . Shen, and Y . Zhang, “Composite backdoor attacks against large language models,” 2024

    2024

  27. [33]

    Backdoor attacks for in-context learning with language models,

    N. Kandpal, M. Jagielski, F. Tram `er, and N. Carlini, “Backdoor attacks for in-context learning with language models,” in The Sec- ond Workshop on New Frontiers in Adversarial Machine Learning, 2023

    2023

  28. [34]

    Badedit: Backdooring large language models by model editing,

    Y . Li, T. Li, K. Chen, J. Zhang, S. Liu, W. Wang, T. Zhang, and Y . Liu, “Badedit: Backdooring large language models by model editing,” 2024

    2024

  29. [35]

    Weight poisoning attacks on pretrained models,

    K. Kurita, P. Michel, and G. Neubig, “Weight poisoning attacks on pretrained models,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds. Association for Computational Linguistics...

    2020

  30. [36]

    Backdoor attacks on pre-trained models by layerwise weight poisoning,

    L. Li, D. Song, X. Li, J. Zeng, R. Ma, and X. Qiu, “Backdoor attacks on pre-trained models by layerwise weight poisoning,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-1...

    2021

  31. [37]

    Catastrophic interference in con- nectionist networks: The sequential learning problem,

    M. McCloskey and N. J. Cohen, “Catastrophic interference in con- nectionist networks: The sequential learning problem,” Psychology of Learning and Motivation , vol. 24, pp. 109–165, 1989

    1989

  32. [38]

    Red alarm for pre-trained models: Universal vul- nerability to neuron-level backdoor attacks,

    Z. Zhang, G. Xiao, Y . Li, T. Lv, F. Qi, Z. Liu, Y . Wang, X. Jiang, and M. Sun, “Red alarm for pre-trained models: Universal vul- nerability to neuron-level backdoor attacks,” Machine Intelligence Research, 03 2023

    2023

  33. [39]

    Uor: Universal backdoor attacks on pre-trained language models,

    W. Du, P. Li, B. Li, H. Zhao, and G. Liu, “Uor: Universal backdoor attacks on pre-trained language models,” 2023

    2023

  34. [41]

    Explaining and harnessing adversarial examples,

    I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , Y . Bengio and Y . LeCun, Eds., 2015. [Onlin...

    2015 arXiv

  35. [42]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrik- son, “Universal and transferable adversarial attacks on aligned language models,” 2023

    2023

  36. [43]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models,

    X. Liu, N. Xu, M. Chen, and C. Xiao, “Autodan: Generating stealthy jailbreak prompts on aligned large language models,” 2024

    2024

  37. [44]

    Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts,

    J. Yu, X. Lin, Z. Yu, and X. Xing, “Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts,” 2023. 13

    2023

  38. [45]

    Codechameleon: Personalized encryp- tion framework for jailbreaking large language models,

    H. Lv, X. Wang, Y . Zhang, C. Huang, S. Dou, J. Ye, T. Gui, Q. Zhang, and X. Huang, “Codechameleon: Personalized encryp- tion framework for jailbreaking large language models,”CoRR, vol. abs/2402.16717, 2024

    2024 arXiv

  39. [46]

    Multilingual jailbreak challenges in large language models,

    Y . Deng, W. Zhang, S. J. Pan, and L. Bing, “Multilingual jailbreak challenges in large language models,” in The Twelfth International Conference on Learning Representations , 2024

    2024

  40. [47]

    Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher,

    Y . Yuan, W. Jiao, W. Wang, J. tse Huang, P. He, S. Shi, and Z. Tu, “Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher,”

  41. [48]

    ”do anything now

    X. Shen, Z. Chen, M. Backes, Y . Shen, and Y . Zhang, “”do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models,” 2023

    2023

  42. [49]

    Ignore previous prompt: Attack techniques for language models,

    F. Perez and I. Ribeiro, “Ignore previous prompt: Attack techniques for language models,” 2022

    2022

  43. [50]

    Why so toxic?: Measuring and triggering toxic behavior in open-domain chatbots,

    W. M. Si, M. Backes, J. Blackburn, E. D. Cristofaro, G. Stringhini, S. Zannettou, and Y . Zhang, “Why so toxic?: Measuring and triggering toxic behavior in open-domain chatbots,” in Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, CCS 2022...

    2022

  44. [51]

    Explore, establish, exploit: Red teaming language models from scratch,

    S. Casper, J. Lin, J. Kwon, G. Culp, and D. Hadfield-Menell, “Explore, establish, exploit: Red teaming language models from scratch,” 2023

    2023

  45. [52]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned,

    D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y . Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, A. Jones, S. Bowman, A. Chen, T. Conerly, N. DasSarma, D. Drain, N. Elhage, S. El- Showk, S. Fort, Z. Hatfield-Dodds, T. Henighan, D. Hernandez, T. Hume, J. Jacobson...

    2022

  46. [53]

    Fuzzllm: A novel and universal fuzzing framework for proactively discovering jailbreak vulnerabilities in large language models,

    D. Yao, J. Zhang, I. G. Harris, and M. Carlsson, “Fuzzllm: A novel and universal fuzzing framework for proactively discovering jailbreak vulnerabilities in large language models,” inICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (IC...

    2024

  47. [54]

    Parafuzz: An interpretability-driven technique for de- tecting poisoned samples in NLP,

    L. Yan, Z. Zhang, G. Tao, K. Zhang, X. Chen, G. Shen, and X. Zhang, “Parafuzz: An interpretability-driven technique for de- tecting poisoned samples in NLP,” in Advances in Neural Informa- tion Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems...

    2023

  48. [56]

    Tree of attacks: Jailbreaking black- box llms automatically,

    A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Ander- son, Y . Singer, and A. Karbasi, “Tree of attacks: Jailbreaking black- box llms automatically,” CoRR, vol. abs/2312.02119, 2023

    2023 arXiv

  49. [57]

    Automatically auditing large language models via discrete optimization,

    E. Jones, A. Dragan, A. Raghunathan, and J. Steinhardt, “Automatically auditing large language models via discrete optimization,” in Proceedings of the 40th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, A. Krause, E. Brunskill, K...

    2023

  50. [58]

    Eraser: Jailbreaking defense in large language models via unlearning harmful knowledge,

    W. Lu, Z. Zeng, J. Wang, Z. Lu, Z. Chen, H. Zhuang, and C. Chen, “Eraser: Jailbreaking defense in large language models via unlearning harmful knowledge,” 2024

    2024

  51. [59]

    Robust prompt optimization for defending language models against jailbreaking attacks,

    A. Zhou, B. Li, and H. Wang, “Robust prompt optimization for defending language models against jailbreaking attacks,” CoRR, vol. abs/2401.17263, 2024

    2024 arXiv

  52. [60]

    Jailbreakbench: An open robustness benchmark for jailbreaking large language models,

    P. Chao, E. Debenedetti, A. Robey, M. Andriushchenko, F. Croce, V . Sehwag, E. Dobriban, N. Flammarion, G. J. Pappas, F. Tramer, H. Hassani, and E. Wong, “Jailbreakbench: An open robustness benchmark for jailbreaking large language models,” 2024. [Online]. Available: https://a...

    2024 arXiv

  53. [61]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal,

    M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, D. Forsyth, and D. Hendrycks, “Harmbench: A standardized evaluation framework for automated red teaming and robust refusal,” 2024. [Online]. Available: https://arxiv.org/abs/2402.04249

    2024 arXiv

  54. [62]

    Break the breakout: Reinventing lm defense against jailbreak attacks with self-refinement,

    H. Kim, S. Yuk, and H. Cho, “Break the breakout: Reinventing lm defense against jailbreak attacks with self-refinement,” 2024

    2024

  55. [63]

    Safe rlhf: Safe reinforcement learning from human feedback,

    J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y . Wang, and Y . Yang, “Safe rlhf: Safe reinforcement learning from human feedback,” 2023

    2023

  56. [64]

    Smoothllm: Defending large language models against jailbreaking attacks,

    A. Robey, E. Wong, H. Hassani, and G. J. Pappas, “Smoothllm: Defending large language models against jailbreaking attacks,” 2023

    2023

  57. [65]

    Defending large language models against jailbreak attacks via semantic smoothing,

    J. Ji, B. Hou, A. Robey, G. J. Pappas, H. Hassani, Y . Zhang, E. Wong, and S. Chang, “Defending large language models against jailbreak attacks via semantic smoothing,” 2024

    2024

  58. [66]

    Certified adversarial robustness via randomized smoothing,

    J. Cohen, E. Rosenfeld, and J. Z. Kolter, “Certified adversarial robustness via randomized smoothing,” in Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9- 15 June 2019, Long Beach, California, USA , ser. Proceedings of Machine Learning Resear...

    2019

  59. [67]

    Dp-forward: Fine-tuning and inference on language models with differential privacy in forward pass,

    M. Du, X. Yue, S. S. M. Chow, T. Wang, C. Huang, and H. Sun, “Dp-forward: Fine-tuning and inference on language models with differential privacy in forward pass,” in Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Se- curity, CCS 2023, Copenhagen, ...

    2023

  60. [68]

    Defending against alignment-breaking attacks via robustly aligned llm,

    B. Cao, Y . Cao, L. Lin, and J. Chen, “Defending against alignment-breaking attacks via robustly aligned llm,” ArXiv, vol. abs/2309.14348, 2023

    2023 arXiv

  61. [69]

    Defending large language models against jailbreaking attacks through goal prioritization,

    Z. Zhang, J. Yang, P. Ke, and M. Huang, “Defending large language models against jailbreaking attacks through goal prioritization,” 2023

    2023

  62. [70]

    RAIN: Your language models can align themselves without finetuning,

    Y . Li, F. Wei, J. Zhao, C. Zhang, and H. Zhang, “RAIN: Your language models can align themselves without finetuning,” in The Twelfth International Conference on Learning Representations , 2024

    2024

  63. [71]

    RAFT: reward ranked finetuning for generative foundation model alignment,

    H. Dong, W. Xiong, D. Goyal, R. Pan, S. Diao, J. Zhang, K. Shum, and T. Zhang, “RAFT: reward ranked finetuning for generative foundation model alignment,” CoRR, vol. abs/2304.06767, 2023

    2023 arXiv

  64. [72]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023...

    2023

  65. [73]

    From solitary directives to interactive encouragement! llm secure code generation by natural language prompting,

    S. Liu, B. Sabir, S. I. Jang, Y . Kansal, Y . Gao, K. Moore, A. Abuadbba, and S. Nepal, “From solitary directives to interactive encouragement! llm secure code generation by natural language prompting,” arXiv preprint arXiv:2410.14321 , 2024

    arXiv

  66. [74]

    Design and evaluation of a multi- domain trojan detection method on deep neural networks,

    Y . Gao, Y . Kim, B. G. Doan, Z. Zhang, G. Zhang, S. Nepal, D. C. Ranasinghe, and H. Kim, “Design and evaluation of a multi- domain trojan detection method on deep neural networks,” IEEE Transactions on Dependable and Secure Computing, vol. 19, no. 4, pp. 2349–2364, 2021

    2021

  67. [75]

    Robust backdoor detection for deep learning via topological evolution dynamics,

    X. Mo, Y . Zhang, L. Y . Zhang, W. Luo, N. Sun, S. Hu, S. Gao, and Y . Xiang, “Robust backdoor detection for deep learning via topological evolution dynamics,” in 2024 IEEE Symposium on Security and Privacy (SP) . IEEE Computer Society, 2024, pp. 171–171

    2024

  68. [76]

    Mm-bd: Post- training detection of backdoor attacks with arbitrary backdoor pattern types using a maximum margin statistic,

    H. Wang, Z. Xiang, D. J. Miller, and G. Kesidis, “Mm-bd: Post- training detection of backdoor attacks with arbitrary backdoor pattern types using a maximum margin statistic,” in 2024 IEEE Symposium on Security and Privacy (SP) . IEEE, 2024, pp. 1994– 2012

    2024

  69. [77]

    Locating and editing factual associations in gpt,

    K. Meng, D. Bau, A. Andonian, and Y . Belinkov, “Locating and editing factual associations in gpt,”Advances in Neural Information Processing Systems, vol. 35, pp. 17 359–17 372, 2022. 14

    2022

  70. [78]

    A comprehensive study of knowledge editing for large language models,

    N. Zhang, Y . Yao, B. Tian, P. Wang, S. Deng, M. Wang, Z. Xi, S. Mao, J. Zhang, Y . Ni et al. , “A comprehensive study of knowledge editing for large language models,” arXiv preprint arXiv:2401.01286, 2024

    2024 arXiv

  71. [79]

    Federatedscope- llm: A comprehensive package for fine-tuning large language models in federated learning,

    W. Kuang, B. Qian, Z. Li, D. Chen, D. Gao, X. Pan, Y . Xie, Y . Li, B. Ding, and J. Zhou, “Federatedscope- llm: A comprehensive package for fine-tuning large language models in federated learning,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Dat...

    2024

  72. [80]

    How to backdoor federated learning,

    E. Bagdasaryan, A. Veit, Y . Hua, D. Estrin, and V . Shmatikov, “How to backdoor federated learning,” in The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy] , ser. Proceed- ings of Machin...

    2020

  73. [81]

    Attack of the tails: Yes, you really can backdoor federated learning,

    H. Wang, K. Sreenivasan, S. Rajput, H. Vishwakarma, S. Agarwal, J. Sohn, K. Lee, and D. S. Papailiopoulos, “Attack of the tails: Yes, you really can backdoor federated learning,” in Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information P...

    2020

  74. [82]

    DBA: distributed backdoor attacks against federated learning,

    C. Xie, K. Huang, P. Chen, and B. Li, “DBA: distributed backdoor attacks against federated learning,” in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020

    2020

  75. [83]

    Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning,

    M. Nasr, R. Shokri, and A. Houmansadr, “Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning,” in 2019 IEEE Symposium on Security and Privacy (SP) , 2019, pp. 739–753

    2019

  76. [84]

    Practical secure aggregation for privacy-preserving machine learning,

    K. A. Bonawitz, V . Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, “Practical secure aggregation for privacy-preserving machine learning,” in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CC...

    2017

  77. [85]

    Machine learning with adversaries: Byzantine tolerant gradient descent,

    P. Blanchard, E. M. E. Mhamdi, R. Guerraoui, and J. Stainer, “Machine learning with adversaries: Byzantine tolerant gradient descent,” in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, ...

    2017

  78. [86]

    The hidden vulnerability of distributed learning in byzantium,

    E. M. E. Mhamdi, R. Guerraoui, and S. Rouault, “The hidden vulnerability of distributed learning in byzantium,” in Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm ¨assan, Stockholm, Sweden, July 10-15, 2018 , ser. Proceedings of Mac...

    2018

  79. [87]

    Fltrust: Byzantine- robust federated learning via trust bootstrapping,

    X. Cao, M. Fang, J. Liu, and N. Z. Gong, “Fltrust: Byzantine- robust federated learning via trust bootstrapping,” in 28th Annual Network and Distributed System Security Symposium, NDSS 2021, virtually, February 21-25, 2021 . The Internet Society, 2021

    2021

  80. [88]

    FLAME: taming backdoors in federated learning,

    T. D. Nguyen, P. Rieger, H. Chen, H. Yalame, H. M¨ollering, H. Fer- eidooni, S. Marchal, M. Miettinen, A. Mirhoseini, S. Zeitouni, F. Koushanfar, A. Sadeghi, and T. Schneider, “FLAME: taming backdoors in federated learning,” in 31st USENIX Security Sym- posium, USENIX Security...

    2022

  81. [89]

    Calibrating noise to sensitivity in private data analysis,

    C. Dwork, F. McSherry, K. Nissim, and A. D. Smith, “Calibrating noise to sensitivity in private data analysis,” in Theory of Cryp- tography, Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006, Proceedings , ser. Lecture Notes in Computer Scie...

    2006

  82. [90]

    CRFL: certifiably robust federated learning against backdoor attacks,

    C. Xie, M. Chen, P. Chen, and B. Li, “CRFL: certifiably robust federated learning against backdoor attacks,” in Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , ser. Proceedings of Machine Learning Research, M. M...

    2021

  83. [91]

    Yes, one-bit-flip matters! universal dnn model infer- ence depletion with runtime code fault injection,

    S. Li, X. Wang, M. Xue, H. Zhu, Z. Zhang, Y . Gao, W. Wu, and X. S. Shen, “Yes, one-bit-flip matters! universal dnn model infer- ence depletion with runtime code fault injection,” in Proceedings of the 33th USENIX Security Symposium , 2024. 15

    2024

  84. [2023]

    Available: https://arxiv.org/abs/2304.05197

    [Online]. Available: https://arxiv.org/abs/2304.05197

    arXiv

  85. [2024]

    Available: https://arxiv.org/abs/2308.06463

    [Online]. Available: https://arxiv.org/abs/2308.06463

    arXiv

  86. [2650]

    Available: https://www.usenix.org/conference/ usenixsecurity21/presentation/carlini-extracting

    [Online]. Available: https://www.usenix.org/conference/ usenixsecurity21/presentation/carlini-extracting

  87. [3876]

    Springer, 2006, pp. 265–284

    2006

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.