Pith. sign in

REVIEW 3 major objections 7 minor 3 cited by

Defending Against Prompt Injection With a Few DefensiveTokens

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding five optimized token embeddings before an LLM's input blocks prompt-injection attacks almost as well as retraining the model.

desk verdict A genuinely useful test-time prompt-injection defense with broad results; the abstract oversells it as comparable to training-time defenses when the paper's own adaptive-GCG numbers show large gaps on half the models. read the letter →

arxiv 2507.07974 v2 pith:Z2PQDCC3 submitted 2025-07-10 cs.CR

Sizhe Chen , Yizhu Wang , Nicholas Carlini , Chawin Sitawarin , David Wagner This is my paper
classification cs.CR
keywords promptinjectiondefensetest-timesofttuningtokenembeddingoptimizationLLMsecurityattacksuccessrateutility-securitytrade-off
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Prompt injection attacks let untrusted data override the instructions an LLM system is supposed to follow. This paper claims that a few new special tokens, whose continuous embeddings are optimized for security, can be prepended to the input at test time to make the system nearly as robust as retraining the model on defensive data. On the largest benchmark, the method cuts optimization-free attack success to 0.24% on average across four 7B/8B models, matching training-time defenses, while losing less utility than any baseline. Because the defense is implemented purely by prepending tokens, developers can switch it on only in security-sensitive cases and leave the model untouched otherwise. The cost is that deployment requires an inference stack that accepts raw embedding vectors rather than only text tokens.

What carries the argument

The central object is a small set of randomly initialized continuous embedding vectors, one per new special token in the model vocabulary, prepended to the input as $[t; x]$. Their parameters are optimized by gradient descent with the StruQ loss, $-\log p_{\theta,t}(y \mid [t; x])$, on a dataset where half the samples are benign and half carry injected instructions, with labels generated by the undefended model itself. The optimized embeddings have average 1-norm around 4332 versus 34 for ordinary vocabulary embeddings, so the defense lives in a continuous region of embedding space that no discrete prompt can reach; this is what the paper credits for the large margin over prompting defenses.

What would settle it

Run the same experiment on an LLM API that strictly accepts discrete tokens (no raw embedding input): if no textual prompt engineered from the optimized embeddings achieves ASR near 0.24% on TaskTracker, the claimed flexibility and no-infrastructure-change property fails for that deployment.

Watch

Extended reading notes

Core claim

DefensiveToken is a test-time prompt injection defense that reaches training-time-level security. The paper shows that optimizing only the embeddings of five newly added special tokens, using the StruQ defensive loss on a self-labeled instruction dataset, makes four open-weight LLMs ignore injected instructions in most cases: 0.24% average ASR on the 31K-sample TaskTracker benchmark versus 0.20–0.51% for training-time defenses, and over 11% for three prompting-based test-time baselines. Utility, measured by AlpacaEval2 WinRate, drops less than any baseline, and the defense remains switchable at test time: omit the tokens and the system runs exactly as the undefended model.

Load-bearing premise

The whole approach presupposes that the production LLM service lets developers prepend arbitrary continuous vectors to the input, not just ordinary text tokens; if APIs only accept text, the defense cannot be applied as claimed.

Editorial extensions

If this is right

  • A model provider can release DefensiveTokens alongside a model, letting each developer decide whether to spend security or utility.
  • Queries with and without DefensiveTokens can be batched together, so the defense requires no infrastructure changes to offer.
  • The defense generalizes to an agentic API-calling benchmark (InjecAgent), reducing ASR fivefold, and to optimization-based GCG attacks, where it halves ASR.
  • The utility cost of the defense is confined to those who choose to apply it, since skipping the tokens restores the undefended model.
  • Increasing the number of tokens (up to 20) can improve security further at a small utility cost, though five suffice across the four tested models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The large embedding-norm gap suggests the mechanism is better described as a learned input filter that steers attention than as a prompt; a testable extension is to check whether the tokens transfer across models that share a tokenizer, revealing whether they encode a model-specific or generic 'ignore data' signal.
  • If production APIs ever require discrete text, a discrete-token distillation of the optimized embeddings could be attempted, though the norm gap suggests such a distillation may lose most of the defense's efficacy.
  • The one-epoch optimization on Cleaned Alpaca resembles prompt tuning, so the same mechanism might extend to other safety goals such as ignoring system-level following attacks, but the paper explicitly scopes out jailbreaks and data extraction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces DefensiveToken, a test-time defense against prompt injection. The method optimizes the embeddings of a small number of newly added special tokens (five in the main experiments) with the StruQ loss and prepends these continuous token embeddings to the LLM input at inference time. The authors evaluate on Llama3-8B-Instruct, Llama3.1-8B-Instruct, Falcon3-7B-Instruct, and Qwen2.5-7B-Instruct, across five benchmarks (AlpacaFarm, SEP, TaskTracker, CyberSecEval2, InjecAgent), comparing to test-time prompting baselines (Reminder, Sandwich, TextGrad) and training-time baselines (StruQ and SecAlign with full/LoRA fine-tuning). The central claim is that DefensiveToken is as effective as training-time defenses on optimization-free attacks, remains the best test-time defense under GCG adaptive attacks, and incurs the smallest utility loss among all baselines while being switchable at test time.

Significance. DefensiveToken is a plausible and well-ablated contribution. Its strengths are the breadth of evaluation (four models, five benchmarks), the inclusion of an adaptive GCG attack, the careful ablations (number of tokens, initialization, loss, position, learning rate), the multi-run variability study, and public code. If the comparability claim held across the full attack spectrum, the method would give developers a flexible alternative to fine-tuning. However, the evidence for the headline claim is mixed: the method is indeed close to training-time defenses on optimization-free benchmarks, but under the adaptive GCG attack it lags by a wide margin on two of four models. In addition, the paper does not flag that the method requires the serving stack to accept raw continuous embeddings, which limits the no-infrastructure-change claim. The contribution is still significant if these caveats are stated, but the current framing overreaches.

major comments (3)
  1. [§4.3, Table 3, Abstract] The claim that DefensiveToken has prompt injection robustness comparable to training-time alternatives is not supported under the adaptive GCG attack. On AlpacaFarm, the average GCG-ASR of DefensiveToken across the four models is 48.8%, versus 38.4% for StruQ-LoRA, 37.4% for SecAlign-LoRA, and 28.8% for StruQ-Full. On Llama3-8B and Llama3.1-8B the gaps are roughly an order of magnitude (37.5% and 24.6% for DefensiveToken versus 1.9-4.8% and 1.0-10.1% for the training-time baselines). Section 4.3 describes this as slightly weaker, which understates the result. The abstract and conclusion should either restrict the comparability claim to optimization-free attacks or provide a substantive argument for why the large GCG gap does not undermine the stated contribution.
  2. [§1, §3.3] The deployment interface is an unstated assumption. DefensiveTokens are continuous embedding vectors (their average 1-norm is 4332, versus 34 for vocabulary tokens, Table 2), and Eq. (1) and Section 3.3 require prepending these raw vectors to the token sequence. The Introduction claims that the provider needs no infrastructure changes for deployment and Section 3.2 claims that defensive tokens require no additional infrastructure changes from the provider side. Standard LLM serving APIs accept discrete text tokens, not arbitrary float vectors; supporting this defense requires the serving stack to accept input embeddings. The paper should state this assumption explicitly and qualify the no-infrastructure-change claim, or the flexibility argument is overstated.
  3. [§4.1, Table 3] The main security results are reported as point estimates from a single optimization run, with no error bars or significance tests. The only multi-run study (Section 4.5) reports ASR 2.81 ± 1.09 on SEP for Llama3.1-8B; this variability is large relative to the differences used to support comparability, e.g., TaskTracker ASR of 0.24% for DefensiveToken versus 0.20% to 0.51% for the training-time baselines. Without repeated runs or confidence intervals, the claim that DefensiveToken is statistically indistinguishable from training-time defenses on optimization-free benchmarks is not established.
minor comments (7)
  1. [Abstract] The abstract says append a few DefensiveTokens before the LLM input, but the method prepends these tokens; prepend would be more accurate and consistent with Section 3.3.
  2. [§1] There is a typo in 'an one-robust-model-fit-all solution' in the Introduction; it should be 'a one-size-fits-all robust model' or similar.
  3. [§3.3] The notation 't=(t1,t2,...,tn) in t in R^{n x e}' contains a duplicated 'in t in' and should read 't=(t1,...,tn) in R^{n x e}'.
  4. [§4.5] The phrase 'We tune the learning rates exponentially' is unclear; the authors likely mean that learning rates were varied by orders of magnitude or logarithmically.
  5. [Table 3] On Falcon3-7B and Qwen2.5-7B, the TextGrad baseline has higher AlpacaFarm ASR (97.1% and 97.6%) than the no-defense model (84.6% and 93.3%); the paper does not discuss this counterproductive outcome of the optimized hard prompt.
  6. [§2] Reference [21] has a typo in the URL: 'ttps://github.com/huggingface/peft' should be 'https://github.com/huggingface/peft'.
  7. [§1] Section 1 spells 'DefensiveTokens' as 'Defensivetokens' in the sentence 'as Defensivetokens are flexible'; the capitalization should be consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: DefensiveToken's security and utility claims are evaluated empirically against external benchmarks and independent baselines.

full rationale

The paper's central claim is that optimizing five prepended token embeddings with the StruQ loss yields a test-time prompt-injection defense comparable to training-time defenses. This claim is not derived from its own components by construction: the embeddings are trained on the Cleaned Alpaca dataset and then evaluated on disjoint benchmarks (AlpacaFarm, SEP, TaskTracker, CyberSecEval2, InjecAgent), with attack samples and user instructions that do not overlap the training data. The training objective in Eq. (1) is the standard negative log-likelihood on defensively labeled data, and the evaluation measures attack success on held-out data; matching the training objective to the evaluation metric is ordinary supervised learning, not circularity. The paper does rely on the authors' prior StruQ loss, dataset recipe, and SecAlign hyperparameters, but those are independent building blocks with their own external evaluations, and the present paper's contribution is tested against them as baselines rather than assumed from them. The reviewer concern that the headline 'comparable to training-time' overstates the adaptive GCG results (e.g., DefensiveToken 48.8% average GCG-ASR vs. 28.8-38.4% for training-time baselines) is a correctness/overclaim issue, not a circularity issue, since the numbers are reported openly in Table 3 and Figure 2. No equation-level reduction, fitted-parameter rename, or self-citation chain forces the claimed result.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The method's central claim depends on several design choices (token count, learning rate, initialization) that were selected using the AlpacaFarm benchmark, on a threat model inherited from the authors' prior work, and on an infrastructure assumption (raw embedding input) that is not currently standard in hosted LLM APIs.

free parameters (3)
  • Number of defensive tokens = 5
    Chosen after ablations on AlpacaFarm; 5 tokens are sufficient for all four models; 1 is insufficient for Falcon3 and Qwen2.5, and 20 hurts utility.
  • Learning rate for token embedding optimization = 0.1
    Tuned exponentially in Table 8; 0.01 gives poor security and 1.0 destabilizes training.
  • Random initialization scale = N(0, I)
    Ablated in Table 5; random initialization outperforms initialization with space or text embeddings.
assumptions (4)
  • domain assumption The attacker can inject instructions into the data portion but cannot modify the prompt template or DefensiveToken embeddings (threat model, Section 3.1).
    This is the standard threat model from StruQ/SecAlign; if the attacker could modify the embeddings or template, the defense would fail.
  • domain assumption The gpt-4o LLM judge correctly determines attack success and utility on all benchmarks (Section 4.1).
    ASR and WinRate scores depend on this judge; no calibration or human validation is reported.
  • domain assumption Training on the Cleaned Alpaca instruction dataset with two injection variants generalizes to the evaluation benchmarks (Section 4.1).
    The defensive tokens are optimized on 51k Cleaned Alpaca samples; the paper's claim of generalization rests on the overlap being low between training and test distributions.
  • ad hoc to paper The LLM can accept raw embedding vectors as prepended soft tokens (Sections 3.3 and 4.1).
    DefensiveTokens are continuous vectors, not tokens in the vocabulary; this is an implementation requirement not available in all serving stacks.
invented entities (1)
  • DefensiveTokens (continuous embedding vectors treated as special tokens)
    purpose: Preprended to LLM input to make the model ignore injected instructions in the data section
    A methodological artifact, not a physical entity. The only support is the paper's own benchmark evaluations; no external or predicted observable is offered, though the GitHub release allows others to rerun.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Defending Against Prompt Injection With a Few DefensiveTokens." pith.science (2026). https://pith.science/paper/Z2PQDCC3

@misc{pith2026250707974,
  author       = {Pith},
  title        = {Pith review of: Defending Against Prompt Injection With a Few DefensiveTokens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z2PQDCC3}},
  note         = {Machine review of arXiv:2507.07974}
}
read the original abstract

When large language model (LLM) systems interact with external data to perform complex tasks, a new attack, namely prompt injection, becomes a significant threat. By injecting instructions into the data accessed by the system, the attacker is able to override the initial user task with an arbitrary task directed by the attacker. To secure the system, test-time defenses, e.g., defensive prompting, have been proposed for system developers to attain security only when needed in a flexible manner. However, they are much less effective than training-time defenses that change the model parameters. Motivated by this, we propose DefensiveToken, a test-time defense with prompt injection robustness comparable to training-time alternatives. DefensiveTokens are newly inserted as special tokens, whose embeddings are optimized for security. In security-sensitive cases, system developers can append a few DefensiveTokens before the LLM input to achieve security with a minimal utility drop. In scenarios where security is less of a concern, developers can simply skip DefensiveTokens; the LLM system remains the same as there is no defense, generating high-quality responses. Thus, DefensiveTokens, if released alongside the model, allow a flexible switch between the state-of-the-art (SOTA) utility and almost-SOTA security at test time. The code is available at https://github.com/Sizhe-Chen/DefensiveToken.

Figures

Figures reproduced from arXiv: 2507.07974 by the authors.

Figure 1
Figure 1. If the LLM provider releases DefensiveTokens along [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The security of DefensiveToken vs. existing test-time and training-time baselines. The values are averaged across all [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The utility-security trade-off on AlpacaFarm and SEP. The triangles mark test-time defenses, and the squares mark [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Data Leakage Prevention in Agentic Applications via Preemptive Hardening

    cs.CR 2026-07 conditional novelty 6.0 of 10

    A build-time pipeline that scans, patches, and validates agentic LLM apps reduced prompt-injection leakage to 0% on most tested apps and by 91% on the hardest stress case.

  2. Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domain

    cs.IR 2025-09 conditional novelty 6.0 of 10

    Misleading health documents in RAG context sharply lower LLM accuracy, and heavily helpful-biased retrieval pools restore it.

  3. Secure Multi-LLM Agentic AI and Agentification for Edge General Intelligence by Zero-Trust: A Survey

    cs.NI 2025-08 conditional novelty 4.0 of 10

    A survey proposing zero-trust architecture for multi-LLM systems in edge computing, with a taxonomy of model- and system-level defenses and a conceptual framework.

Reference graph

Works this paper leans on

59 extracted references · 29 canonical work pages · cited by 3 Pith papers

  1. [1]

    Sahar Abdelnabi, Aideen Fay, Giovanni Cherubin, Ahmed Salem, Mario Fritz, and Andrew Paverd. 2025. Get my drift? Catching LLM Task Drift with Activation Deltas. arXiv:2406.00799 https://arxiv.org/abs/2406.00799

    arXiv 2025

  2. [2]

    Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, et al. 2024. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models. arXiv:2404.13161 https://arxiv.org/abs/2404.13161

    arXiv 2024

  3. [3]

    Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2025. StruQ: Defending against prompt injection with structured queries. In USENIX Security Symposium. https://arxiv.org/abs/2402.06363

    arXiv 2025

  4. [4]

    Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, Kamalika Chaudhuri, David Wagner, and Chuan Guo. 2025. SecAlign: Defending Against Prompt Injection with Preference Optimization. In The ACM Conference on Computer and Communications Security (CCS). https://arxiv.org/abs/2410.05451

    arXiv 2025

  5. [5]

    Sizhe Chen, Arman Zharmagambetov, David Wagner, and Chuan Guo. 2025. Meta SecAlign: A Secure Foundation LLM Against Prompt Injection Attacks. arXiv:2507.02735 (2025). https://arxiv.org/abs/2507.02735

    arXiv 2025

  6. [6]

    Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: an open platform for evaluating LLMs by human preference. In International Conference on Machine Learning (ICML) . JMLR.org, Article 331, 30 pages. https://dl.acm....

    doi:10.5555/3692070 2024

  7. [7]

    Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Car- lini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis, and Flo- rian Tramèr. 2025. Defeating prompt injections by design. arXiv preprint arXiv:2503.18813 (2025). https://arxiv.org/abs/2503.18813

    arXiv 2025

  8. [8]

    Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feed- back. In Advances in Neural Information Processing Systems (NeurIPS) . Article 1308, 31 pages. https://dl.acm.org/doi/10.5555/366612...

    work page arXiv 2023

Show all 59 references
  1. [9]

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not What You’ve Signed Up For: Compro- mising Real-World LLM-Integrated Applications with Indirect Prompt Injec- tion. In ACM Workshop on Artificial Intelligence and Securit...

    2023

  2. [10]

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations (ICLR) . https: //arxiv.org/abs/2106.09685

    2022 arXiv

  3. [11]

    Hsu, and Pin-Yu Chen

    Kuo-Han Hung, Ching-Yun Ko, Ambrish Rawat, I-Hsin Chung, Winston H. Hsu, and Pin-Yu Chen. 2025. Attention Tracker: Detecting Prompt Injection Attacks in LLMs. In Findings of the Association for Computational Linguistics (NAACL) . 2309–2322. https://aclanthology.org/2025.findin...

    2025

  4. [12]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. DSPy: Compiling Declarative Language Model Calls into Self...

    2024

  5. [13]

    Learn Prompting. 2023. Learn Prompting: Your guide to communicating with AI. https://learnprompting.org

    2023

  6. [14]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. In Empirical Methods in Natural Language Processing (EMNLP), Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). 3045–3059. doi:10.18653...

    2021 doi

  7. [15]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. AlpacaEval: An Au- tomatic Evaluator of Instruction-following Models. https://github.com/tatsu- lab/alpaca_eval

    2023

  8. [16]

    Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. InAssociation for Computational Linguistics (ACL). 4582–

    2021

  9. [17]

    Huawei Lin, Yingjie Lao, Tong Geng, Tan Yu, and Weijie Zhao. 2025. UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adver- sarial Attacks in Large Language Models. https://arxiv.org/abs/2502.13141

    2025 arXiv

  10. [18]

    Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. 2024. Automatic and Universal Prompt Injection Attacks against Large Language Mod- els. https://arxiv.org/abs/2403.04957

    2024 arXiv

  11. [19]

    Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong

  12. [20]

    Yupei Liu, Yuqi Jia, Jinyuan Jia, Dawn Song, and Neil Zhanqiang Gong. 2025. DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks . In IEEE Symposium on Security and Privacy (SP) . IEEE Computer Society, 2190–2208. doi:10.1109/SP61157.2025.00250

    2025

  13. [21]

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods. ttps://github.com/huggingface/peft

    2022

  14. [22]

    Meta. 2024. Prompt Guard. https://llama.meta.com/docs/model-cards-and- prompt-formats/prompt-guard

    2024

  15. [23]

    Yichuan Mo, Yuji Wang, Zeming Wei, and Yisen Wang. 2024. Fight back against jailbreaking via prompt adversarial tuning. In Annual Conference on Neural Infor- mation Processing Systems (NeurIPS) . https://proceedings.neurips.cc/paper_files/ paper/2024/file/759ca99a82e2a9137c6be...

    2024

  16. [24]

    OpenAI. 2024. Safety evaluations hub. https://openai.com/safety/evaluations- hub

    2024

  17. [25]

    OpenAI. 2025. ChatGPT Agent System Card. (2025). https://cdn.openai.com/ pdf/839e66fc-602c-48bf-81d3-b21eacc3459d/chatgpt_agent_system_card.pdf

    2025

  18. [26]

    OWASP. 2023. OWASP Top 10 for LLM Applications. https://llmtop10.com

    2023

  19. [27]

    Dario Pasquini, Martin Strohmeier, and Carmela Troncoso. 2024. Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks. In Workshop on Artificial Intelligence and Security (AISec) . 89–100. doi:10.1145/ 3689932.3694764

    2024

  20. [28]

    Julien Piet, Maha Alrashed, Chawin Sitawarin, Sizhe Chen, Zeming Wei, Elizabeth Sun, Basel Alomair, and David Wagner. 2023. Jatmo: Prompt Injection Defense by Task-Specific Finetuning. In European Symposium on Research in Computer Security (ESORICS). 105–124. doi:10.1007/978-3...

    2023 doi

  21. [29]

    PromptArmor. 2024. Data Exfiltration from Slack AI via indirect prompt injection. https://promptarmor.substack.com/p/data-exfiltration-from-slack-ai-via

    2024

  22. [30]

    Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng

  23. [31]

    Embrace The Red. 2025. ChatGPT Operator: Prompt Injection Exploits & De- fenses. https://embracethered.com/blog/posts/2025/chatgpt-operator-prompt- injection-exploits

    2025

  24. [32]

    Johann Rehberger. 2023. Hacking Google Bard - From Prompt Injection to Data Exfiltration. https://embracethered.com/blog/posts/2023/google-bard-data- exfiltration

    2023

  25. [33]

    Johann Rehberger. 2024. ZombAIs: From Prompt Injection to C2 with Claude Computer Use. https://embracethered.com/blog/posts/2024/claude-computer- use-c2-the-zombais-are-coming

    2024

  26. [34]

    Gene Ruebsamen. 2024. Cleaned Alpaca Dataset. https://github.com/gururise/ AlpacaDataCleaned

    2024

  27. [35]

    Sander Schulhoff. 2024. Instruction Defense. https://learnprompting.org/docs/ prompt_hacking/defensive_measures/instruction

    2024

  28. [36]

    Sander Schulhoff. 2024. Sandwich Defense. https://learnprompting.org/docs/ prompt_hacking/defensive_measures/sandwich_defense

    2024

  29. [37]

    Chongyang Shi, Sharon Lin, Shuang Song, Jamie Hayes, Ilia Shumailov, Itay Yona, Juliette Pluto, Aneesh Pappu, Christopher A Choquette-Choo, Milad Nasr, et al

  30. [38]

    Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. 2024. The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv:2404.13208 https://arxiv.org/abs/2404.13208

    2024 arXiv

  31. [39]

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How Does LLM Safety Training Fail?. In Neural Information Processing Sys- tems (NeurIPS) . https://proceedings.neurips.cc/paper_files/paper/2023/hash/ fd6613131889a4b656206c50a8bd7790-Abstract-Conference.html

    2023

  32. [40]

    Zeming Wei, Yifei Wang, and Yisen Wang. 2024. Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations. In International Conference on Machine Learning (ICML) . https://arxiv.org/abs/2310.06387

    2024 arXiv

  33. [41]

    Simon Willison. 2022. Prompt Injection Attacks against GPT-3. https:// simonwillison.net/2022/Sep/12/prompt-injection/

    2022

  34. [42]

    Wang, and Prateek Mittal

    Tong Wu, Chong Xiang, Jiachen T. Wang, and Prateek Mittal. 2025. Effectively Controlling Reasoning Models through Thinking Intervention. arXiv:2503.24370 https://arxiv.org/abs/2503.24370

    2025 arXiv

  35. [43]

    Tong Wu, Shujian Zhang, Kaiqiang Song, Silei Xu, Sanqiang Zhao, Ravi Agrawal, Sathish Reddy Indurthi, Chong Xiang, Prateek Mittal, and Wenxuan Zhou

  36. [44]

    Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. 2023. Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment. https://arxiv.org/abs/2312.12148

    2023 arXiv

  37. [45]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR) . https://par.nsf. gov/servlets/purl/10451467

    2023

  38. [46]

    Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2025. Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models. In ACM SIGKDD Conference on Defending Against Prompt Injection With a Few DefensiveToke...

    2025 doi

  39. [47]

    Li Yin and Zhangyang Wang. 2025. LLM-AutoDiff: Auto-Differentiate Any LLM Workflow. https://ui.adsabs.harvard.edu/abs/2025arXiv250116673Y/abstract

    2025

  40. [48]

    In International Conference on Learning Representations (ICLR)

    Instructional Segment Embedding: Improving LLM Safety with Instruc- tion Hierarchy. In International Conference on Learning Representations (ICLR) . https://arxiv.org/abs/2410.09102

    arXiv

  41. [49]

    Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents. In Findings of the Association for Computational Linguistics (ACL) . Association for Computational Linguistics, Ba...

    2024 doi

  42. [50]

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. 2023. Pytorch FSDP: experiences...

    2023 doi

  43. [51]

    Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. 2024. On Prompt-Driven Safeguarding for Large Language Models. In International Conference on Machine Learning (ICML) . 61593–61613. https://arxiv.org/abs/2401.18018

    2024 arXiv

  44. [52]

    Andy Zhou, Bo Li, and Haohan Wang. 2024. Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks. In Annual Conference on Neural Information Processing Systems (NeurIPS) . https://arxiv.org/abs/2401. 17263

    2024

  45. [53]

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. 2025. Optimizing Generative AI by Backpropagating Language Model Feedback. In Nature, Vol. 639. 609–616. doi:10.1038/s41586-025-08661-4

    2025 doi

  46. [54]

    Egor Zverev, Sahar Abdelnabi, Mario Fritz, and Christoph H Lampert. 2025. Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?. In International Conference on Learning Representations (ICLR) . https: //openreview.net/pdf?id=8EtSBX41mt

    2025

  47. [58]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. https://arxiv.org/abs/2307.15043

    2023 arXiv

  48. [2023]

    Gradient Descent

    Automatic Prompt Optimization with “Gradient Descent” and Beam Search. In Empirical Methods in Natural Language Processing (EMNLP) . 7957–7968. doi:10. 18653/v1/2023.emnlp-main.494

    2023

  49. [2024]

    In USENIX Security Symposium

    Formalizing and benchmarking prompt injection attacks and defenses. In USENIX Security Symposium. 1831–1847. https://www.usenix.org/conference/ usenixsecurity24/presentation/liu-yupei

  50. [2025]

    arXiv preprint arXiv:2505.14534 (2025)

    Lessons from Defending Gemini Against Indirect Prompt Injections. arXiv preprint arXiv:2505.14534 (2025). https://arxiv.org/abs/2505.14534

    2025 arXiv

  51. [4597]

    doi:10.18653/v1/2021.acl-long.353

    2021 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.