Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

PromptShield: Deployable Detection for Prompt Injection Attacks

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A detector that catches 65% of prompt injections while flagging only 0.1% of benign traffic.

desk verdict A useful, publicly released benchmark and an honest paper whose headline detection number is in-sample until the threshold is re-calibrated on held-out data. read the letter →

arxiv 2501.15145 v2 pith:O6KLQYOC submitted 2025-01-25 cs.CR

Dennis Jacob , Hend Alzahrani , Zhanhao Hu , Basel Alomair , David Wagner This is my paper · ORCID
classification cs.CR
keywords promptinjectiondetectionlargelanguagemodelsbenchmarkfine-tuningfalsepositiveratelow-FPRdeploymentLLMsecurity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PromptShield is a benchmark and detector built around a deployment-centric view: prompt injection detectors should be judged at the false positive rates that real traffic forces, because benign requests vastly outnumber attacks. The benchmark splits the input space into conversational data (declared always benign) and application-structured data (the only injection surface), and its train/evaluation split is engineered so detectors cannot memorize the evaluation distribution. Fine-tuning Llama 3.1 8B on this benchmark yields a detector that catches 65.3% of prompt injections at 0.1% false positive rate on the evaluation split, while PromptGuard, the strongest prior scheme, catches 9.4% at the same FPR. The paper concludes that careful data curation plus larger base models, not novel architectures, are what make low-FPR deployment feasible.

What carries the argument

The central mechanism is the PromptShield benchmark itself: a two-category taxonomy (conversational data treated as always benign, application-structured data treated as the only injection surface) instantiated as a curated mix of benign corpora (Ultrachat, LMSYS, Alpaca, databricks-dolly, natural-instructions, SPP, IFEval) and attack corpora (FourAttacks built from StruQ's naive/ignore/completion/combined templates, HackAPrompt, OpenPromptInject). Its train/evaluation split is designed to prevent memorization by using disjoint datasets and disjoint injection link phrases across the two splits. The companion deployment scheme selects a decision threshold from the ROC curve to hit target false positive rates of 1%, 0.5%, 0.1%, and 0.05%, and reports true positive rates at those thresholds; this calibration is what separates the paper's numbers from default-threshold accuracy.

What would settle it

Take the final PromptShield model, choose its threshold to hit 0.1% FPR on a held-out validation set (not the evaluation split), then measure FPR and TPR on a corpus of multi-turn chatbot sessions that include web-browsing or file-upload content where an attacker has embedded an injected instruction; if the empirical FPR exceeds the target by a wide margin or the detector misses those embedded injections, the paper's deployability claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that a prompt injection detector can be highly sensitive and still deployable if it is fine-tuned on a benchmark that mirrors the real distribution of LLM traffic and evaluated at threshold-calibrated low false positive rates. On the PromptShield evaluation split, the fine-tuned Llama-3.1-8B detector achieves 94.8% true positive rate at 1% FPR, 87.8% at 0.5% FPR, 65.3% at 0.1% FPR, and 47.5% at 0.05% FPR, while PromptGuard, the strongest prior scheme, achieves 12.8%, 12.4%, 9.4%, and 1.5% at the same FPRs. Ablations show that enlarging the training set from 1K to 20K samples more than doubles true positive rate at the most stringent FPRs, and that adding conversational training data cuts false positives on chatbot traffic by about 25% at the least strict threshold while a modest TPR cost appears only at the lowest FPRs. The authors conclude that careful curation and larger base models, rather than architectural innovation, drive performance in the deployment-relevant regime.

Load-bearing premise

The entire 'deployable' framing rests on the assumption that real traffic is fully described by two categories—conversational requests, which are always benign, and application-structured data, which is the only place injections appear—with multi-turn chats, function calling, and injections inside conversational content excluded as out of scope.

Editorial extensions

If this is right

  • With the deployment scheme's threshold in place, a provider-deployed detector can block roughly two-thirds of prompt injections at 0.1% FPR, meaning fewer than one in a thousand benign requests gets refused.
  • Reporting TPR at target FPRs instead of AUC exposes that prior detectors (PromptGuard, ProtectAI, InjecGuard, Fmops) are effectively non-functional in the low-FPR regime; the paper suggests low-FPR TPR should become the standard metric.
  • Because scaling the training set from 1K to 20K samples and the base model from 61M to 8B parameters both substantially raise low-FPR TPR, the results point to data and model scale as the practical levers for detector improvement.
  • The extendable benchmark construction means newly published attacks and datasets can be added, so future detectors can be trained against a moving attack distribution rather than a static one.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If real chatbots gain web-browsing or file-upload features, an attacker can plant an injection inside content a user requests, which the benchmark's taxonomy would still label as benign conversational data; testing PromptShield on such embedded injections would show whether its low-FPR advantage survives outside the taxonomy.
  • The paper admits it selected decision thresholds using the evaluation split and recommends a validation split instead; re-running the protocol with a disjoint calibration set is the direct way to measure how much the headline 65.3%-at-0.1% figure is an artifact of evaluation-split threshold tuning.
  • Since optimization-based attacks such as GCG are excluded from scope, the claimed performance is established only against template-based attacks; an adaptive adversary who optimizes a suffix to flip the detector's output would test whether the low-FPR edge holds under attack.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces PromptShield, a benchmark and a fine-tuned detector for prompt injection attacks in LLM-integrated applications. It defines a deployment-oriented taxonomy separating conversational data from application-structured data, curates benign and attack samples from public datasets and published attack strategies, and provides non-overlapping train and evaluation splits. The PromptShield detector is fine-tuned from several base models, and the authors report AUC and TPR at low target FPRs using a threshold-calibration procedure. The central empirical claim is that the Llama-3.1-8B variant detects 65.3% of attacks at 0.1% FPR on the evaluation split, far outperforming prior detectors such as PromptGuard (9.4%). The paper also includes ablations on model size, training set size, and training data composition, plus a limitations section.

Significance. If the reported performance holds out of sample, the paper makes a useful contribution: it provides a public, reusable benchmark with an explicitly deployment-oriented taxonomy, emphasizes the low-FPR regime that is relevant to practical deployment, and shows a substantial improvement over existing detectors. The curation effort is significant, and the release of the benchmark and code increases the value of the work. At the same time, the headline quantitative claim is currently an in-sample quantity because thresholds are calibrated on the evaluation split, and the single-run results carry no uncertainty estimates. The benchmark contribution is likely to be useful regardless of the exact detector numbers, but the 'deployable' conclusion requires the evaluation protocol to be fixed or carefully qualified.

major comments (3)
  1. [Section 3.2.2 and Table 4] The headline result in Section 1 and Table 4 is obtained by selecting decision thresholds on the evaluation split itself. Section 3.2.2 states that the authors "cache model output scores on the evaluation split" and interpolate the ROC curve on that split to choose thresholds, and it acknowledges that the validation split should have been used. This makes the reported 65.33% TPR at 0.1% FPR an in-sample fitted quantity: the FPR is enforced by construction, and the TPR at that threshold has no demonstrated out-of-sample counterpart. The problem is especially severe at 0.1% FPR because the validation split described in Section 3.2.1 has only about 1000 points, so the targeted FPR corresponds to roughly one false positive. Please recalibrate thresholds on a held-out split and report the resulting TPR, or report both in-sample and held-out values. The AUC numbers in Table 4 and the relative ordering of detectors are less affected, but the absolute deployability claim is not yet supported. Appendix A.3 already notes that the validation-split choice causes variation at very low FPR, which further cautions against treating the point estimate as stable.
  2. [Appendix A.3, Tables 5 and 6] All results are single runs without confidence intervals or variance estimates. Appendix A.3 reports that the choice of training/validation split changes performance at extremely low FPR levels, and Tables 5 and 6 contain several differences of only a few percentage points in this regime (for example, FLAN-T5-base versus Llama-3.2-1B at 0.05% FPR, or the 5K versus 10K training-set comparison at 0.05% FPR). Without multiple seeds or split resamples, the reader cannot determine whether these comparisons are meaningful. Please add an uncertainty estimate for at least the primary PromptShield configuration and for the model-size and training-size comparisons.
  3. [Sections 2.3, 2.4, and 7] The deployability claim is scoped to a specific taxonomy: all conversational traffic is treated as benign, and multi-turn interaction and function calling are explicitly excluded in Section 2.4. The benchmark labels first-turn conversational data from Ultrachat and LMSYS as benign, while real provider-deployed systems must also handle indirect injections embedded in fetched web content, uploaded files, and multi-turn state. The reported low FPR may therefore not transfer to deployments that include these out-of-scope categories. This is a defensible scoping decision, but the paper should either evaluate on a small sample of such traffic or explicitly restrict the "deployable" conclusion to the single-turn, text-only, two-category setting considered in the benchmark.
minor comments (5)
  1. [Section 5.1] The sentence "they report a TPR of 71% and FPR of 1% [7]" cites the Llama 3 herd technical report; the intended source appears to be the CYBERSECEVAL 3 report [30]. Please correct the citation.
  2. [Section 4] There is a typo in "plateaues"; it should read "plateaus."
  3. [Appendix B.2] The fact that PromptGuard's "jailbreak" class is used as a proxy for prompt injection is important for interpreting Table 4; consider noting this caveat in Section 5.1 where PromptGuard is first discussed.
  4. [Table 9] The caption and table for the conversational-only evaluation leave the AUC column blank; add a short explanation (for example, that ROC analysis cannot be computed on a single-class set) in the caption.
  5. [Table 4 footnote] The footnote "there does not exist a threshold that achieves the desired FPR aside from 1.0" is ambiguous; clarify whether the issue is tied score values or some other property of the score distribution.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline TPR@0.1% FPR is obtained by calibrating the decision threshold on the evaluation split itself; the reported FPR is enforced by construction, so the absolute deployable claim is in-sample.

  1. fitted input called prediction [Section 3.2.2 (Deployment scheme), Table 4, and Section 1 headline result]
    "we cache model output scores on the evaluation split and compute both true positive rates (TPR) and false positive rates (FPR) across a range of decision thresholds; these values are used to build a receiver operating characteristic (ROC) curve. We then use linear interpolation on the curve to find a threshold that results in a FPR close to our target... Note that in real-life deployment settings model maintainers will not necessarily have access to test data. In retrospect, we should have used the validation split for this calibration step."

    The headline claim, 'our model detects 65.3% of prompt injection attacks with 0.1% FPR on our benchmark's evaluation split,' is produced by selecting the decision threshold on that same evaluation split. Because the threshold is interpolated on the eval-split ROC curve to force FPR near 0.1%, the reported 0.1% FPR holds by construction on that split, and the associated TPR is an in-sample value rather than an out-of-sample deployment prediction. The paper explicitly concedes this: 'In retrospect, we should have used the validation split for this calibration step.' Thus Table 4's TPR@FPR numbers, including the 65.33% figure, are fitted to the evaluation split; only the AUC column and the relative ordering of detectors are threshold-free evidence.

full rationale

The paper contains one significant circular step: the headline TPR at 0.1% FPR is obtained by calibrating the threshold on the evaluation split, and the FPR is enforced by construction on that same split. The paper itself acknowledges this in Section 3.2.2, recommending that future work use the validation split for calibration. This affects the absolute 'deployable' performance numbers, although the comparison with competitors is less affected because the same evaluation-split calibration is applied to all models. The AUC results are threshold-free and provide independent evidence of ranking, but they do not rescue the specific headline TPR-at-low-FPR numbers as out-of-sample predictions. I found no load-bearing self-citation circularity: citations to prior work by the same group (e.g., StruQ for attack implementations) are used for data construction, not to justify the central detection claim. The benchmark itself is composed of external datasets, and the train/evaluation split is by dataset, so the model is not trained on the evaluation data. The core circularity is therefore confined to the threshold-calibration protocol, which is a fitted input presented as a deployment prediction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central performance numbers rest on a small number of domain assumptions rather than on fitted mathematical parameters. The most important are that conversational data is benign, that the disjoint evaluation split measures out-of-distribution generalization, and that the PromptGuard jailbreak output is a valid proxy for the paper's definition of prompt injection. The decision thresholds are the main free parameters, and they are fitted to the evaluation split, which is the paper's acknowledged weakness. No new entities are introduced.

free parameters (2)
  • Decision thresholds for target FPRs = not reported; fitted to evaluation split to hit 1%, 0.5%, 0.1%, 0.05% FPR
    Section 3.2.2 uses the evaluation split to choose thresholds, so the reported TPR@FPR is measured at a threshold chosen on the same data.
  • LMSYS toxicity filtering threshold = 1% toxicity score
    Appendix A.1 sets a strict 1% threshold with OpenAI moderation to remove toxic content; this hand-chosen threshold shapes the benign conversational data distribution.
assumptions (3)
  • domain assumption Conversational data is essentially always benign and free of prompt injection risk.
    Stated in Sections 2.3 and 2.4, used to label all conversational data as benign in the benchmark. If false, the benchmark mislabels attacks as benign and the deployability claim weakens.
  • domain assumption The evaluation split is out-of-distribution relative to the training split and representative of deployment.
    Section 3.1.3 relies on disjoint datasets and disjoint injection link phrases to establish OOD evaluation; if the attack structures remain too similar, the evaluation overstates generalization.
  • domain assumption PromptGuard's 'jailbreak' output approximates prompt injection detection.
    Appendix B.2: after developer communication, the paper uses the jailbreak label as a proxy for prompt injection, which may not perfectly match the paper's definition.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PromptShield: Deployable Detection for Prompt Injection Attacks." pith.science (2026). https://pith.science/paper/O6KLQYOC

@misc{pith2026250115145,
  author       = {Pith},
  title        = {Pith review of: PromptShield: Deployable Detection for Prompt Injection Attacks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O6KLQYOC}},
  note         = {Machine review of arXiv:2501.15145}
}
read the original abstract

Application designers have moved to integrate large language models (LLMs) into their products. However, many LLM-integrated applications are vulnerable to prompt injections. While attempts have been made to address this problem by building prompt injection detectors, many are not yet suitable for practical deployment. To support research in this area, we introduce PromptShield, a benchmark for training and evaluating deployable prompt injection detectors. Our benchmark is carefully curated and includes both conversational and application-structured data. In addition, we use insights from our curation process to fine-tune a new prompt injection detector that achieves significantly higher performance in the low false positive rate (FPR) evaluation regime compared to prior schemes. Our work suggests that careful curation of training data and larger models can contribute to strong detector performance.

Figures

Figures reproduced from arXiv: 2501.15145 by the authors.

Figure 1
Figure 1. PromptShield for prompt injection detection. Realistic deployment settings require the ability to handle both [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Our scheme performs far better than all prior de [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Deployment scheme for the PromptShield detector. In the left panel we obtain raw output scores. In the middle panel [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Data Leakage Prevention in Agentic Applications via Preemptive Hardening

    cs.CR 2026-07 conditional novelty 6.0 of 10

    A build-time pipeline that scans, patches, and validates agentic LLM apps reduced prompt-injection leakage to 0% on most tested apps and by 91% on the hardest stress case.

  2. PromptArmor: Simple yet Effective Prompt Injection Defenses

    cs.CR 2025-07 conditional novelty 5.0 of 10

    PromptArmor prompts a separate LLM to detect and remove injected prompts from agent inputs, achieving below 1% false positive and false negative rates on AgentDojo.

  3. Quantifying Conversation Drift in MCP via Latent Polytope

    cs.CL 2025-08 reject novelty 4.0 of 10

    SecMCP flags MCP conversation drift by thresholding per-layer activation distances from benign anchors, reporting AUROC above 0.915 on Llama3, Vicuna, and Mistral.

Reference graph

Works this paper leans on

43 extracted references · 11 canonical work pages · cited by 3 Pith papers

  1. [1]

    Synthetic Python Problems(SPP) Dataset

    2023. Synthetic Python Problems(SPP) Dataset. https://huggingface.co/datasets/ wuyetao/spp

    work page 2023

  2. [2]

    Blueteam AI. 2024. Fmops/Distilbert-Prompt-Injection. https://huggingface.co/ fmops/distilbert-prompt-injection. 10 PromptShield: Deployable Detection for Prompt Injection Attacks Table 7: Ablation experiment, where we measure the effect of training on conversational data, evaluated on all test data. Training Set AUC TPR𝛼 FPR𝛼 TPR𝛽 FPR𝛽 TPR𝛿 FPR𝛿 TPR𝛾 FPR...

    work page 2024

  3. [3]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating Large Language Models Trained on Code. doi:10.48550/arXiv.2107.03374 arXiv:2107.03374 [cs]

    arXiv doi:10.48550/arxiv.2107.03374 2021

  4. [4]

    Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2024. StruQ: Defending Against Prompt Injection with Structured Queries. InUSENIX Security

    work page 2024

  5. [5]

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin

    work page

  6. [6]

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing Chat Language Models by Scaling High-quality Instructional Conversations. In EMNLP 2023 . arXiv. doi:10.48550/arXiv.2305.14233 arXiv:2305.14233 [cs]

    arXiv doi:10.48550/arxiv.2305.14233 2023

  7. [7]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The Llama 3 Herd of Models. doi:10.48550/arXiv.2407. 21783 arXiv:2407.21783 [cs]

    arXiv doi:10.48550/arxiv.2407 2024

  8. [8]

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not What You’ve Signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. In CCS 2023 Workshop on Artificial Intelligence and Security (AISec 2023) . arXiv. doi:10.48550/arXiv.2302.12173 arXiv:2302.12173 [cs]

    arXiv doi:10.48550/arxiv.2302.12173 2023

Show all 43 references
  1. [9]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DeBERTaV3: Improv- ing DeBERTa Using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing. In ICLR 2023 . arXiv. doi:10.48550/arXiv.2111.09543 arXiv:2111.09543 [cs]

    2023 doi arXiv

  2. [10]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR 2022 . arXiv. doi:10.48550/arXiv.2106.09685 arXiv:2106.09685 [cs]

    2021 doi arXiv

  3. [11]

    Hsu, and Pin-Yu Chen

    Kuo-Han Hung, Ching-Yun Ko, Ambrish Rawat, I.-Hsin Chung, Winston H. Hsu, and Pin-Yu Chen. 2024. Attention Tracker: Detecting Prompt Injection Attacks in LLMs. doi:10.48550/arXiv.2411.00348 arXiv:2411.00348 [cs]

    2024 doi arXiv

  4. [12]

    Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. Challenges and Applications of Large Language Models. doi:10.48550/arXiv.2307.10169 arXiv:2307.10169 [cs]

    2023 doi arXiv

  5. [13]

    Hao Li and Xiaogeng Liu. 2024. InjecGuard: Benchmarking and Mitigating Over- defense in Prompt Injection Guardrail Models. doi:10.48550/arXiv.2410.22770 arXiv:2410.22770 [cs]

    2024 doi arXiv

  6. [14]

    Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong

  7. [15]

    OpenAI. 2023. Text-Davinci-003. https://platform.openai.com/docs/ deprecations

    2023

  8. [16]

    OpenAI. 2024. Omni-Moderation-Latest. https://platform.openai.com/docs/api- reference/moderations

    2024

  9. [17]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, et al . 2024. GPT-4 Technical Report. doi:10.48550/arXiv.2303.08774 arXiv:2303.08774 [cs]

    2024 doi arXiv

  10. [18]

    Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al

  11. [19]

    Fábio Perez and Ian Ribeiro. 2022. Ignore Previous Prompt: Attack Techniques For Language Models. In NeurIPS 2022 Workshop on Machine Learning Safety . arXiv. doi:10.48550/arXiv.2211.09527 arXiv:2211.09527 [cs]

    2022 doi arXiv

  12. [20]

    Julien Piet, Maha Alrashed, Chawin Sitawarin, Sizhe Chen, Zeming Wei, Elizabeth Sun, Basel Alomair, and David Wagner. 2024. Jatmo: Prompt Injection Defense by Task-Specific Finetuning. In ESORICS 2024. arXiv. doi:10.48550/arXiv.2312.17673 arXiv:2312.17673 [cs]

    2024 doi arXiv

  13. [21]

    ProtectAI.com. 2023. Fine-Tuned DeBERTa-v3-base for Prompt Injection Detec- tion. https://huggingface.co/protectai/deberta-v3-base-prompt-injection-v2

    2023

  14. [22]

    ProtectAI.com. 2023. Fine-Tuned DeBERTa-v3 for Prompt Injection Detec- tion. https://huggingface.co/protectai/deberta-v3-base-prompt-injection. doi:10. 57967/hf/2739

    2023

  15. [23]

    Abhinav Rao, Sachin Vashistha, Atharva Naik, Somak Aditya, and Monojit Choud- hury. 2024. Tricking LLMs into Disobedience: Formalizing, Analyzing, and De- tecting Jailbreaks. In LREC-COLING 2024. arXiv. doi:10.48550/arXiv.2305.14965 arXiv:2305.14965 [cs]

    2024 doi arXiv

  16. [24]

    Sander Schulhoff, Jeremy Pinto, Anaum Khan, Louis-François Bouchard, Chen- glei Si, Svetlina Anati, Valen Tagliabue, Anson Liu Kost, Christopher Carnahan, and Jordan Boyd-Graber. 2024. Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Globa...

    2024 doi arXiv

  17. [25]

    Do Anything Now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. In CCS 2024. arXiv. doi:10.48550/arXiv.2308. 03825 arXiv:2308.03825 [cs]

    2024 doi arXiv

  18. [26]

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA Model. https://github.com/tatsu-lab/stanford_ alpaca

    2023

  19. [27]

    Dai, Anja Hauth, Katie Millican, et al

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, et al. 2024. Gemini: A Family of Highly Capable Multimodal Models. doi:10. 48550/arXiv.2312.11805 arXiv:2312.11805 [cs] 11 D...

    2024 doi arXiv

  20. [28]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and Efficient Foundation Language Models. doi:10.48550/arXiv.2302.13971 arXiv:2302.13971 [cs]

    2023 doi arXiv

  21. [29]

    Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. 2024. The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. doi:10.48550/arXiv.2404.13208 arXiv:2404.13208 [cs]

    2024 doi arXiv

  22. [30]

    Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, et al . 2024. CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models...

    2024 doi arXiv

  23. [31]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. In ACL 2023. arXiv. doi:10.48550/arXiv. 2212.10560 arXiv:2212.10560 [cs]

    2023 doi arXiv

  24. [32]

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amir- reza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022. Super-NaturalInstructions: Generaliza- tion via Declarative Instructions on 1600+ NLP Tasks....

    2022 doi arXiv

  25. [33]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned Language Models Are Zero-Shot Learners. In ICLR 2022. arXiv. doi:10.48550/arXiv.2109. 01652 arXiv:2109.01652 [cs]

    2022 doi arXiv

  26. [34]

    Zeming Wei, Yifei Wang, Ang Li, Yichuan Mo, and Yisen Wang. 2024. Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations. In ICML 2024. arXiv. doi:10.48550/arXiv.2310.06387 arXiv:2310.06387 [cs]

    2024 doi arXiv

  27. [35]

    Steve Wilson and Ads Dawson. 2024. OWASP Top 10 for LLM Applications 2025

    2024

  28. [36]

    Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2024. Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models. doi:10.48550/arXiv.2312.14197 arXiv:2312.14197 [cs]

    2024 doi arXiv

  29. [37]

    Xing, et al

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhang- hao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, et al. 2024. LMSYS- Chat-1M: A Large-Scale Real-World LLM Conversation Dataset. In ICLR 2024. arXiv. doi:10.48550/arXiv.2309.11998 arXiv:2309.1...

    2024 doi arXiv

  30. [38]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-Following Evaluation for Large Language Models. doi:10.48550/arXiv.2311.07911 arXiv:2311.07911 [cs]

    2023 doi arXiv

  31. [39]

    hello!” are classified as an injection). Communications with the model developers revealed that the “jailbreak

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. doi:10.48550/arXiv.2307.15043 arXiv:2307.15043 [cs] 12 PromptShield: Deployable Detection for Prompt Inject...

    2023 doi arXiv

  32. [2022]

    doi:10.48550/arXiv.2203.02155 arXiv:2203.02155 [cs]

    Training Language Models to Follow Instructions with Human Feedback. doi:10.48550/arXiv.2203.02155 arXiv:2203.02155 [cs]

    doi arXiv

  33. [2023]

    https://www.databricks.com/blog/2023/04/12/dolly-first-open- commercially-viable-instruction-tuned-llm

    Free Dolly: Introducing the World’s First Truly Open Instruction- Tuned LLM. https://www.databricks.com/blog/2023/04/12/dolly-first-open- commercially-viable-instruction-tuned-llm

    2023

  34. [2024]

    In USENIX Security 2024

    Formalizing and Benchmarking Prompt Injection Attacks and Defenses. In USENIX Security 2024. arXiv. doi:10.48550/arXiv.2310.12815 arXiv:2310.12815 [cs]

    2024 doi

  35. [2025]

    doi:10.48550/arXiv.2402.06363 arXiv:2402.06363 [cs]

    arXiv. doi:10.48550/arXiv.2402.06363 arXiv:2402.06363 [cs]

    doi arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.