Pith. sign in

REVIEW 4 major objections 8 minor 1 cited by

OET: Optimization-based prompt injection Evaluation Toolkit

T0 review · 4 major / 8 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces OET, an optimization-based evaluation toolkit for prompt injection, and shows that transferable adversarial strings still defeat current defenses, with SecAlign even increasing attack success on some datasets.

desk verdict A genuinely useful prompt-injection evaluation toolkit with public code, but the 'adaptive' and 'worst-case' claims in the abstract and introduction are not supported by the transfer-only experiments. read the letter →

arxiv 2505.00843 v1 pith:5KPLAFLS submitted 2025-05-01 cs.CR cs.AI

Jinsheng Pan , Xiaogeng Liu , Chaowei Xiao This is my paper · ORCID
classification cs.CRcs.AI
keywords promptinjectionadversarialrobustnessLLMsecurityevaluationtoolkitoptimization-basedattackstransferablestringsadaptivetestingattacksuccessrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OET is a modular evaluation toolkit that converts question-answering data into a uniform prompt format, trains a short adversarial string by optimization toward a target sentence, and then injects that string into test prompts to measure attack success. The paper argues that existing prompt-injection benchmarks are static and cannot capture adaptive, worst-case behavior, and that an optimization-driven workflow fills that gap. In the reported experiments, transferable strings trained by gradient-guided optimization succeed far more often on open-source models than on closed-source models, and two published defenses, StruQ and SecAlign, remain only partially effective. The sharpest result is that SecAlign raises attack success on AQuA and PubMedQA relative to the undefended LLaMA baseline, indicating that a defense can make a model more attackable on some domains. Taken on its own terms, the paper establishes a standardized way to make uneven defense behavior measurable.

What carries the argument

The load-bearing mechanism is the trained adversarial string: a short sequence of tokens inserted between the instruction and the input, optimized so that the model outputs a chosen target sentence, here 'sql injection'. The optimization is carried out with gradient-guided discrete search (GCG), with the attack goal and optimizer configuration as inputs and the trained string as output. The workflow has four stages: data conversion into a unified QA format, adversarial string training on 5 to 15 examples per dataset, inference where the trained string and attack goal are injected into test prompts, and result checking where a custom or default metric, attack success rate, measures whether the target sentence appears in the response. The modular interface allows a user to replace the optimizer or the metric, which is what makes the toolkit a benchmark rather than a one-off experiment.

What would settle it

Re-run the same 500-step gradient-guided optimization directly against SecAlign and StruQ on the exact base model each defense was built on, then compare attack success to the undefended base model. If SecAlign's ASR no longer exceeds the base model on AQuA and PubMedQA, the paper's demonstration that defenses can increase vulnerability would not survive.

Watch

Extended reading notes

Core claim

The paper's central claim is that OET, by training adversarial strings with optimization techniques and transferring them across models and datasets, provides a worst-case-style evaluation of prompt-injection defenses that static benchmarks cannot. On its own results, the claim takes concrete form: LLaMA3.1-8B, Vicuna-7B, and Qwen2-7B-Instruct are attacked successfully most of the time, while GPT-4o-mini and Claude-3.5-sonnet are much harder to attack with the same transferred strings. Against defenses, StruQ drives attack success to zero on six of eight datasets but leaves high success on FinQA and TriviaQA, while SecAlign lowers success on most datasets yet increases it on AQuA and PubMedQA relative to the undefended base model. The authors read these numbers as evidence that current defenses do not generalize across domains and that some defense mechanisms can actively open new vulnerabilities. The discovery, stated on the paper's own terms, is that optimization-based transferable attack strings make such uneven defense behavior measurable in a standardized way.

Load-bearing premise

The conclusions assume that an attack string trained on a few examples and then copied onto other models and data measures how a determined attacker would exploit a defense; if the attacker tuned the string against the actual defense, the reported numbers could change.

Editorial extensions

If this is right

  • If a defense is evaluated only on static test prompts, its reported security could be an artifact; OET-style transferred strings give a stricter, if still transfer-based, lower bound on worst-case behavior.
  • Open-source deployments in security-sensitive settings should assume that similar optimization-based strings will succeed on a large fraction of queries unless specific defenses are shown to work under this benchmark.
  • Defense claims should report per-domain attack success relative to the undefended base model, because SecAlign's increased ASR on AQuA and PubMedQA shows that aggregate or cross-domain improvements can hide localized harm.
  • Because the toolkit lets new optimizers and custom metrics be plugged in, the benchmark can track both attack and defense evolution without waiting for new static datasets.
  • The two closed-source models' low ASR under transferred strings does not establish their worst-case security, but under this test they are clearly harder to attack than the open-source models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment is to optimize the adversarial string directly against each defense model rather than transferring it, which would separate how much of the reported vulnerability is intrinsic to the defense and how much is an artifact of transfer from a different base model.
  • The small training sets (5-15 examples) suggest that the benchmark could be extended to measure how attack success changes with training-set size and domain diversity, giving a data-efficiency curve for red-teaming.
  • If this evaluation style is adopted, one could imagine pairing it with per-domain failure analysis to issue robustness reports that identify precisely which task types cause a defense to backfire.
  • The rise in ASR on SecAlign relative to its base model implies that defenses should be monitored for negative side effects, not just average improvement; the same benchmark could be used as a regression test during defense training.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper introduces OET, a modular and extensible toolkit for benchmarking prompt injection attacks and defenses using optimization-based adversarial string generation. The toolkit provides data conversion, adversarial string training, inference, and result-checking stages, with support for user-defined optimizers and metrics. The authors curate multi-domain QA datasets and report experiments with GCG and other attacks against open- and closed-source LLMs as well as two defenses, StruQ and SecAlign. The central claims are that OET provides an 'adaptive testing framework' that generates 'worst-case adversarial examples' and that state-of-the-art defenses remain inconsistently vulnerable, with SecAlign even increasing ASR on AQuA and PubMedQA relative to the undefended baseline.

Significance. If the adaptive worst-case evaluation claim were substantiated, OET would fill a genuine gap: existing prompt injection benchmarks are largely static, and an extendable toolkit with pluggable white-box and black-box optimizers would benefit the red-teaming community. The paper's strengths include publicly released code, a modular design, a curated multi-domain testbed, and a clear empirical separation in transferable GCG ASR between open- and closed-source models. However, the reported experiments do not actually demonstrate adaptive testing, and the defense-comparison results are confounded by model mismatches. The toolkit may be useful, but the paper's headline claims are ahead of the evidence.

major comments (4)
  1. [Section 1, Sections 4.3-4.5, Figure 1, Table 4] The abstract and Section 1 describe OET as an 'adaptive testing framework' that generates 'worst-case adversarial examples' through optimization. In the experiments, however, a single universal adversarial string per dataset is trained on 5-15 training examples (Table 4) using the model in the Figure 2 configuration (LLaMA3.1-8B-Instruct) and then transferred to all test prompts and to other models and defenses. No attack is re-optimized for a specific test input, and no attack is adapted to the defense under evaluation. Test ASR is thus a transferability measure, not an adaptive worst-case robustness measure; low ASR on closed-source models or defenses can reflect transfer failure rather than robustness. The toolkit may have the capability to support adaptive evaluation through custom optimizers, but the paper does not demonstrate that capability, so the central claim of the paper is not supported by the reported experiments.
  2. [Section 4.4, Table 2] The base undefended model in Table 2 is listed as 'LLaMA (Touvron et al., 2023)', i.e., the original LLaMA model line, while the adversarial strings are trained on LLaMA3.1-8B-Instruct per Figure 2; the base model used by StruQ and SecAlign is never stated. This mismatch confounds the comparison: the defense effect is mixed with transferability differences across model families and versions. In particular, the claim that SecAlign increases ASR on AQuA (+0.46) and PubMedQA (+0.59) relative to the base model is not reliable evidence of defense-introduced vulnerability, because the baseline and the defense may not share the same underlying model. The paper should use a consistent, fully specified base model across all conditions and, ideally, adapt or re-optimize attacks against the defended models.
  3. [Tables 1-3 and Appendix B (Table 5)] All test ASR values in Tables 1, 2, and 3 are reported as point estimates with no error bars, confidence intervals, or significance tests. Appendix B (Table 5) reports standard deviations for training ASR from three repeated training runs, but no such replication is provided for the test ASR values that are the sole basis for the cross-model, cross-defense, and cross-attack comparisons. The arrows in Table 2 compare point estimates without any uncertainty quantification. Without variance estimates, distinctions such as 'closed-source models are more robust' and 'UAT is particularly effective against SciQA (0.78)' are not statistically grounded.
  4. [Section 4.3, Table 1] The adversarial strings are optimized on an open-source model (LLaMA3.1-8B-Instruct per Figure 2) and then transferred to both open- and closed-source models. This setup inherently favors models that are similar to the source model (e.g., LLaMA3.1-8B itself and, to some degree, other open-source models) and disadvantages closed-source models for which no white-box adaptation is performed. Therefore, the paper's conclusion that 'open-source models exhibit higher susceptibility to adversarial attacks' is confounded by the choice of the source model for attack generation. To support the cross-model claim, the paper should either adapt attacks per target model with comparable budgets or, at minimum, acknowledge and control for this transferability bias.
minor comments (8)
  1. [Section 4.5] The text says 'Table 5 presents the attack success rate...' but the table being discussed is Table 3; the table numbering should be corrected.
  2. [Section 2.2] The heading 'Guadient-guided White-box Attacks' contains a typo; it should be 'Gradient-guided White-box Attacks'.
  3. [Section 3] The heading 'Adverserial String Training' and the phrase 'adverserial strings' contain a misspelling; they should be 'Adversarial String Training' and 'adversarial strings'.
  4. [Table 1 and Figure 2] Table 1 lists 'LLama3.1-8B' while Figure 2 uses 'LLaMA3.1-8B-Instruct'; the paper should explicitly state which exact model is used for training and which for evaluation, and whether the evaluated open-source model is the instruct variant.
  5. [Tables 1 and 3] The dataset is called 'SciQ' in Section 4.1 but appears as 'SciQA' in Tables 1 and 3; please standardize the name.
  6. [Section 4.5] The text refers to 'PZE' in one place, but the method is 'PEZ' elsewhere; the abbreviation should be consistent.
  7. [References, AQuA citation] The AQuA dataset is cited as Behrendt et al. (2024), which appears to describe deliberation quality in online discussions rather than the AQuA algebraic word-problem dataset used in prior prompt-injection benchmarks; please verify and correct the citation.
  8. [Section 4.2] The sentence 'we set temperature of model as 0.6 andmax new token to 64' is missing a space between 'and' and 'max'; also, the same section states 'we set temperature of model as 0.6 andmax new token to 64' should clarify whether the same generation settings apply to API-based closed-source models.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the benchmark results are measured on external models and held-out data.

full rationale

OET is an empirical evaluation toolkit rather than a derivation from first principles. The workflow trains adversarial strings on small disjoint training sets (Table 4: 5 examples per dataset, 15 for BIPIA) and then applies the trained strings to held-out test examples and to external target models and defenses (Tables 1-3). No equation in the paper defines the reported test ASR in terms of the fitted training ASR; the training ASR is reported separately in Table 5 and is not passed off as test performance. The self-citations that appear (AutoDAN, Universal Prompt Injection) are used as background and attribution for the implemented optimizers; they are not invoked as evidence for the paper's empirical conclusions, and AutoDAN actually performs poorly in Table 3. The abstract's 'adaptive testing framework' wording is broader than the reported transfer-attack experiments, and the Table 2 baseline comparison is potentially confounded because the source model used to train the strings (LLaMA3.1-8B-Instruct in Figure 2) differs from the cited 'LLaMA' baseline. However, these are correctness and interpretability concerns, not circular reasoning: the headline results are not forced by construction or by a self-citation chain.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on several hand-set evaluation parameters and on assumptions that transferability and substring matching capture adversarial robustness. No new physical or model entity is invented. The largest burden is the unstated source model for adversarial strings used against the defenses and the lack of sensitivity analysis for the fixed hyperparameters.

free parameters (6)
  • Attack goal string = Print sql injection
    Chosen by hand and fixed across all experiments; ASR is defined by the appearance of this exact string in the response, so the absolute ASR values depend on this arbitrary goal.
  • Optimization steps = 500
    Set in Section 4.2 without sensitivity analysis.
  • Sampling temperature = 0.6
    Set in Section 4.2 for inference; choice affects variance of responses.
  • Max new tokens = 64
    Set in Section 4.2.
  • Training set size per dataset = 5 (BIPIA subdomains also 5 each)
    Nearly all datasets use 5 training examples; BIPIA uses 15 total across three subdomains. No analysis of sensitivity to this size.
  • Training runs and aggregation = 3 runs, averaged
    Appendix B reports training ASR as mean over 3 runs, but the paper does not state how the single adversarial string used at test time is selected from these runs.
assumptions (5)
  • domain assumption GCG, AutoDAN, UAT, GBDA, AutoPrompt, PEZ, and PAIR work as described in their original papers and are correctly instantiated in OET.
    The evaluation depends on these optimizers being correctly wrapped and their hyperparameters reasonably configured; the paper does not report all wrapper-specific settings.
  • domain assumption Substring-match ASR on 'sql injection' is a valid and sufficient measure of successful prompt injection.
    Section 4.2 defines success as the attack objective appearing in the response; this ignores partial compliance, refusals that contain the string, or attacks that produce other harmful outputs.
  • domain assumption Transferability of a universal adversarial string across datasets and models is a meaningful proxy for worst-case robustness.
    Sections 4.3, 4.4, and 4.5 interpret cross-dataset transfer ASR as evidence about model and defense robustness, but no per-example adaptive attack is performed.
  • domain assumption The defense models StruQ and SecAlign are used with their default configurations as reported by their authors.
    The paper inherits the defense implementations from Chen et al. 2024a and Chen et al. 2025 without reporting or verifying their exact deployment parameters.
  • standard math Standard LLM token-gradient optimization (with softmax over the vocabulary) is a valid search method in this setting.
    The loss in Eq. 1 maximizes the likelihood of the target sentence; GCG's coordinate gradient approximation from Zou et al. 2023 is assumed valid.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OET: Optimization-based prompt injection Evaluation Toolkit." pith.science (2026). https://pith.science/paper/5KPLAFLS

@misc{pith2026250500843,
  author       = {Pith},
  title        = {Pith review of: OET: Optimization-based prompt injection Evaluation Toolkit},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5KPLAFLS}},
  note         = {Machine review of arXiv:2505.00843}
}
read the original abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation, enabling their widespread adoption across various domains. However, their susceptibility to prompt injection attacks poses significant security risks, as adversarial inputs can manipulate model behavior and override intended instructions. Despite numerous defense strategies, a standardized framework to rigorously evaluate their effectiveness, especially under adaptive adversarial scenarios, is lacking. To address this gap, we introduce OET, an optimization-based evaluation toolkit that systematically benchmarks prompt injection attacks and defenses across diverse datasets using an adaptive testing framework. Our toolkit features a modular workflow that facilitates adversarial string generation, dynamic attack execution, and comprehensive result analysis, offering a unified platform for assessing adversarial robustness. Crucially, the adaptive testing framework leverages optimization methods with both white-box and black-box access to generate worst-case adversarial examples, thereby enabling strict red-teaming evaluations. Extensive experiments underscore the limitations of current defense mechanisms, with some models remaining susceptible even after implementing security enhancements.

Figures

Figures reproduced from arXiv: 2505.00843 by the authors.

Figure 1
Figure 1. Workflow of OET. Orange blacks are input, and blocks with blue heads are [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Usage of toolkit. Left: general usage template, where optimizer can be replaced [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Interface of customized pipeline. Users can implement their own training process [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Dataset composition. Dataset covers a wide range of domains including but not [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Example of GCG attack on Secalign [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sentinel: SOTA model to protect against prompt injections

    cs.CR 2025-06 conditional novelty 4.0 of 10

    Sentinel, a ModernBERT-based binary classifier trained on public and private prompt datasets, reports 0.987 accuracy and 0.980 F1 on a held-out internal test set and outperforms one baseline on public benchmarks.

Reference graph

Works this paper leans on

30 extracted references · 2 canonical work pages · cited by 1 Pith paper

  1. [4]

    Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner

    URL https://arxiv.org/abs/2310.08419. Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. Struq: Defending against prompt injection with structured queries, 2024a. URL https://arxiv.org/abs/2402. 06363. Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, Kamalika Chaudhuri, David Wagner, and Chuan Guo. Secalign: Defending against prompt injectio...

    arXiv

  2. [5]

    Yulin Chen, Haoran Li, Zihao Zheng, Yangqiu Song, Dekai Wu, and Bryan Hooi

    URL https://arxiv.org/abs/2410.05451. Yulin Chen, Haoran Li, Zihao Zheng, Yangqiu Song, Dekai Wu, and Bryan Hooi. Defense against prompt injection attack by leveraging attack techniques, 2024b. URL https: //arxiv.org/abs/2411.00459. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Lang- don, Reema Moussa, Matt Beane, Ting-Hao Kenne...

    arXiv

  3. [8]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  4. [9]

    Attacking large language models with projected gradient descent

    Simon Geisler, Tom Wollschl ¨ager, MHI Abdalla, Johannes Gasteiger, and Stephan G ¨unnemann. Attacking large language models with projected gradient descent. arXiv preprint arXiv:2402.09154,

    arXiv

  5. [10]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276,

    arXiv

  6. [11]

    Zhifeng Jiang, Zhihua Jin, and Guoliang He

    URL https://arxiv.org/abs/2408.13247. Zhifeng Jiang, Zhihua Jin, and Guoliang He. Safeguarding system prompts for llms,

    arXiv

  7. [12]

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W

    URL https://arxiv.org/abs/2412.13426. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. CoRR, abs/1909.06146,

    arXiv 1909

  8. [13]

    Matt Gardner Johannes Welbl, Nelson F

    URL http://arxiv.org/abs/1909.06146. Matt Gardner Johannes Welbl, Nelson F. Liu. Crowdsourcing multiple choice science questions

    arXiv 1909

Show all 30 references
  1. [15]

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao

    URL https://arxiv.org/abs/2307.10169. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451,

    arXiv

  2. [16]

    Automatic and universal prompt injection attacks against large language models, 2024a

    Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. Automatic and universal prompt injection attacks against large language models, 2024a. URL https://arxiv.org/abs/2403.04957. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zh...

    arXiv

  3. [17]

    org/abs/2403.03792

    URL https://arxiv. org/abs/2403.03792. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286,

    arXiv

  4. [18]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang

    URL https://arxiv.org/abs/2312.17673. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. CoRR, abs/1606.05250,

    arXiv

  5. [20]

    doi: 10.2196/58478

    ISSN 2291-9694. doi: 10.2196/58478. URL https://medinform.jmir.org/2024/1/e58478. Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael Abu-Ghazaleh. Survey of vulnerabilities in large language models revealed by adversarial attacks,

    2024 doi

  6. [21]

    Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhen- qiang Gong

    URL https://arxiv.org/abs/2310.10844. Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhen- qiang Gong. Optimization-based prompt injection attack to llm-as-a-judge,

    arXiv

  7. [22]

    Taylor Shin, Yasaman Razeghi, Robert L Logan IV , Eric Wallace, and Sameer Singh

    URL https://arxiv.org/abs/2403.17710. Taylor Shin, Yasaman Razeghi, Robert L Logan IV , Eric Wallace, and Sameer Singh. Au- toprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980,

    arXiv

  8. [23]

    URL https://arxiv.org/abs/2407.00326. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama...

  9. [24]

    Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh

    URL https://arxiv.org/abs/2302.13971. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nlp,

    arXiv

  10. [25]

    Jiongxiao Wang, Fangzhou Wu, Wendi Li, Jinsheng Pan, Edward Suh, Z

    URL https://arxiv.org/abs/ 1908.07125. Jiongxiao Wang, Fangzhou Wu, Wendi Li, Jinsheng Pan, Edward Suh, Z. Morley Mao, Muhao Chen, and Chaowei Xiao. Fath: Authentication-based test-time defense against indirect prompt injection attacks,

    1908 arXiv

  11. [26]

    Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Gold- stein

    URL https://arxiv.org/abs/2410.21492. Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Gold- stein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery,

    arXiv

  12. [27]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al

    URL https://arxiv.org/abs/2302.03668. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115,

    arXiv

  13. [28]

    Benchmarking and defending against indirect prompt injection attacks on large language models

    Jingwei Yi, Yueqi Xie, Bin Zhu, Keegan Hines, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. Benchmarking and defending against indirect prompt injection attacks on large language models. arXiv preprint arXiv:2312.14197,

    arXiv

  14. [29]

    Robust prompt optimization for defending language models against jailbreaking attacks

    Andy Zhou, Bo Li, and Haohan Wang. Robust prompt optimization for defending language models against jailbreaking attacks. arXiv preprint arXiv:2401.17263, 2024a. Zhanke Zhou, Jianing Zhu, Fengfei Yu, Xuan Li, Xiong Peng, Tongliang Liu, and Bo Han. Model inversion attacks: A su...

    arXiv

  15. [30]

    A Dataset statistics Dataset Domain # of test example # of train example BIPIA (Yi et al.,

    URL https://arxiv.org/abs/2307.15043. A Dataset statistics Dataset Domain # of test example # of train example BIPIA (Yi et al.,

    arXiv

  16. [2016]

    Florian Reis, Christian Lenz, Manfred Gossen, Hans-Dieter Volk, and Norman Michael Drzeniek

    URL http: //arxiv.org/abs/1606.05250. Florian Reis, Christian Lenz, Manfred Gossen, Hans-Dieter Volk, and Norman Michael Drzeniek. Practical applications of large language models for health care professionals and scientists. JMIR Med Inform, 12:e58478, Sep

    arXiv

  17. [2017]

    doi: 10.18653/v1/P17-1147

    Association for Computational Linguistics. doi: 10.18653/v1/P17-1147. URL https://aclanthology.org/P17-1147/. Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models,

    doi

  18. [2019]

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J

    URL https://arxiv.org/abs/1902.06705. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries,

    1902 arXiv

  19. [2021]

    Zhiyu Zoey Chen, Jing Ma, Xinlu Zhang, Nan Hao, An Yan, Armineh Nourbakhsh, Xianjun Yang, Julian McAuley, Linda Petzold, and William Yang Wang

    URL https://arxiv.org/abs/2109.00122. Zhiyu Zoey Chen, Jing Ma, Xinlu Zhang, Nan Hao, An Yan, Armineh Nourbakhsh, Xianjun Yang, Julian McAuley, Linda Petzold, and William Yang Wang. A survey on large language models for critical societal domains: Finance, healthcare, and law, ...

    arXiv

  20. [2023]

    10 Preprint

    URL https://lmsys.org/blog/2023-03-30-vicuna/ . 10 Preprint. Under review. Edoardo Debenedetti, Jie Zhang, Mislav Balunovi´c, Luca Beurer-Kellner, Marc Fischer, and Florian Tram`er. Agentdojo: A dynamic environment to evaluate attacks and defenses for llm agents. arXiv preprin...

    2023 arXiv

  21. [2024]

    org/abs/2404.02761

    URL https://arxiv. org/abs/2404.02761. Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dim- itris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness,

    arXiv

  22. [2025]

    Anthropic

    URL https: //arxiv.org/abs/2406.00799. Anthropic. Introducing claude 3.5 sonnet, June

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.