{"id":"52667c54-913b-41e3-9243-b9a5471ac83d","arxiv_id":"2505.00843","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"OET is an optimization-based evaluation toolkit that benchmarks prompt injection attacks and defenses across eight datasets and shows current defenses remain vulnerable in several domains.","lead":"Researchers built OET, a modular toolkit that tests how well large language models resist prompt injection attacks by generating optimized adversarial strings and attacking models across many domains. The evaluation finds open-source models are far more vulnerable and that existing defenses, StruQ and SecAlign, fail inconsistently across domains.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'adaptive testing' claim is not supported by the reported experiments: all ASR numbers come from a single universal adversarial string per dataset trained on 5–15 examples and transferred, so the benchmark does not measure worst-case adaptive robustness.","rationale":"The central contribution is the adaptive evaluation toolkit, and the experiments are the primary evidence that this contribution works as claimed. The weakest link is not the code itself but the mapping from experiment to claim: a universal trigger trained on 5–15 examples and transferred across models is not an adaptive, worst-case attack. The defense evaluation adds a second confound by comparing against a different base model than the one used for attack generation, making the SecAlign vulnerability finding fragile. If the concrete check shows the vulnerabilities vanish under a matched transfer protocol, the central empirical claims cannot stand; if they persist, the toolkit's value is supported. This is addressable by rerunning with matched base models and by either implementing true adaptive per-instance/per-defense attacks or softening the 'adaptive' and 'worst-case' wording. The reader's weakest assumption identified the same transfer-proxy issue, and my analysis agrees with that assessment. I therefore recommend CONDITIONAL acceptance rather than REJECT, because the toolkit and code may still be valid once the experimental protocol is aligned with the claims.","tokens_in":12111,"tokens_out":4796,"duration_ms":53297,"concrete_test":"Run the Section 4.4 experiment with a matched protocol: set target_model in the OET config to the exact base checkpoint used by SecAlign (and StruQ) as stated in their public releases, optimize a fresh GCG string per dataset on that checkpoint, and evaluate SecAlign and the undefended base under identical decoding settings (temperature 0.6, max_new_tokens 64). If SecAlign's ASR on AQuA and PubMedQA is no longer above the corrected base model's ASR, the paper's key defense-vulnerability claim is unsupported. As a second check, run a per-test-instance adaptive variant (e.g., optimize the adversarial string on each test prompt) to see whether the current transferable ASR understates worst-case vulnerability.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The abstract and Section 1 claim OET provides an 'adaptive testing framework' that generates 'worst-case adversarial examples' for strict red-teaming. Yet Sections 4.3–4.5 report only transferable attacks: for each dataset, an adversarial string is trained on the small training sets in Table 4 (5 examples, 15 for BIPIA) using the target model in the Figure 2 config (LLaMA3.1-8B-Instruct) and then appended to all test prompts. No attack is re-optimized for a specific test input or adapted to the defense under evaluation. Appendix B confirms that only training-set ASR is optimized; test ASR is a transfer number. Low ASR on closed-source models or defenses could therefore reflect transfer failure rather than robustness, while high ASR is not a worst-case bound. The defense comparison in Table 2 is additionally confounded: the base undefended model is listed as 'LLaMA (Touvron et al., 2023)', not the LLaMA3.1-8B-Instruct model used to train the strings, and the base model for StruQ and SecAlign is never stated. The key empirical finding—that SecAlign increases ASR on AQuA and PubMedQA—compares a string tuned on a different model family to a mismatched baseline. As written, the experiments do not establish the headline claim that the toolkit exposes worst-case adaptive vulnerabilities, even though the toolkit itself may be capable of supporting such evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OET, a modular and extensible toolkit for benchmarking prompt injection attacks and defenses using optimization-based adversarial string generation. The toolkit provides data conversion, adversarial string training, inference, and result-checking stages, with support for user-defined optimizers and metrics. The authors curate multi-domain QA datasets and report experiments with GCG and other attacks against open- and closed-source LLMs as well as two defenses, StruQ and SecAlign. The central claims are that OET provides an 'adaptive testing framework' that generates 'worst-case adversarial examples' and that state-of-the-art defenses remain inconsistently vulnerable, with SecAlign even increasing ASR on AQuA and PubMedQA relative to the undefended baseline.","tokens_in":12436,"tokens_out":6601,"duration_ms":61049,"significance":"If the adaptive worst-case evaluation claim were substantiated, OET would fill a genuine gap: existing prompt injection benchmarks are largely static, and an extendable toolkit with pluggable white-box and black-box optimizers would benefit the red-teaming community. The paper's strengths include publicly released code, a modular design, a curated multi-domain testbed, and a clear empirical separation in transferable GCG ASR between open- and closed-source models. However, the reported experiments do not actually demonstrate adaptive testing, and the defense-comparison results are confounded by model mismatches. The toolkit may be useful, but the paper's headline claims are ahead of the evidence.","major_comments":[{"comment":"The abstract and Section 1 describe OET as an 'adaptive testing framework' that generates 'worst-case adversarial examples' through optimization. In the experiments, however, a single universal adversarial string per dataset is trained on 5-15 training examples (Table 4) using the model in the Figure 2 configuration (LLaMA3.1-8B-Instruct) and then transferred to all test prompts and to other models and defenses. No attack is re-optimized for a specific test input, and no attack is adapted to the defense under evaluation. Test ASR is thus a transferability measure, not an adaptive worst-case robustness measure; low ASR on closed-source models or defenses can reflect transfer failure rather than robustness. The toolkit may have the capability to support adaptive evaluation through custom optimizers, but the paper does not demonstrate that capability, so the central claim of the paper is not supported by the reported experiments.","section":"Section 1, Sections 4.3-4.5, Figure 1, Table 4"},{"comment":"The base undefended model in Table 2 is listed as 'LLaMA (Touvron et al., 2023)', i.e., the original LLaMA model line, while the adversarial strings are trained on LLaMA3.1-8B-Instruct per Figure 2; the base model used by StruQ and SecAlign is never stated. This mismatch confounds the comparison: the defense effect is mixed with transferability differences across model families and versions. In particular, the claim that SecAlign increases ASR on AQuA (+0.46) and PubMedQA (+0.59) relative to the base model is not reliable evidence of defense-introduced vulnerability, because the baseline and the defense may not share the same underlying model. The paper should use a consistent, fully specified base model across all conditions and, ideally, adapt or re-optimize attacks against the defended models.","section":"Section 4.4, Table 2"},{"comment":"All test ASR values in Tables 1, 2, and 3 are reported as point estimates with no error bars, confidence intervals, or significance tests. Appendix B (Table 5) reports standard deviations for training ASR from three repeated training runs, but no such replication is provided for the test ASR values that are the sole basis for the cross-model, cross-defense, and cross-attack comparisons. The arrows in Table 2 compare point estimates without any uncertainty quantification. Without variance estimates, distinctions such as 'closed-source models are more robust' and 'UAT is particularly effective against SciQA (0.78)' are not statistically grounded.","section":"Tables 1-3 and Appendix B (Table 5)"},{"comment":"The adversarial strings are optimized on an open-source model (LLaMA3.1-8B-Instruct per Figure 2) and then transferred to both open- and closed-source models. This setup inherently favors models that are similar to the source model (e.g., LLaMA3.1-8B itself and, to some degree, other open-source models) and disadvantages closed-source models for which no white-box adaptation is performed. Therefore, the paper's conclusion that 'open-source models exhibit higher susceptibility to adversarial attacks' is confounded by the choice of the source model for attack generation. To support the cross-model claim, the paper should either adapt attacks per target model with comparable budgets or, at minimum, acknowledge and control for this transferability bias.","section":"Section 4.3, Table 1"}],"minor_comments":[{"comment":"The text says 'Table 5 presents the attack success rate...' but the table being discussed is Table 3; the table numbering should be corrected.","section":"Section 4.5"},{"comment":"The heading 'Guadient-guided White-box Attacks' contains a typo; it should be 'Gradient-guided White-box Attacks'.","section":"Section 2.2"},{"comment":"The heading 'Adverserial String Training' and the phrase 'adverserial strings' contain a misspelling; they should be 'Adversarial String Training' and 'adversarial strings'.","section":"Section 3"},{"comment":"Table 1 lists 'LLama3.1-8B' while Figure 2 uses 'LLaMA3.1-8B-Instruct'; the paper should explicitly state which exact model is used for training and which for evaluation, and whether the evaluated open-source model is the instruct variant.","section":"Table 1 and Figure 2"},{"comment":"The dataset is called 'SciQ' in Section 4.1 but appears as 'SciQA' in Tables 1 and 3; please standardize the name.","section":"Tables 1 and 3"},{"comment":"The text refers to 'PZE' in one place, but the method is 'PEZ' elsewhere; the abbreviation should be consistent.","section":"Section 4.5"},{"comment":"The AQuA dataset is cited as Behrendt et al. (2024), which appears to describe deliberation quality in online discussions rather than the AQuA algebraic word-problem dataset used in prior prompt-injection benchmarks; please verify and correct the citation.","section":"References, AQuA citation"},{"comment":"The sentence 'we set temperature of model as 0.6 andmax new token to 64' is missing a space between 'and' and 'max'; also, the same section states 'we set temperature of model as 0.6 andmax new token to 64' should clarify whether the same generation settings apply to API-based closed-source models.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is under review and the term 'adaptive testing' is used much more strongly than the experiments support. The toolkit is a genuine contribution, but the empirical evaluation needs substantial strengthening (adaptive attack runs, consistent base models, uncertainty estimates) before the headline claims are credible. The authors should also revisit whether the title and abstract overstate the scope of what is demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — here's my take on OET. The toolkit is the real contribution; the experiments are the weak part.\n\nWhat's new: a modular, extensible evaluation harness with public code, a curated eight-dataset QA benchmark spanning law, finance, science, and more, and a systematic comparison of seven optimization-based attack methods against StruQ and SecAlign. That is genuinely useful; the community lacks a common harness. The workflow (data conversion, adversarial string training, inference, result checking) is clean, and the custom optimizer/metric interface is a good design. The paper also adds a new data point: transferable GCG strings trained on a few examples achieve high ASR on open models and low ASR on closed models, while defenses behave unevenly across domains.\n\nNow the soft spots, in order of severity. The 'adaptive testing framework' claim is not what is reported. Sections 4.3–4.5 use one universal adversarial string per dataset, trained on 5–15 training examples (Table 4), then appended to every test prompt. Nothing is re-optimized for the test input or for the defense under evaluation. That is a transfer-attack benchmark, not an adaptive or worst-case evaluation. The paper's own intro describes 'a transferability setting,' so the authors know, but the abstract and Section 1 oversell it. Second, Table 2 compares defenses against a 'Base Undefended Model: LLaMA (Touvron et al., 2023)' — i.e., LLaMA-1 — while the adversarial strings were trained on LLaMA3.1-8B-Instruct per Figure 2. That mismatch confounds the striking SecAlign/StruQ comparisons, including the claim that SecAlign increases ASR on AQuA and PubMedQA. Third, test ASR has no error bars; the appendix shows training ASR variance can be large (e.g., AutoDAN on CaseHold: 0.33±0.34), so the stability of the transfer numbers is unknown. Minor but telling: Table 3's caption says 'Table 5 presents...' and there's a PZE/PEZ typo. Easy fixes.\n\nOverall, the toolkit deserves engagement and likely adoption. But the authors should either re-run with per-defense adaptation to earn the 'adaptive' label, or rewrite the claims to describe a transfer-based benchmark. As written, the central 'worst-case adaptive robustness' conclusion is not established.\n\nI would bring this to reading group as a useful case study in how benchmark claims can outpace the experiments. I'd cite it for the toolkit, not for the defense conclusions. It deserves serious review, but the referees should push for heavy revision.","headline":"A genuinely useful prompt-injection evaluation toolkit with public code, but the 'adaptive' and 'worst-case' claims in the abstract and introduction are not supported by the transfer-only experiments.","tokens_in":12945,"tokens_out":2280,"would_cite":true,"duration_ms":23090,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces OET, an optimization-based evaluation toolkit for prompt injection, and shows that transferable adversarial strings still defeat current defenses, with SecAlign even increasing attack success on some datasets.","keywords":["prompt injection","adversarial robustness","LLM security","evaluation toolkit","optimization-based attacks","transferable adversarial strings","adaptive testing","attack success rate"],"falsifier":"Re-run the same 500-step gradient-guided optimization directly against SecAlign and StruQ on the exact base model each defense was built on, then compare attack success to the undefended base model. If SecAlign's ASR no longer exceeds the base model on AQuA and PubMedQA, the paper's demonstration that defenses can increase vulnerability would not survive.","tokens_in":11911,"feed_emoji":"🛡️","tokens_out":8658,"duration_ms":80030,"temperature":0.7,"pith_summary":"OET is a modular evaluation toolkit that converts question-answering data into a uniform prompt format, trains a short adversarial string by optimization toward a target sentence, and then injects that string into test prompts to measure attack success. The paper argues that existing prompt-injection benchmarks are static and cannot capture adaptive, worst-case behavior, and that an optimization-driven workflow fills that gap. In the reported experiments, transferable strings trained by gradient-guided optimization succeed far more often on open-source models than on closed-source models, and two published defenses, StruQ and SecAlign, remain only partially effective. The sharpest result is that SecAlign raises attack success on AQuA and PubMedQA relative to the undefended LLaMA baseline, indicating that a defense can make a model more attackable on some domains. Taken on its own terms, the paper establishes a standardized way to make uneven defense behavior measurable.","feed_headline":"Optimized prompt-injection tests show defenses can backfire","feed_subtitle":"A transferable-string benchmark finds SecAlign makes models more vulnerable on AQuA and PubMedQA.","key_machinery":"The load-bearing mechanism is the trained adversarial string: a short sequence of tokens inserted between the instruction and the input, optimized so that the model outputs a chosen target sentence, here 'sql injection'. The optimization is carried out with gradient-guided discrete search (GCG), with the attack goal and optimizer configuration as inputs and the trained string as output. The workflow has four stages: data conversion into a unified QA format, adversarial string training on 5 to 15 examples per dataset, inference where the trained string and attack goal are injected into test prompts, and result checking where a custom or default metric, attack success rate, measures whether the target sentence appears in the response. The modular interface allows a user to replace the optimizer or the metric, which is what makes the toolkit a benchmark rather than a one-off experiment.","core_discovery":"The paper's central claim is that OET, by training adversarial strings with optimization techniques and transferring them across models and datasets, provides a worst-case-style evaluation of prompt-injection defenses that static benchmarks cannot. On its own results, the claim takes concrete form: LLaMA3.1-8B, Vicuna-7B, and Qwen2-7B-Instruct are attacked successfully most of the time, while GPT-4o-mini and Claude-3.5-sonnet are much harder to attack with the same transferred strings. Against defenses, StruQ drives attack success to zero on six of eight datasets but leaves high success on FinQA and TriviaQA, while SecAlign lowers success on most datasets yet increases it on AQuA and PubMedQA relative to the undefended base model. The authors read these numbers as evidence that current defenses do not generalize across domains and that some defense mechanisms can actively open new vulnerabilities. The discovery, stated on the paper's own terms, is that optimization-based transferable attack strings make such uneven defense behavior measurable in a standardized way.","pith_inferences":["A natural next experiment is to optimize the adversarial string directly against each defense model rather than transferring it, which would separate how much of the reported vulnerability is intrinsic to the defense and how much is an artifact of transfer from a different base model.","The small training sets (5-15 examples) suggest that the benchmark could be extended to measure how attack success changes with training-set size and domain diversity, giving a data-efficiency curve for red-teaming.","If this evaluation style is adopted, one could imagine pairing it with per-domain failure analysis to issue robustness reports that identify precisely which task types cause a defense to backfire.","The rise in ASR on SecAlign relative to its base model implies that defenses should be monitored for negative side effects, not just average improvement; the same benchmark could be used as a regression test during defense training."],"forward_implications":["If a defense is evaluated only on static test prompts, its reported security could be an artifact; OET-style transferred strings give a stricter, if still transfer-based, lower bound on worst-case behavior.","Open-source deployments in security-sensitive settings should assume that similar optimization-based strings will succeed on a large fraction of queries unless specific defenses are shown to work under this benchmark.","Defense claims should report per-domain attack success relative to the undefended base model, because SecAlign's increased ASR on AQuA and PubMedQA shows that aggregate or cross-domain improvements can hide localized harm.","Because the toolkit lets new optimizers and custom metrics be plugged in, the benchmark can track both attack and defense evolution without waiting for new static datasets.","The two closed-source models' low ASR under transferred strings does not establish their worst-case security, but under this test they are clearly harder to attack than the open-source models."],"supporting_citations":[{"why":"Supplies GCG, the gradient-guided discrete optimizer used to train the adversarial strings in the main attack and defense tables.","marker":"(Zou et al., 2023)"},{"why":"StruQ, one of the two defense models benchmarked; its near-zero ASR on most datasets but failures on FinQA and TriviaQA carry the generality claim.","marker":"(Chen et al., 2024a)"},{"why":"SecAlign, the other defense model; its increased ASR on AQuA and PubMedQA relative to the base model is the paper's sharpest evidence that defenses can backfire.","marker":"(Chen et al., 2025)"},{"why":"Introduces universal prompt injection via optimization, the line of attack OET generalizes into a modular toolkit.","marker":"(Liu et al., 2024a)"},{"why":"Existing agentic benchmark that the paper contrasts as static, establishing the gap OET claims to fill.","marker":"(Debenedetti et al., 2024)"},{"why":"Existing LLM security probing framework limited to fixed attacks and models, used to justify the need for user-extensible optimization-based testing.","marker":"(Derczynski et al., 2024)"},{"why":"Argument that adaptive, worst-case evaluation is necessary for adversarial robustness, which the paper cites to motivate its dynamic testing design.","marker":"(Carlini et al., 2019)"},{"why":"Provides LLaMA, the undefended base model against which StruQ and SecAlign results in Table 2 are compared.","marker":"(Touvron et al., 2023)"}],"fun_headline_variants":["Optimization-based attacks reveal defenses can backfire","Transferable attack strings expose defense backfire","Adaptive testing shows defenses can backfire","SecAlign increases vulnerability on some datasets","Optimized prompt-injection benchmarks expose defense flaws"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusions assume that an attack string trained on a few examples and then copied onto other models and data measures how a determined attacker would exploit a defense; if the attacker tuned the string against the actual defense, the reported numbers could change.","fun_headline_variants_meta":{"raw":{"variants":["Optimization-based attacks reveal defenses can backfire","Transferable attack strings expose defense backfire","Adaptive testing shows defenses can backfire","SecAlign increases vulnerability on some datasets","Optimized prompt-injection benchmarks expose defense flaws"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00106,"raw_usage":{"total_tokens":4431,"prompt_tokens":916,"completion_tokens":3515,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":3448}},"tokens_in":532,"tokens_out":3515,"duration_ms":26408,"temperature":1.0,"reasoning_tokens":3448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:33:11.009578+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same 500-step gradient-guided optimization directly against SecAlign and StruQ on the exact base model each defense was built on, then compare attack success to the undefended base model. If SecAlign's ASR no longer exceeds the base model on AQuA and PubMedQA, the paper's demonstration that defenses can increase vulnerability would not survive.","supporting_citations":[],"review_version":1}