{"id":"34bebddb-f728-4d64-89c1-df8c3a392ebe","arxiv_id":"2506.09956","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A competition with 839 participants yielded 208,095 unique indirect prompt injection attacks against an LLM email assistant, released as a benchmark dataset with analysis of defense effectiveness.","lead":"This paper presents LLMail-Inject, a public challenge dataset of 208,095 unique prompt injection attacks against an LLM email assistant protected by multiple defenses. It offers new evidence on how adaptive attackers bypass defenses and which defenses hold up best.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-team submission flooding and near-duplicate LLM-generated template variants may inflate the 208,095 'unique attack' count and bias the submission-level defense metrics; the paper does not report per-team or semantic diversity checks.","rationale":"I focused on dataset composition rather than the broader 'realism' concern because the paper's primary deliverable is the dataset, and the released files make a composition check directly possible. The paper deserves credit for releasing the full submissions, code, and annotations; the multi-stage outcome diagrams add useful transparency, and the authors acknowledge several limitations. However, the abstract's '208,095 unique attack submissions' is the headline quantitative claim, and 'unique' is defined by exact string equality (Appendix A.1). The competition scoring (Appendix G) rewards solving many sub-levels and does not penalize submission volume, and Appendix I.2 describes LLM-based bulk generation of variants. Under those incentives, the unique-prompt count and per-submission rates in Section 4 can be driven by a small number of high-volume teams. TSR was introduced precisely to correct for this in difficulty comparisons, but it is not applied to the detection-recall and ASR results. A per-team and semantic-diversity audit is therefore the minimal check that would validate the benchmark's diversity claim. This does not disprove the paper's conclusions; it determines how much weight they should carry. The reader's weakest assumption (environment proxy) is related but distinct, so I partially agree with it. The appropriate outcome remains conditional acceptance, with the diversity audit added to the revision requirements.","tokens_in":26312,"tokens_out":7194,"duration_ms":88194,"concrete_test":"Using the released dataset, compute: (1) per-team counts of unique prompts and the top-5 teams' share of the 208,095; (2) cluster Phase 1 unique prompts by embedding similarity or template signature (e.g., after removing subject/body boilerplate) to estimate semantic clusters; (3) recompute Figure 2 and Table 3 on a per-team capped sample (e.g., at most 100 submissions per team per sub-level) and on cluster representatives. If the top-5 share exceeds roughly 40% or the capped/clustered rates change the relative ordering of Prompt Shield, TaskTracker, and LLM Judge, the 'large-scale diverse attack dataset' and defense-effectiveness conclusions need explicit qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central deliverable is a large, diverse benchmark of adaptive indirect prompt injections. Two properties are load-bearing: (i) the 208,095 'unique' prompts are unique only as raw strings (Appendix A.1), and (ii) the submission-level analyses in Section 4 (Figures 2-3, Tables 3-4) weight every submission equally. The paper itself reports in Section 3 and Appendix I.2 that a winning team 'used an LLM to automatically generate variants' of a working template, producing one successful bypass every 10-15 variants. If one or a few teams submitted thousands of LLM-generated variants, those variants can dominate both the unique-prompt count and the per-submission success/detection rates, making the dataset appear more diverse and shifting the measured defense ordering. The paper does not report per-team unique-prompt shares, semantic deduplication, or any cap/flooding analysis. TSR (Section 4.3) addresses team-level difficulty but is restricted to 62 teams and is not applied to the headline recall and ASR figures. This is not a claim of bad faith; it is an unverified distributional property of the released data that directly affects the benchmark and defense-effectiveness claims.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents LLMail-Inject, a public challenge and dataset for indirect prompt injection attacks against LLM-based email assistants. Participants adaptively attacked a simulated assistant with retrieval, multiple defenses (Spotlighting, Prompt Shield, LLM Judge, TaskTracker), and two LLMs, across four retrieval levels and two phases. The authors report 208,095 unique attack submissions from 839 participants, assess defense effectiveness via submission-level success rates and team-level success rates, analyze the difficulty of end-to-end attacks, and release the challenge code, dataset, and analysis scripts.","tokens_in":26579,"tokens_out":2477,"duration_ms":23427,"significance":"If the dataset and analyses are sound, this is a valuable community resource: it is the largest public collection of adaptive, indirect prompt injection attacks against multiple defenses in an end-to-end retrieval setting, and it provides concrete, real-competition evidence about defense ordering and attack strategies. The paper is transparent about several limitations (LLM annotator labels, restricted attack objectives, Phi-3 non-determinism). The release of the dataset, challenge code, and analysis scripts is a strength, as is the use of a team-level metric (TSR) that is less sensitive to submission flooding than raw submission counts.","major_comments":[{"comment":"The headline '208,095 unique prompts' and the submission-level analyses in Figures 2-3 and Tables 3-4 may be substantially affected by per-team submission flooding, especially by LLM-generated template variants. Appendix I.2 states that a winning team 'used an LLM to automatically generate variants' of a working template, producing one successful bypass every 10-15 variants. Since submissions are not penalized and 'unique' is defined only by raw string identity (Appendix A.1), a small number of teams may dominate both the unique-prompt count and the per-submission success/detection rates. The paper does not report per-team submission distributions, semantic deduplication rates, or an analysis with per-team caps. Please add a robustness analysis: report the number of unique prompts per team, the effect of capping submissions per team on Figures 2 and Tables 3-4, and an estimate of the fraction of submissions that are near-duplicates of a common template.","section":"Sections 3 and 4; Appendix A.1 and I.2"},{"comment":"The recall analysis in Tables 3-4 is computed only over submissions that called the send_email tool (25,323 in Phase 1). This subset is not representative of all attack attempts: it excludes attacks that were detected before tool invocation and attacks that were retrieved but never invoked the tool. As a result, the reported recall values do not measure end-to-end defense effectiveness and may not generalize to the full distribution of submitted attacks. Please either compute recall over all submissions labeled as attacks by the LLM annotator (with appropriate caveats about annotator quality) or explicitly state that the reported recall is conditional on tool invocation and cannot be compared directly to the attack success rate figures.","section":"Section 4.5, Tables 3 and 4"},{"comment":"The claim that the challenge 'closely approximates real-world attacks' is not fully supported. The environment has one fixed user query per level, a fixed retrieval configuration, no interactive user behavior, a small synthetic email database, and a single fixed exfiltration objective. This is a reasonable controlled benchmark, but real deployments involve diverse user requests, evolving mailboxes, and varied attacker goals. Please soften this claim or provide explicit evidence for ecological validity, such as a comparison with real-world indirect prompt injection incidents, or a discussion of which aspects are realistic and which are simplified.","section":"Section 7, Discussion"}],"minor_comments":[{"comment":"The abstract and Section 3 state '208,095 unique prompts' without defining that 'unique' means raw-string uniqueness; Appendix A.1 later clarifies this. Consider using 'distinct raw prompts' or defining the term at first use to avoid overstating semantic diversity.","section":"Abstract and Section 3"},{"comment":"The x-axis of Figures 2a and 2c is labeled 'level1' through 'level4' while the text refers to 'retrieval level' and 'LLM'. The captions would be clearer if they stated exactly which sub-levels are averaged in each panel.","section":"Section 4.1, Figure 2"},{"comment":"Table 1 reports TSR for 'level 4 GPT-4 sub-levels' but the text does not state the number of teams or the number of sub-levels included. Please add the sample size to the table or caption.","section":"Section 4.3, Table 1"},{"comment":"The LLM-annotator labels are a central part of the dataset, yet the paper reports no manual validation or agreement statistics for the annotator. A small manual validation study would strengthen confidence in the labels, given that the annotation prompt explicitly aims to catch social-engineering attacks.","section":"Section A.1, Annotations (phase 1)"},{"comment":"The claim that LLMail-Inject is 'the first to explore a realistic setting in which the LLM has the ability to call tools' is stated without comparing to concurrent tool-based evaluation frameworks such as AgentDojo, which is cited but not discussed in this context. Please clarify the distinction.","section":"Section 5, Related Work"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe dataset is the real contribution here: 208k unique submissions, 839 participants, multiple defenses, a tool-calling email assistant, and an adaptive adversarial setting. That combination is new; prior competitions focused on direct injection or lacked tool calls. If you work on prompt injection defenses, this is a resource you will want.\n\nThe paper does a few things well. The challenge design is demanding, with retrieval, an undisclosed tool name, and exfiltration in level 4. The release includes code, the full dataset, and annotations. The authors are transparent about limitations: LLM annotator, narrow objectives, and Phi-3 non-determinism. They also introduce the Team Success Rate metric, which is a thoughtful way to compare defense difficulty without being distorted by teams that submit many attempts.\n\nNow the soft spots, in proportion. The 208,095 'unique' prompts are unique as strings, not as attack strategies. The paper itself reports a winning team used an LLM to auto-generate variants of a working template, with one success per 10-15 variants. That means a handful of teams can flood the dataset with near-duplicates. The submission-level analyses in Figures 2-3 and Tables 3-4 weight each submission equally, so those variants can bias the measured defense recall and attack success rates. The paper does not report per-team submission shares or any semantic deduplication, so we cannot tell how much this matters. TSR partially addresses this, but only for 62 teams and not for the headline recall rates. This is an analysis gap, not evidence of misconduct.\n\nThe defense-effectiveness numbers should be read as 'against these participants, given these challenge incentives', not as population estimates. The claim in Section 7 that the setup 'closely approximates real-world attacks' is overstated; it is a realistic proxy, but transferability is untested. Confidence intervals on recall rates would help.\n\nWho will get value: anyone building or evaluating prompt injection defenses, especially in email or agentic settings. Treat the distribution as submission-weighted, not adversary-weighted. The dataset alone deserves a serious referee. I would recommend acceptance with a request to add per-team and semantic diversity analysis, and to soften the generalizability claim. I'd cite it in my own work on LLM security, and I'd bring it to a reading group for the dataset, not for the analysis.\n\nRecommendation: send it to peer review.","headline":"A substantial, useful dataset of adaptive indirect prompt injections, but the submission-level metrics need per-team and semantic diversity checks before trusting the headline numbers.","tokens_in":27180,"tokens_out":2855,"would_cite":true,"duration_ms":30919,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A public challenge in which 839 participants wrote 208,095 unique attack emails against an LLM email assistant is released as the first large-scale benchmark of adaptive indirect prompt injection.","keywords":["indirect prompt injection","adaptive attacks","LLM email assistant","defense evaluation","attack dataset","benchmark","instruction-data separation","tool calling security"],"falsifier":"Run the same four defenses under the same adaptive red-team conditions on a production email-assistant workload with real mailboxes, real user queries, and attackers who do not know the defenses: if the relative difficulty ordering (LLM Judge hardest, then TaskTracker, then Spotlighting, then Prompt Shield) or the under-1% end-to-end success rates do not approximately reproduce, the challenge's proxy assumption is falsified.","tokens_in":26164,"feed_emoji":"🛡️","tokens_out":12765,"duration_ms":118285,"temperature":0.7,"pith_summary":"The paper reports the results of LLMail-Inject, a public challenge in which 839 participants tried to smuggle malicious instructions into emails that an LLM-based email assistant would retrieve and act upon. The paper's claim is that the resulting dataset — 208,095 unique attack submissions collected across four retrieval scenarios, several defenses, and two LLM architectures — is a large-scale record of adaptive indirect prompt injection, and that analyzing it reveals which defenses slow skilled attackers down and why. A sympathetic reader should care because systematic measurement of adaptive adversaries is exactly what is missing from the prompt-injection literature, where defenses are usually tested against fixed, non-adaptive attack sets. The concrete findings include a consistent difficulty ordering of defenses, evidence that GPT-4o-mini's instruction-hierarchy training makes it materially harder to attack than Phi-3, and the observation that many successful attacks look like ordinary emails rather than explicit commands.","feed_headline":"208,095 live attack prompts rank LLM email defenses","feed_subtitle":"Open challenge data show adaptive attackers struggle most against LLM judges and least against text classifiers.","key_machinery":"The carrying mechanism is the challenge pipeline itself: a fixed user query, a simulated mailbox into which the attacker's email is inserted, a retriever that selects and ranks emails, a defense layer, and an LLM with tool calling, where success is defined by a fully specified tool call with the correct destination and content. The defense layer alternates among Spotlighting (data wrapped in secret delimiters and randomized word separators), Prompt Shield (a proprietary black-box text classifier), LLM Judge (an LLM-as-judge guardrail prompted to flag injections in email context), TaskTracker (a linear probe over activation deltas between query processing and data processing), and a combination-of-all condition, evaluated on two LLMs: Phi-3-medium-128k-instruct and GPT-4o-mini. Each submission was recorded with fine-grained outcome flags (retrieved, undetected, exfil.sent, exfil.destination, exfil.content), which is what turns raw attack prompts into a measurable multi-stage dataset; the Phase 2 Conformal Blocklist, which blocks known attack sentences and their paraphrases with per-sentence embedding-space thresholds, is a secondary mechanism showing how collected attack data can be recycled into defense.","core_discovery":"The paper's central claim is that a realistically staged, competition-grade environment can capture how indirect prompt injection behaves under adaptive pressure, and that the captured attacks constitute a benchmark that static test sets cannot provide. Participants had to clear a complete pipeline for a submission to count: the email had to be retrieved from a simulated mailbox, evade the active defense, cause the assistant to call a tool with an undisclosed name, and supply exactly the right destination and content arguments — with the hardest level also requiring exfiltration of a figure from a separate email. From 461,640 total submissions the paper derives a consistent difficulty ordering: adaptive teams succeeded least often against the LLM Judge, then TaskTracker, then Spotlighting, then Prompt Shield (level-4 GPT-4 team success rates 0.318, 0.440, 0.529, 0.600), stacking all defenses dropped this to 0.166, and the end-to-end success rates were 0.8% (Phase 1) and 0.3% (Phase 2). The paper concludes that end-to-end evaluation is necessary because single-stage 'did it follow the instruction' tests miss most of the difficulty, that successful attacks are often declarative or contextually plausible rather than explicit commands, and that the dataset — with 25,323 tool-triggering and 104,583 annotator-labeled injections in Phase 1 — is a resource for studying instruction-data separation.","pith_inferences":["The timestamped raw submissions could be replayed to separate two sources of a defense's strength: how much comes from genuinely raising the cost of each attempt, versus from forcing attackers to learn per-defense quirks that would not generalize.","A natural extension the authors do not pursue is to train a detector on the 104,583 Phase 1 annotator-labeled injections and evaluate it on Phase 2 as an adaptive holdout; the paper's own warning against directly training on this dataset would need to be stress-tested.","If the measured defense ordering transfers to other tool-using agent surfaces such as code execution or browser agents — a transfer I am extrapolating, not one the paper claims — then LLM-judge-style contextual guardrails and internal-state monitors deserve deployment priority over text classifiers.","The Conformal Blocklist's validity guarantee holds only against the paraphrase engine used to build it, which raises a testable arms-race question: whether fresh, non-paraphrased payloads like those Phase 2 winners crafted erode the blocklist's advantage at a measurable rate."],"forward_implications":["Under adaptive attackers, stacking heterogeneous defenses is the practical recipe: combining all defenses cut team success rate on the hardest GPT-4 sub-level to 0.166, far below any single defense.","Models with instruction-hierarchy training are materially harder to inject: team success rates were consistently lower on GPT-4o-mini than on Phi-3 across defenses and retrieval levels.","End-to-end attacks were rare — 0.8% of Phase 1 submissions and 0.3% of Phase 2 — because the email had to be retrieved, evade detection, and match exact tool arguments, so single-stage benchmarks that only check whether the model follows the injected instruction miss most of the real difficulty.","Successful attacks are often declarative and contextually plausible rather than explicit instructions, which supports the paper's hypothesis that text-based classifiers cannot provide contextual detection and that internal-state monitors catch more.","Updating defenses with attack data works: Phase 2's redesigned LLM Judge, newer TaskTracker models, and the Conformal Blocklist raised detection recall on tool-calling attacks, and no attack succeeded against all defenses combined with GPT-4o in Phase 2."],"supporting_citations":[{"why":"Defines indirect prompt injection in LLM-integrated applications, the threat model the challenge operationalizes end-to-end.","marker":"[Greshake et al., 2023]"},{"why":"Supplies Spotlighting, the delimiter-based preventative defense whose team success rates are measured.","marker":"[Hines et al., 2024]"},{"why":"Supplies Prompt Shield, the proprietary black-box text classifier used as one detection defense.","marker":"[Microsoft, 2024]"},{"why":"Supplies TaskTracker's activation-delta detection method and its training data.","marker":"[Abdelnabi et al., 2025]"},{"why":"Grounds the LLM Judge defense in the LLM-as-judge approach.","marker":"[Gu et al., 2024]"},{"why":"Instruction-hierarchy training, the property that motivates GPT-4o-mini's inclusion and explains its higher resistance.","marker":"[Wallace et al., 2024]"},{"why":"Documents real-world prompt-injection-driven data exfiltration from an email assistant, motivating the attack objective.","marker":"Rehberger [2024]"},{"why":"Frames the instruction-data separation question that the dataset is presented as illuminating.","marker":"[Zverev et al., 2025a]"},{"why":"The prior SaTML LLM capture-the-flag competition whose dataset-release model and comparison baseline this challenge builds on.","marker":"[Debenedetti et al., 2024a]"}],"fun_headline_variants":["Adaptive attackers struggle most vs LLM judge","208k adaptive attacks rank email defenses","LLM judge wins prompt injection defense race","Realistic challenge: classifiers fail, LLM judges hold"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the simulated email-assistant environment is a valid proxy for real-world indirect prompt injection — same retrieval behavior, same user queries, same attack goals — so that the measured defense rankings and attack strategies transfer to production systems.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive attackers struggle most vs LLM judge","208k adaptive attacks rank email defenses","LLM judge wins prompt injection defense race","Realistic challenge: classifiers fail, LLM judges hold"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000809,"raw_usage":{"total_tokens":3575,"prompt_tokens":994,"completion_tokens":2581,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":2523}},"tokens_in":610,"tokens_out":2581,"duration_ms":18424,"temperature":1.0,"reasoning_tokens":2523,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:36:44.879237+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four defenses under the same adaptive red-team conditions on a production email-assistant workload with real mailboxes, real user queries, and attackers who do not know the defenses: if the relative difficulty ordering (LLM Judge hardest, then TaskTracker, then Spotlighting, then Prompt Shield) or the under-1% end-to-end success rates do not approximately reproduce, the challenge's proxy assumption is falsified.","supporting_citations":[],"review_version":1}