{"id":"94f53534-4784-4694-8328-6d6d5aa6762f","arxiv_id":"2505.18172","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"The authors present a proactive testing framework using GenAI-powered red and blue teaming agents and report high classification accuracy on the SPML prompt injection dataset, though the evaluation methodology has significant limitations.","lead":"This paper describes an agentic red and blue teaming framework for testing GenAI applications against prompt injection attacks, with an evaluation on the SPML dataset. A generalist reader might care because it addresses the practical need to automate security testing for LLM-based products.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proxy evaluation in Section IV-B measures binary classification, not the Red Teaming agent's attack generation; the reported F1 score does not support the claim of empirically proven framework effectiveness.","rationale":"The reader's core objection is that the evaluation uses a proxy rather than measuring the Red Teaming agent's actual attack-generation capability. I agree. The paper's stated goal is to 'empirically prove the effectiveness' of a framework where the Red Teaming agent generates prompt injections and the Blue Teaming agent recommends mitigations. The load-bearing condition for that claim is that the reported metrics measure these functions. Section IV-B explicitly says 'we use a proxy in order to evaluate the agent,' and the proxy is a binary classification task on the dataset's user prompts. A high F1 on that classification task shows that the evaluator/agent combination can detect injected prompts in the dataset, but it says nothing about whether the agent can generate attacks that would succeed against a live system. This is an internal mismatch between the claimed capability and the measured quantity, not merely a disagreement with current consensus. The Blue Teaming result is also insufficiently supported: 'strong positive correlation' is asserted from a density plot with no coefficient or statistical test, and the recommendation count is capped at 4, making a linear severity-recommendation relationship questionable. However, the Red Teaming proxy is the more fundamental problem because, without demonstrated attack generation, there are no successful attacks for the Blue Teaming agent to analyze. Since the reader already recommended REJECT, my review does not change the verdict; it strengthens the same conclusion.","tokens_in":3421,"tokens_out":3179,"duration_ms":32367,"concrete_test":"Re-run the evaluation in generative mode. For each SPML system prompt, have the Red Teaming agent generate candidate attack user prompts; then submit those generated prompts to a target LLM configured with the system prompt. Determine attack success by whether the target LLM's output violates the system-prompt constraint, or matches the SPML ground-truth injection labels. Report the attack success rate and false-negative rate, and compare against a baseline formed by the SPML injected prompts and by random/template attacks. If the generated prompts' success rate is not substantially above baseline, the Section IV-B proxy does not support the effectiveness claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the proposed agentic framework is effective because the Red Teaming agent generates successful prompt injections and the Blue Teaming agent mitigates them. The reported F1 of 0.985, however, comes from a proxy that collapses evaluation into a binary classification task: an evaluator model (gemini-1.5) checks whether the agent can label a dataset 'User Prompt' as an injection for a given 'System Prompt' (Section IV-B). This metric measures the agent's ability to detect injected prompts, not its ability to generate novel, successful attacks. The paper's own example in Section IV-C supports this reading: it describes the Red Teaming agent failing to classify a nuanced injection, i.e., a detection error. No experiment measures whether prompts generated by the Red Teaming agent induce a target GenAI system to deviate from its instructions, bypass safety filters, or achieve any attack objective. The Blue Teaming evaluation similarly only compares the dataset's 'Degree' field with the count of recommendations chosen from a fixed list of 4; no correlation coefficient, confidence interval, or baseline is reported. Without a direct measure of generated-attack success, the reported F1 is evidence about the proxy classifier, not about the framework's ability to outsmart bots.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an agentic \"proactive testing framework\" for GenAI security that combines a Red Teaming agent (tasked with generating prompt injection attacks) and a Blue Teaming agent (tasked with recommending mitigations), supported by a shared memory and knowledge base. The authors evaluate the framework on the SPML Chatbot Prompt Injection Dataset. The Red Teaming agent is evaluated via an indirect proxy: an evaluator model (gemini-1.5) uses a binary classification task to check whether the agent can label a user prompt as an injection for a given system prompt, achieving an F1 score of 0.985. The Blue Teaming agent is evaluated by comparing the dataset's \"Degree\" severity field with the number of mitigation recommendations (chosen from a fixed list of 4), and the paper claims a strong positive correlation based on a density scatter plot.","tokens_in":3676,"tokens_out":2898,"duration_ms":31350,"significance":"If the framework actually worked as described, a unified, automated red/blue teaming pipeline for GenAI applications would be a useful contribution to security practice. However, the paper's central empirical claim—that the framework is \"effective\" and \"well-suited for real-world applications\"—rests entirely on an indirect proxy evaluation that does not measure the claimed capabilities. The paper provides no direct evidence that prompts generated by the Red Teaming agent cause target GenAI models to deviate from instructions or bypass safety filters, and no statistical quantification for the Blue Teaming correlation. The conceptual architecture is described only at a high level, with no implementation details or code, limiting reproducibility. The potential significance is therefore not realized in the current manuscript.","major_comments":[{"comment":"The Red Teaming agent's objective, stated in Section III, is to generate prompt injection attacks that circumvent safety filters. The evaluation, however, measures the agent's ability to perform binary classification of dataset prompts into \"injection\" vs. \"no injection\" via an evaluator model. The F1 score of 0.985 therefore characterizes this classification proxy, not the agent's attack generation ability. The paper even acknowledges this by calling it a \"proxy\" (Section IV-B) and by giving an example in Section IV-C of the agent failing to classify a nuanced injection. No experiment tests whether prompts actually generated by the Red Teaming agent induce any target GenAI system to behave maliciously. This is load-bearing because the abstract and conclusion claim empirical proof of the framework's effectiveness.","section":"Section IV-B, Table I"},{"comment":"The Blue Teaming evaluation claims a \"strong positive correlation\" between the attack severity (Degree field) and the number of recommendations made by the agent, but no correlation coefficient, confidence interval, or statistical test is reported. The density scatter plot and hand-drawn trend line are not a quantitative substitute. Additionally, the number of recommendations from a fixed list of four is a weak proxy for mitigation quality; recommending more services does not necessarily mean the mitigation is more effective. The claim in Section V that the agent \"showed a clear understanding of the severity of prompt injections\" is therefore unsupported.","section":"Section IV-B, Fig. 3"},{"comment":"The evaluation methodology is circular with respect to the Red Teaming agent's role: the agent is asked to generate attacks, but the proxy evaluates it as a detector. The paper's own framing in Section IV-B transforms the problem into \"binary classification,\" which is a different task than attack generation. This mismatch means that even a perfect F1 score would not justify the conclusion that the framework can \"outsmart the bots\" or \"generate and investigate potential scenarios\" as claimed in the architecture section. The evaluation does not measure any adversarial success metric, such as attack success rate against a target model.","section":"Section IV-B and Section III"},{"comment":"The architecture lacks the detail needed to reproduce or verify the claimed results. The paper does not specify the prompt templates used for the agents, the contents or construction of the Knowledge Base, the implementation of shared memory, the exact evaluator prompt, or the number of trials. Without these details, the empirical results in Tables I and II and Figure 3 cannot be independently checked, which is a significant shortcoming for a paper whose main contribution is an empirically validated framework.","section":"Section III and IV"}],"minor_comments":[{"comment":"The confusion matrix shows 312 false negatives out of 12,533 ground-truth injection cases; the paper should discuss whether these are concentrated in particular attack types or severity levels, since this could affect the interpretation of the model's practical utility.","section":"Section IV-C, Table II"},{"comment":"The limitations listed for traditional red/blue teaming (stagnation, scalability, scope, resource intensity, reactivity, human bias) are asserted without citations; providing references or concrete examples would strengthen the motivation.","section":"Section II, list of limitations"},{"comment":"The references are a mix of relevant prior work (e.g., SurrogatePrompt, SneakyPrompt, SPML, Red-Teaming for Generative AI), but the paper does not position itself relative to existing automated red teaming frameworks, such as those using reinforcement learning or adversarial prompting; adding a brief related-work discussion would help contextualize the contribution.","section":"References"},{"comment":"There are several typographical and clarity issues, including \"tøinvestigate\" in Section III, the undefined acronym SPML in the abstract (defined only in Section IV-A), and Figure 1 appearing before its first mention in the text.","section":"General"},{"comment":"The density scatter plot is referenced but not shown in a way that allows the reader to assess the claimed trend; the red dotted line should be derived from a stated regression or trend model rather than drawn informally.","section":"Section IV-C, Fig. 3"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this as an engineering write-up, not as a scientific validation. The core idea—two GenAI agents, one generating injection attempts, one recommending mitigations, with a shared memory and knowledge base—is sensible and roughly what practitioners are already building. The paper does several things well: it cites the critical literature (including Feffer et al. on red teaming as \"security theater\"), it uses a public dataset (SPML), and it is unusually candid in Section IV-B that the red team evaluation is a proxy because there is no good open benchmark for generated injections. That honesty deserves credit.\n\nThe soft spot is load-bearing. The proxy converts the red team agent's evaluation into a binary classification task: can the agent label an existing user prompt as an injection for a given system prompt. The reported F1 of 0.985 therefore measures detection, not generation. It says nothing about whether the agent can craft a prompt that makes a target model deviate, bypass safety filters, or achieve an attack objective. The paper's own example in IV-C confirms this: the failure case is the agent misclassifying a nuanced injection, not failing to generate one. The blue team result is even thinner—a density scatter plot with a hand-drawn \"red dotted line\" claiming positive correlation between an ordinal \"Degree\" field and the count of recommendations from a fixed list of four. No correlation coefficient, no confidence interval, no baseline. The abstract and conclusion say the framework is \"empirically proven\" and \"robust, balanced, and well-suited for real-world applications.\" That is not what the experiments show.\n\nI would not call this a fraudulent or incoherent paper. The authors know they used a proxy and say so. The weakness is that the conclusions do not respect that limitation. The citation pattern looks fine; self-citation is not an issue. No code or data is shipped, but the dataset is public and the metrics are reproducible enough.\n\nWho gets value? A developer looking for a high-level architecture sketch for agentic security testing might find the system description useful. A researcher evaluating security claims will not learn much from the numbers. I would not cite it in my own work on LLM security evaluation, but I would send it to a workshop as a \"systems and practice\" submission with strong encouragement to either fix the evaluation or reframe the claims as a demo, not a proof. It deserves a serious referee only if the venue is explicitly practice-oriented; for a research venue, the current evidence does not support the central claim.","headline":"A practical agentic red/blue teaming framework whose reported evaluation measures the wrong thing; the paper is honest about its proxy but then overclaims empirical proof.","tokens_in":4163,"tokens_out":1651,"would_cite":false,"duration_ms":19508,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An automated red/blue agent loop catches prompt injections with 0.985 F1.","keywords":["GenAI security","prompt injection","red teaming","blue teaming","agentic framework","LLM testing","proactive security","SPML dataset"],"falsifier":"Take the same Red Teaming agent's flagged and unflagged prompts from the SPML dataset and have independent human security reviewers label them under the same system-prompt/user-prompt pairing; if human-judged true positive rate on the agent's 'no injection' calls is materially higher than the proxy evaluator's implied false-negative rate, the 0.985 F1 overstates the agent's real detection ability.","tokens_in":3239,"feed_emoji":"🛡️","tokens_out":5506,"duration_ms":50835,"temperature":0.7,"pith_summary":"This paper tries to establish that a fully automated, two-agent testing framework can move GenAI security from reactive patching to proactive testing. A Red Teaming agent probes chatbot system prompts for injection vulnerabilities, and a Blue Teaming agent recommends mitigation services for attacks that succeed. Using the SPML Chatbot Prompt Injection Dataset, the authors report that the Red Teaming agent identifies injected user prompts with an F1 of 0.985, and that the Blue Teaming agent's number of recommendations rises with the dataset's severity score. A sympathetic reader would take the central claim to be that a generative-model-driven red/blue team loop can be both accurate enough and scalable enough for real-world GenAI application security.","feed_headline":"Two-agent AI security loop catches prompt injections at 98.5% F1","feed_subtitle":"Automated red and blue team agents scale GenAI testing and tie mitigation effort to attack severity.","key_machinery":"The load-bearing mechanism is the two-agent architecture: a Red Teaming agent that uses a knowledge base of prompt-attack techniques (SurrogatePrompt, SneakyPrompt) and a shared short-/long-term memory to generate or recognize injection attempts, and a Blue Teaming agent that consumes the Red agent's findings and returns a set of mitigation recommendations. What carries the argument is the proxy evaluation loop: the Red agent's output is scored as a binary classifier by an external evaluator model, and the Blue agent is calibrated by comparing recommendation count against the dataset's Degree field. The framework's claims of automation and adaptability rest on this loop's ability to feed new attack knowledge back into the agents.","core_discovery":"The central discovery is that an agentic red-team/blue-team loop, with each agent built on a large language model and connected through shared memory and a knowledge base of known attack techniques, can detect prompt injections and suggest proportional mitigations without human involvement at test time. The authors reframe Red-agent evaluation as a binary classification problem: given a system prompt and a user prompt, the agent decides whether the user prompt is an injection, and an external evaluator model checks that decision against the dataset's labels. Reported results are an F1 of 0.985 (precision 0.9951, recall 0.9751) over the 16K-record SPML dataset, with more false negatives than false positives. For the Blue agent, the paper reports a positive association between the dataset's Degree severity field and the number of mitigation recommendations, and interprets both results as evidence that the framework is suitable for real-world proactive testing.","pith_inferences":["Editorial inference: the paper's proxy evaluation measures classification agreement with the dataset's labels, not whether an attack would actually succeed against a deployed application; a field trial against live chatbots would be the direct test of the framework's real-world claim.","Editorial inference: the same two-agent loop could be applied to other GenAI failure modes, such as data poisoning, jailbreaks, or hallucination-inducing prompts, wherever a severity-graded labeled corpus exists, since neither the architecture nor the evaluation logic is injection-specific.","Editorial inference: the Blue Teaming agent's 'more recommendations for worse attacks' result suggests the recommendation list could be turned into a risk-scoring signal that automatically prioritizes which vulnerabilities to patch first; the paper does not make that prioritization claim."],"forward_implications":["If the framework works as reported, organizations can run prompt-injection testing continuously instead of at snapshot points, keeping defenses current as new bypass techniques appear.","Because both agents are driven by the same generative model family, the testing loop can scale across many system prompts without requiring the same human expert hours that traditional red teams demand.","The measured F1 of 0.985 implies the Red Teaming agent can surface the large majority of injected prompts while keeping false alarms low enough for practical screening.","The observed severity-to-recommendation correlation gives the Blue Teaming agent a way to focus mitigation effort on the attacks that matter most, rather than treating all injections equally.","The paper's framework is designed to be extended: shared memory and the knowledge base allow newly discovered attacks to be folded back into future test runs."],"supporting_citations":[{"why":"Supplies the SPML Chatbot Prompt Injection Dataset: the 16K system-prompt/user-prompt pairs, the binary injection labels used to score the Red Teaming agent, and the Degree field used to calibrate the Blue Teaming agent.","marker":"[5]"},{"why":"SurrogatePrompt is one of the attack families in the Knowledge Base the Red Teaming agent draws on to probe injection techniques.","marker":"[3]"},{"why":"SneakyPrompt is another attack family in the Knowledge Base, giving the Red Teaming agent a known jailbreaking strategy to build variations from.","marker":"[4]"},{"why":"Frames the red-teaming methodology the framework builds on and motivates the attempt to overcome the limitations of traditional red teams.","marker":"[2]"},{"why":"Motivates the adversarial-injection threat the framework targets by showing how indirect instruction injection can be abused in multimodal LLMs.","marker":"[1]"}],"fun_headline_variants":["Two-agent LLM loop catches prompt injections at 98.5% F1","Proactive GenAI testing: red-blue agents hit 98.5% F1","Automated red-team/blue-team framework thwarts prompt injections","GenAI security: agentic loop detects injections with 98.5% F1","Red-blue agent loop: 98.5% F1 on prompt injection detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measured effectiveness rests on accepting the external evaluator model's binary labels and the dataset's Degree ratings as accurate stand-ins for whether a prompt injection would actually succeed and how much defense it needs.","fun_headline_variants_meta":{"raw":{"variants":["Two-agent LLM loop catches prompt injections at 98.5% F1","Proactive GenAI testing: red-blue agents hit 98.5% F1","Automated red-team/blue-team framework thwarts prompt injections","GenAI security: agentic loop detects injections with 98.5% F1","Red-blue agent loop: 98.5% F1 on prompt injection detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000663,"raw_usage":{"total_tokens":2972,"prompt_tokens":834,"completion_tokens":2138,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":2034}},"tokens_in":450,"tokens_out":2138,"duration_ms":14335,"temperature":1.0,"reasoning_tokens":2034,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:33:21.536904+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same Red Teaming agent's flagged and unflagged prompts from the SPML dataset and have independent human security reviewers label them under the same system-prompt/user-prompt pairing; if human-judged true positive rate on the agent's 'no injection' calls is materially higher than the proxy evaluator's implied false-negative rate, the 0.985 F1 overstates the agent's real detection ability.","supporting_citations":[],"review_version":1}