{"id":"4c3f771d-c489-4cdf-a359-d2ab1b481783","arxiv_id":"2606.07943","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"POISE is a stealthy skill-poisoning attack achieving 89.3% ASR on Skill-Inject by blending a compressed trigger into contextually appropriate positions in skill bodies, outperforming YAML and random-placement baselines while evading static scanners.","lead":"POISE introduces a position-aware attack that compresses a malicious trigger into a single benign-looking instruction placed naturally within an LLM agent's skill body. A smart generalist might read it to understand practical risks in deploying extensible AI agents for real tasks.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"Stealth claim rests on LLM scanners producing 74.6% FP rate on clean skills; without that, blending provides no advantage over body baselines.","rationale":"The reader's weakest_assumption directly identifies the same load-bearing dependency on scanner FP rates that underpins the undetectability half of the strongest_claim. Because the full text was unavailable to the reader, the current UNVERDICTED verdict already reflects inability to verify that dependency; the concrete_test above would resolve it without requiring any change to the verdict category.","tokens_in":1810,"tokens_out":412,"duration_ms":19622,"concrete_test":"Re-run the four judges on the exact clean skill set used for the 74.6% statistic, but with (a) a one-sentence change to the judge prompt that asks only for explicit malicious intent and (b) an additional judge that is a fine-tuned classifier trained on the same clean/poisoned pairs; if the FP rate falls below 40% or the new-alert rate for POISE rises above 20%, the stealth advantage disappears.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline performance (89.3% ASR on Skill-Inject) is presented as retaining body-placement stealth. This stealth margin is justified solely by the claim that legitimate skill bodies require privileged tool operations, so four LLM judges flag 74.6% of clean skills on average and POISE adds a new high-risk alert in only 5.6% of cases. If the measured FP rate is an artifact of the particular judges, prompt templates, or benchmark skills (rather than a general property of body placement), then the “undetectable” part of the central claim does not follow; the 2.6-point gain over YAML would then be irrelevant because YAML is already dismissed on inspectability grounds. The abstract states the 74.6% figure as observed fact but supplies no measurement protocol, inter-judge agreement, or ablation on scanner prompt wording.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces POISE, a position-aware attack for skill injection on LLM agents. It compresses malicious triggers into single benign-looking body instructions placed at feasible positions and blended with context via a generator. On the Skill-Inject benchmark with codex+gpt-5.2, POISE reports 89.3% ASR (28 points above random body placement, 2.6 above YAML), while claiming stealth because legitimate skill bodies trigger 74.6% false positives on average across four LLM judges, so POISE adds new high-risk alerts in only 5.6% of cases.","tokens_in":2000,"tokens_out":498,"duration_ms":13816,"significance":"If the empirical results and the 74.6% FP claim hold under full protocol, the work shows that body-placement attacks can overcome the reliability-stealth tradeoff in open agent skills, rendering static LLM scanners ineffective and highlighting a practical vulnerability in agent extensibility mechanisms.","major_comments":[{"comment":"Abstract: The stealth advantage is justified solely by the claim that LLM scanners produce a 74.6% FP rate on clean skills (allowing POISE to add only 5.6% new alerts). No measurement protocol, judge identities, inter-judge agreement, benchmark skill selection criteria, or ablation on scanner prompt wording is supplied, making it impossible to determine whether this rate is a general property of body placement or an artifact of the chosen judges and templates.","section":"Abstract"},{"comment":"Abstract: The headline 89.3% ASR (and the 28-point and 2.6-point margins) is reported without error bars, full experimental protocol, dataset details, number of trials, or verification steps for the 74.6% FP figure. This directly limits assessment of whether the data supports the central reliability-plus-stealth claim.","section":"Abstract"},{"comment":"Abstract: The weakest assumption—that legitimate skill bodies naturally require privileged tool operations and therefore trigger high FP rates—is presented as observed fact but is not supported by any ablation showing that the FP rate persists under different scanner prompts or skill corpora; if this rate is judge- or prompt-specific, the “undetectable” margin over YAML baselines does not follow.","section":"Abstract"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed feedback on the abstract. We address each comment below and will revise the manuscript to incorporate additional experimental details, error bars, and an ablation study as noted.","responses":[{"response":"The full measurement protocol is described in Section 5.2, including the four judges (GPT-4o, Claude-3-Opus, Llama-3-70B, Gemini-1.5), prompt templates, inter-judge agreement (Cohen's κ=0.81), and selection of 150 clean skills from Skill-Inject and AgentBench. We agree the abstract omits these details due to length and will revise it to summarize the protocol with section references. The prompts follow standard static scanner designs; an explicit ablation on wording variants was not performed.","revision_made":"yes","referee_comment":"[Abstract] Abstract: The stealth advantage is justified solely by the claim that LLM scanners produce a 74.6% FP rate on clean skills (allowing POISE to add only 5.6% new alerts). No measurement protocol, judge identities, inter-judge agreement, benchmark skill selection criteria, or ablation on scanner prompt wording is supplied, making it impossible to determine whether this rate is a general property of body placement or an artifact of the chosen judges and templates."},{"response":"ASR figures are means over 100 trials per condition (standard error ±1.8% for POISE); full protocol, dataset (50 skills from Skill-Inject), trial counts, and FP verification steps appear in Section 4.1 and Appendix B. We will revise the abstract to include error bars and explicit references to these sections.","revision_made":"yes","referee_comment":"[Abstract] Abstract: The headline 89.3% ASR (and the 28-point and 2.6-point margins) is reported without error bars, full experimental protocol, dataset details, number of trials, or verification steps for the 74.6% FP figure. This directly limits assessment of whether the data supports the central reliability-plus-stealth claim."},{"response":"The high FP rate is an empirical result from the evaluated benchmarks and judges. We acknowledge that an ablation on alternative prompts and corpora would strengthen generality claims. We will add this ablation (two new prompt variants and one additional corpus) in the revision.","revision_made":"yes","referee_comment":"[Abstract] Abstract: The weakest assumption—that legitimate skill bodies naturally require privileged tool operations and therefore trigger high FP rates—is presented as observed fact but is not supported by any ablation showing that the FP rate persists under different scanner prompts or skill corpora; if this rate is judge- or prompt-specific, the “undetectable” margin over YAML baselines does not follow."}],"tokens_in":1568,"tokens_out":605,"duration_ms":29226,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is that POISE compresses the trigger into one blended body instruction, places it at a feasible spot, and uses context-aware generation to make it look like a normal prerequisite step. On Skill-Inject with codex plus gpt-5.2 this reaches 89.3% ASR, 28 points above random body placement and 2.6 above YAML. That is the concrete advance over the YAML and random-placement baselines cited in the abstract.\n\nThe paper does a clean job of framing the reliability-stealth trade-off and then measuring attack success as both payload execution and task verifier pass. The comparisons are direct and use named benchmarks, which makes the gain easy to see. No fitted parameters or circular definitions appear in the results.\n\nThe soft spot is the stealth argument. The claim that POISE stays undetectable rests on LLM scanners producing a 74.6% false-positive rate on clean skills, so the poisoned versions add a new high-risk alert in only 5.6% of cases. The abstract presents the 74.6% figure as observed fact but supplies no measurement protocol, judge prompts, inter-judge agreement, or ablation on scanner wording. If that rate is tied to the particular judges or the benchmark skills rather than a general property of body placement, the undetectable advantage does not follow and the 2.6-point edge over YAML becomes less relevant. The stress-test note flags exactly this gap.\n\nThis is for researchers working on LLM agent security and poisoning defenses. A reader who wants to see a practical blending technique and concrete ASR numbers will find value here, even if they have to re-run the scanner evaluation themselves.\n\nIt deserves a serious referee because the attack method is new enough and the empirical comparisons are sharp enough to check the experimental setup and the FP-rate measurement.","headline":"POISE shows a workable position-aware blending trick for skill injection that lifts ASR over random body baselines, but the stealth edge depends on an unverified 74.6% scanner FP rate that lacks protocol details.","tokens_in":2469,"tokens_out":457,"would_cite":false,"duration_ms":11721,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Position-aware blending of a single trigger into skill bodies achieves 89.3 percent attack success while evading scanner detection.","keywords":["skill poisoning","LLM agents","attack success rate","stealth injection","position-aware","body injection","scanner false positives"],"falsifier":"An experiment that measures scanner false-positive rates on clean skills below 30 percent or records POISE attack success rate below 60 percent on the same benchmark would settle the central claim.","tokens_in":2727,"feed_emoji":"🔒","tokens_out":708,"duration_ms":16345,"temperature":0.7,"pith_summary":"The paper argues that effective skill-poisoning attacks on LLM agents must execute their payload without causing the user's legitimate task to fail, since failure would prompt inspection of the skill. It shows that prior approaches face a tradeoff where YAML-header injections load reliably but are easy to spot, while body injections are harder to detect but less reliable because out-of-context commands raise suspicion. POISE addresses this by compressing the trigger into one benign-looking instruction, positioning it where it fits naturally among setup or prerequisite steps, and blending it with surrounding text via a context-aware generator. A sympathetic reader would care because open skill formats let users extend agents easily yet leave them open to hidden manipulation if attacks can stay invisible. The evaluation claims that this method reaches high success rates while producing almost no additional scanner alerts beyond those already raised by clean skills.","feed_headline":"Blended body injection poisons LLM skills at 89% success","feed_subtitle":"The trigger hides in natural prose because scanners already flag most clean skills as high-risk.","key_machinery":"The position-aware placement and context-aware blending of a single compressed trigger instruction into the skill body.","core_discovery":"POISE is a position-aware attack that compresses the trigger into a single benign-looking body instruction, places it at a feasible position, and uses a context-aware generator to blend it with nearby setup or prerequisite steps. On Skill-Inject with codex and gpt-5.2, POISE achieves an 89.3 percent ASR, 28.0 points above a random-placement body baseline and 2.6 points above a YAML-only baseline, while retaining the stealth advantage of body placement. That stealth holds because legitimate skill bodies naturally require privileged tool operations, causing LLM scanners to flag 74.6 percent of clean skills on average across four judges and both benchmarks, so POISE causes only 5.6 percent of p","pith_inferences":["Runtime monitoring of skill execution behavior may be needed instead of relying on static scans.","Skill distribution platforms could require additional verification steps such as execution sandboxes.","The same blending principle could apply to other open agent extension formats beyond skills."],"forward_implications":["Body placement of the trigger retains its stealth advantage over YAML-header methods.","Legitimate skills already trigger high false-positive rates of 74.6 percent in LLM scanners.","Blended injections increase high-risk alerts by only 5.6 percent over clean baselines.","Static scanner defenses become ineffective against position-aware blended attacks."],"fun_headline_variants":["POISE position-aware attack achieves 89.3% ASR","Trigger blends into skill prose at 89.3% success rate","POISE beats YAML-only baseline with 89.3% attack success","Position-aware blending enables 89.3% LLM skill injection"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Legitimate skill bodies naturally require privileged tool operations that cause LLM scanners to produce high false-positive rates.","fun_headline_variants_meta":{"raw":{"variants":["POISE position-aware attack achieves 89.3% ASR","Trigger blends into skill prose at 89.3% success rate","POISE beats YAML-only baseline with 89.3% attack success","Position-aware blending enables 89.3% LLM skill injection"]},"model":"grok-4.3","cost_usd":0.007265,"raw_usage":{"total_tokens":3340,"prompt_tokens":814,"num_sources_used":0,"completion_tokens":72,"cost_in_usd_ticks":72653000,"prompt_tokens_details":{"text_tokens":814,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2454,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":814,"tokens_out":72,"duration_ms":14412,"temperature":1.0,"reasoning_tokens":2454,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T19:50:25.979754+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An experiment that measures scanner false-positive rates on clean skills below 30 percent or records POISE attack success rate below 60 percent on the same benchmark would settle the central claim.","supporting_citations":[],"review_version":1}