{"id":"c97cff3a-2ae1-43ac-a0ff-562de74fbbf2","arxiv_id":"2607.02357","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Payload-preserving SkillCloak evasions defeat install-time skill scanners at ≥90% with SFS packing, while runtime SkillDetonate detects ~97% of attacks via sandbox taint evidence.","lead":"Static scanners for third-party LLM agent skills can be bypassed at high rates by payload-preserving cloaks, while a sandbox runtime auditor that tracks OS-boundary information flow detects most of the same attacks. This matters because agent skills already ship with developer privileges and public marketplaces are largely unvetted.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"SkillDetonate’s reported detection rates are conditional on a single agent session actually exercising the malicious path; the paper’s own miss analysis shows this is the dominant failure mode.","rationale":"The reader’s weakest_assumption is exactly this path-coverage dependence, and the manuscript already surfaces it honestly in the failure analysis and Discussion. The SkillCloak bypass measurements (RQ1) and utility preservation (RQ2) stand independently and are not undermined by the concern. SkillDetonate’s controlled results, ablation, and stability under cloaks remain valid under the stated experimental conditions. Because the paper does not over-claim universality beyond what a single-session dynamic auditor can observe, the existing CONDITIONAL verdict (accept-shaped contribution pending stronger path-coverage/anti-sandbox evaluation and artifacts) is already the right posture; no further adjustment is required.","tokens_in":19645,"tokens_out":563,"duration_ms":28730,"concrete_test":"Re-run SkillDetonate on the same 622 wild skills (and the SkillJect suite) under a forced-execution agent prompt that systematically follows every natural-language directive and branch (the mitigation suggested in §VI-A). Recompute detection rate and FPR. If detection rises substantially (e.g., >95%) while FPR stays ≤2%, the path-coverage gap is confirmed as the primary limiter of the reported 87%; if detection remains near 87%, other factors dominate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SkillDetonate detects ~95–98% of SkillJect attacks (2% FPR) and ~87% of executable wild skills while remaining stable under SkillCloak rests on the assumption that one sandboxed session with the reference agent (opencode+DeepSeek) will materialize and run the payload often enough for OS-boundary marker taint and information-flow policies to fire. §V-A and Fig. 6 classify the majority of wild misses as “Payload not triggered” (agent reads the instruction but does not execute it), with secondary environment-dependency and timeout causes; §VI-A explicitly frames this natural-language coverage gap (and anti-sandbox steering) as the deepest open problem. Consequently the headline percentages are path-exercise-conditional under the chosen agent and harness; they do not by themselves establish that the same skills would be caught by production agents that decline the branch or by adversarially gated skills. This does not falsify the measured numbers or the bypass results for SkillCloak, but it is the least-secure condition for the stronger conclusion that behavior-centric auditing is the load-bearing defense.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper studies the security of third-party agent skills for LLM coding agents. It first introduces SkillCloak, a payload-preserving evasion framework with Structural Obfuscation (rewrite/disperse scanner signals) and Self-Extracting Skill (SFS) Packing (hide payload until runtime). On 1,613 in-the-wild malicious skills and eight/nine open-source scanners, SFS Packing achieves ≥90% bypass on every scanner and Structural Obfuscation often ≥80%, while SkillsBench shows no statistically detectable utility loss on Codex and Claude Code. Motivated by this, it proposes SkillDetonate, a sandboxed runtime auditor that uses on-demand closure lift (execute instructions materialized under the skill root) and marker-based taint (FUSE-planted #dataN markers plus eBPF inode-level graphs) under confidentiality/integrity policies. On SkillJect (with cloaked variants) it reports ~95–98% detection at 2% FPR and ~87% on a filtered MalSkillBench subset, remaining stable under SkillCloak; ablations isolate taint (FP control) vs. closure lift (packed multi-stage coverage).","tokens_in":20079,"tokens_out":1385,"duration_ms":29230,"significance":"If the results hold, the paper makes a timely and concrete contribution to an emerging supply-chain surface: agent skills as privileged, marketplace-distributed capability packages. The adversarial study is unusually strong for this niche—large real-world corpus, multiple deployed scanners, an independent evasion baseline (VulMask), and utility preservation on production agents—establishing that install-time appearance auditing is systematically insufficient under payload-preserving transforms. SkillDetonate’s designs (closure lift for staged skills; markers that make the LLM context taint-trackable without in-model instrumentation) are technically well-motivated adaptations of classical detonation/taint ideas to agent-mediated execution. Honest miss analysis and complementary ablations strengthen credibility. The work is significant for both marketplace operators and agent runtime designers even if dynamic coverage remains incomplete.","major_comments":[{"comment":"Abstract, §V-A, Fig. 6, and §VIII: The headline claim that SkillDetonate “detects 97% of attacks at a 2% false-positive rate and sustains 87% detection on real-world malicious skills” is path-exercise-conditional. Fig. 6 shows the dominant wild-miss cause is “Payload not triggered” (agent reads but does not execute the malicious instruction), with environment dependency and timeouts secondary; §VI-A correctly calls natural-language coverage the deepest open problem. The measured numbers are still valuable, but the abstract/conclusion should state that rates are under a single sandboxed session with the reference agent and that non-execution leaves no OS-boundary evidence. Without that qualification, the stronger framing that behavior-centric auditing is the load-bearing defense overstates what the evaluation establishes.","section":"Abstract; §V-A; Fig. 6; §VI-A; §VIII"},{"comment":"§IV-B3 and §V-A (Table V): SkillDetonate’s detection evaluation uses opencode+DeepSeek as the reference agent, whereas RQ2 utility is measured on OpenAI Codex (GPT-5.4) and Claude Code (Sonnet 4.6). Compliance with staged/malicious natural-language instructions can differ across models and harnesses; if DeepSeek is more willing to follow decode-and-run or exfil steps, detection rates may be inflated relative to production agents that refuse or sanitize. At least a multi-agent detection sweep (or a clear argument that the reference stack is a conservative lower bound) is needed for the claim that SkillDetonate remains effective when skills are run as victims would run them.","section":"§IV-B3; §V-A Table V; §III-D"},{"comment":"§IV-B2–3 and free configuration: Marker-based policies depend on the sensitive-file set, integrity allowlists, and egress allowlists. The paper treats these as configuration but does not report sensitivity of detection/FPR to reasonable allowlist choices (e.g., skills that legitimately call package managers or cloud APIs). A short sensitivity or default-policy study would show that the 2% FPR and high detection are not artifacts of an overly tight or loose policy tailored to SkillJect/MalSkillBench sinks.","section":"§IV-B2; §IV-B3; §V-A"}],"minor_comments":[{"comment":"Abstract says “eight scanners”; Table I and §III-C refer to nine surveyed/runnable scanners. Align the count and naming throughout.","section":"Abstract; Table I; §III-C"},{"comment":"§III-C: “delopyed” → “deployed”; elsewhere “adversial” → “adversarial”. Light copy-edit pass recommended.","section":"§III-C; §IV-A"},{"comment":"Fig. 4 / Table III: Report absolute n per scanner (skills flagged pre-evasion) in the figure or caption so bypass rates are fully interpretable without reconstructing from finding counts.","section":"Fig. 4; Table III"},{"comment":"§V-A: The filter from 703 to 622 MalSkillBench skills (“executable ground truth… Linux-runnable effect”) is important; a short appendix listing exclusion categories would help reproducibility.","section":"§V-A"},{"comment":"Table II operator examples are useful; a one-line note on which operators fire most often on the 1,613 corpus would connect the design to the empirical bypass results.","section":"Table II; §III-C"}],"recommendation":"minor_revision","confidential_remarks":"Strong, timely systems/security paper for the agent-skills supply chain. SkillCloak results look publication-ready; SkillDetonate is solid but the abstract should not outrun the path-coverage caveat the authors already document. Fit is good for a top security venue if claims are tightened. No integrity red flags; circularity risk is low given external scanners, IOC-derived corpus, SkillJect/MalSkillBench, and VulMask."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The headline result is solid: on 1,613 wild malicious skills, SkillCloak’s SFS Packing clears every surveyed scanner at ≥90%, Structural Obfuscation often ≥80% (96% on Cisco), and SkillsBench shows no utility loss on Codex/Claude. That is the first real adversarial measurement of this new supply-chain surface, not a toy rewrite of packing.\n\nWhat is new is the skill-specific packaging of known ideas. Structural Obfuscation is a counterexample-guided least-edit loop against open scanners; SFS Packing is SFX for agent bundles. SkillDetonate’s two mechanisms—on-demand closure lift for runtime-materialized .md, and marker-based taint that carries provenance through LLM context and opaque cross-process transforms via FUSE+eBPF—are the right adaptations. Ablations isolate them cleanly: drop taint and FPR jumps 8×; drop closure lift and packed detection collapses 65 pp. VulMask as an independent evasion baseline helps. Failure analysis is honest about path coverage, environment gates, and timeouts.\n\nThe soft spot the stress-test flags is real but already owned by the paper. Detection is conditional on one sandboxed session actually firing the payload; Fig. 6 shows “payload not triggered” as the dominant wild miss, and §VI names natural-language coverage and anti-sandbox steering as open problems. That does not falsify the measured 97%/2% SkillJect or ~87% wild numbers, or the scanner-bypass claims. It does mean the stronger slogan “behavior-centric auditing is the load-bearing defense” is true only for paths the agent takes. Policies (sensitive files, allowlists) are configuration, wild set is filtered to executable Linux effects, and artifacts are not shipped—standard systems caveats, not circularity.\n\nMath is light (systems work); data and citation pattern look fine—malware taint/packing lineage is acknowledged, not reinvented. For anyone working agent supply chain, marketplace scanners, or dynamic analysis of LLM tools, this is worth reading. I would send it to peer review; strengthen path-coverage/anti-sandbox evaluation and ship the harness if possible, but the core evidence already earns referee time.","headline":"Strong first systematic attack on skill scanners plus a workable runtime detector; detection rates are real but path-exercise-conditional.","tokens_in":20646,"tokens_out":546,"would_cite":true,"duration_ms":6292,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Appearance-based scanners for third-party agent skills are systematically bypassable by payload-preserving cloaks, so trust must rest on observed runtime effects.","keywords":["LLM agents","agent skills","software supply chain","dynamic analysis","information-flow tracking","evasion attacks","skill scanners","taint analysis"],"falsifier":"Run SkillDetonate on a large set of cloaked wild skills that independently produce the expected exfiltration or integrity-violating syscalls under a force-execution harness; if detection falls far below the reported 87–97% while those syscalls still fire, the claim that OS-boundary behavioral auditing withstands the cloaks fails.","tokens_in":20570,"feed_emoji":"🕵️","tokens_out":1074,"duration_ms":20616,"temperature":0.7,"pith_summary":"LLM coding agents install third-party skills that run with the agent's privileges, opening a supply-chain path to credential theft, source exfiltration, and backdoors. This paper shows that existing install-time skill scanners—pattern matching, LLM-as-judge, or hybrid—fail against adaptive evasions that keep the malicious behavior intact while changing how the payload looks. SkillCloak rewrites scanner-visible signals into equivalent forms and packs the real skill so it only reappears when the agent runs it; across eight scanners and 1,613 wild malicious skills, packing bypasses every scanner above 90%, and structural rewrites clear most static tools above 80% without destroying utility on production agents. The authors therefore build SkillDetonate, a sandbox auditor that re-injects instructions the skill materializes at runtime and tracks sensitive data with markers through the agent context, files, processes, and network. SkillDetonate detects 97% of controlled attacks at a 2% false-positive rate and about 87% of executable wild malicious skills, and it stays stable under the same cloaks that collapse static tools.","feed_headline":"Cloaked agent skills bypass every static scanner","feed_subtitle":"Runtime taint analysis detects 97% of attacks that appearance checks miss","key_machinery":"SkillCloak is the payload-preserving evasion framework (Structural Obfuscation rewrites or disperses scanner signals; Self-Extracting Skill Packing hides the payload outside the scanner's view and rematerializes it at run time). SkillDetonate is the runtime auditor that pairs on-demand closure lift—feeding natural-language instructions the skill writes into its own tree back into the agent session—with marker-based taint that tracks sensitive reads across agent context, inode graphs, and network sinks under confidentiality and integrity policies.","core_discovery":"Current skill auditors that judge install-time appearance are systematically defeatable by payload-preserving transforms: Self-Extracting Skill Packing bypasses every surveyed scanner at over 90%, and Structural Obfuscation exceeds 80% on most static scanners and reaches 96% on a hybrid scanner, while cloaked skills remain functional on Claude Code and Codex. A behavior-centric runtime auditor that decides from OS-boundary information-flow evidence recovers detection, reaching 97% at 2% false positives on SkillJect and 87% on executable wild skills, without collapsing under the cloaks.","pith_inferences":["Standardized agent sandboxes with canary credentials and mocked external services will become as central to skill security as instrumented malware sandboxes are for binaries.","Force-execution agents that drive every natural-language branch could close the dominant wild-miss mode without changing the detector's core logic.","The same packing and rewrite patterns are likely to transfer to other agent-extensibility formats that ship natural-language instructions plus code.","Post-install tree-integrity checks raise packing cost but leave structural obfuscation intact, so layered defense still needs a runtime stage."],"forward_implications":["Install-time appearance checks cannot be the sole trust gate once adaptive authors can rewrite or stage payloads while preserving attack semantics.","Marketplaces and CI pipelines need a dynamic detonation path that observes OS-level effects before a skill is fully trusted.","Static scanners remain useful only as cheap hygiene pre-filters, not as decisive allow/block mechanisms.","Multi-stage packed skills require the auditor to re-inject runtime-generated instruction files into the same agent session.","False-positive control depends on sensitive-to-untrusted information-flow evidence rather than mere file or network contact."],"fun_headline_variants":["Payload-preserving cloaks defeat every static skill scanner","Self-extracting skill packs bypass scanners at over 90%","Structural obfuscation hits 96% bypass on hybrid skill scanners","Runtime OS taint analysis detects 97% of cloaked skill attacks","Install-time skill audits fail; behavior-centric detection holds"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The sandboxed agent must actually follow the skill's malicious instructions so the attack produces OS-boundary effects; if the agent skips a branch, the environment is incomplete, or the skill steers away from the sandbox, there is nothing for the detector to judge.","fun_headline_variants_meta":{"raw":{"variants":["Payload-preserving cloaks defeat every static skill scanner","Self-extracting skill packs bypass scanners at over 90%","Structural obfuscation hits 96% bypass on hybrid skill scanners","Runtime OS taint analysis detects 97% of cloaked skill attacks","Install-time skill audits fail; behavior-centric detection holds"]},"model":"grok-4.5","effort":"low","cost_usd":0.005252,"raw_usage":{"total_tokens":1556,"prompt_tokens":925,"num_sources_used":0,"completion_tokens":88,"cost_in_usd_ticks":52520000,"prompt_tokens_details":{"text_tokens":925,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":543,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":925,"tokens_out":88,"duration_ms":4416,"temperature":1.0,"reasoning_tokens":543,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T08:08:11.173016+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run SkillDetonate on a large set of cloaked wild skills that independently produce the expected exfiltration or integrity-violating syscalls under a force-execution harness; if detection falls far below the reported 87–97% while those syscalls still fire, the claim that OS-boundary behavioral auditing withstands the cloaks fails.","supporting_citations":[],"review_version":2}