{"id":"902dc131-ce12-495b-b075-612a251824f4","arxiv_id":"2608.09732","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ColluSkill splits a malicious intent across three interdependent agent skills to evade six skill scanners in 96.0% of cases, while ChainGuard reduces this to 22.5% by analyzing a candidate skill with its installed-skill context.","lead":"Agent skills are an emerging attack surface for LLM-based agents. This paper proposes ColluSkill, a multi-skill-chain attack that hides malicious intent across three locally plausible skills to evade existing scanners, and ChainGuard, a defense that checks each new skill against the skills already installed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 96.0% ASR is measured against the same six scanners used for feedback refinement; without held-out scanner evaluation, the headline overstates generalizable evasion.","rationale":"The reader's weakest_assumption correctly identifies the circularity between scanner-feedback refinement and evaluation. This is the most load-bearing concern because the paper's headline contribution is the quantitative evasiveness claim, not merely the conceptual observation that cross-skill composition can be risky. The ablation shows that most of the gain over a naive split comes from feedback refinement against the exact evaluated scanners, so the 96.0% number is inherently tied to that scanner set. However, the concern does not invalidate the paper's core qualitative finding: cross-skill composition with ordered artifact flow can evade single-skill scanners even before refinement (68.2% ASR with chain planning alone), and the runtime activation study provides some evidence that such chains execute in real agents. The ChainGuard defense results are also plausibly interesting, though they share the same unverified dataset and procedure. Given the missing code, data, and appendices, the reader's CONDITIONAL verdict remains appropriate; a held-out scanner evaluation would either reinforce or weaken the central claim, but does not change the verdict category.","tokens_in":13199,"tokens_out":3600,"duration_ms":32607,"concrete_test":"Hold out two or three scanners (e.g., Snyk Agent Scan, SkillProbe, or a newer version of Cisco Skill Scanner) from the refinement loop. Run ColluSkill with chain planning alone and with refinement using only the remaining scanners, then measure ASR on the held-out scanners. Additionally, rerun the full experiment with an adaptive defender that updates scanner weights or appends a watermark after each batch of queries. If held-out ASR falls materially below the reported 96.0% (e.g., below 70%), the generalizability of the central claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim rests on an evaluation protocol where the target scanners serve simultaneously as the refinement oracle and the test set. In Section 'Refine with scanner feedback', ColluSkill submits each sub-skill to 'the target scanner set D' and rewrites flagged sub-skills until they pass; Table 1 then reports ASR against that same set D. This makes the 96.0% average ASR a measure of query-based overfitting to six specific detectors, not of generalizable cross-skill evasion. The ablation in Table 3 is telling: chain planning alone reaches 68.2% ASR, and scanner-feedback refinement contributes the remaining ~28 points. Because refinement is performed on the exact scanners used for evaluation, the second-stage gain would not transfer to unseen scanners, updated scanners, or scanners that watermark or rate-limit repeated queries. The paper provides no held-out scanner experiment, no adaptive-defender baseline, and no sensitivity analysis to scanner version or query budget. Consequently, the claim that existing skill scanners have a practical cross-skill blind spot is supported only for this fixed, queryable scanner set. The qualitative threat remains credible, but the headline number should be interpreted as an upper bound for a specific adversary model.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies cross-skill composition as an attack surface for LLM agent skill scanners. The authors propose ColluSkill, which decomposes a malicious payload into three ordered, interdependent sub-payloads packaged as separate skills, uses LLM-based chain planning to preserve chain-level semantics, and then iteratively rewrites sub-skills that are flagged by a set of target scanners. They evaluate ColluSkill against six skill scanners, reporting an average attack success rate of 96.0%, and show chain activation on OpenCode, Claude Code, and Codex. They also propose ChainGuard, a context-aware scanner that jointly analyzes a candidate skill with installed skills; it reduces ColluSkill's ASR to 22.5% while passing 99.5% of benign workflows. The central claims are that existing scanners have a practical blind spot for cross-skill composition and that ChainGuard closes much of that gap.","tokens_in":13466,"tokens_out":5501,"duration_ms":42911,"significance":"Cross-skill composition is a timely and plausible threat: the attack idea is well-motivated by the modular skill ecosystem, the framework is concrete, and the runtime activation study goes beyond pure scanner evasion. The use of interdependence chains and artifact passing is a useful conceptual framing, and the ablation separating naive split, chain planning, and scanner-feedback refinement is informative. ChainGuard's candidate-with-context design is a reasonable defense direction, and the benign-workflow pass rate is a welcome check. However, the headline 96.0% ASR is currently a measure of overfitting to the exact six scanners used as refinement oracles, and the defense numbers come from the same team without an adaptive adversary. If the evaluation is reworked with held-out scanners and proper uncertainty quantification, the framework could be an important contribution; in its present form the empirical support for the generalizable-blind-spot claim is not yet sufficient.","major_comments":[{"comment":"The evaluation protocol is circular in a way that is load-bearing for the headline result. In the 'Refine with scanner feedback' paragraph, every flagged sub-skill is rewritten until it passes 'the target scanner set D', and Table 1 then reports the attack success rate against that same set D. ColluSkill is therefore optimized directly on the exact six detectors used for evaluation, and the 96.0% average ASR (Table 1, last row) is best interpreted as an upper bound for an adversary with unlimited queries to those specific scanner instances, not as evidence of a generalizable cross-skill blind spot. The paper needs a held-out scanner evaluation (e.g., scanners or scanner versions not used during refinement), an adaptive-defender baseline, and a query-budget sensitivity analysis; without these, the contribution of scanner-feedback refinement shown in Table 3 (from 68.2% to 96.0%) cannot be separated from overfitting.","section":"Refine with scanner feedback; Table 1"},{"comment":"Hyperparameters are selected post hoc from the same evaluation curves used to report performance. The 3-step chain is chosen 'because it achieves the highest final average ASR' (subsection 'Effect of the number of sub-skills'), and the reported 7-iteration budget is described after observing that the curve 'becomes much flatter' after iteration 5 (subsection 'Effect of LLM refinement iterations'). Selecting the chain length and stopping iteration on the test set inflates the reported 96.0% and can bias the comparison against baselines that were not tuned with scanner feedback. Please use a validation split or nested selection procedure and report the selected configuration's performance on a held-out test set.","section":"Effect of LLM refinement iterations; Figure 3; Effect of the number of sub-skills"},{"comment":"No uncertainty quantification is reported for any of the central point estimates. Tables 1, 2, 3, and 4 report single runs without confidence intervals, standard deviations, or significance tests, even though both the attack generation and the four LLM-based scanners use GPT-5.5 and are therefore stochastic. Claims such as 'consistently outperforms' (Table 1 discussion) and the chain-length ranking in Figure 3 require repeated trials or a statistical test before they can be accepted at face value.","section":"Experimental Setup; Tables 1 and 2"},{"comment":"The defense evaluation is not yet convincing as evidence of a practical defense. ChainGuard is proposed and evaluated by the same team, and the reported 22.5% ASR is obtained against a fixed set of ColluSkill chains that were optimized against the six third-party scanners, not against ChainGuard. There is no adaptive attacker that can observe ChainGuard's context-based decisions and rewrite chains, and there is no sensitivity analysis of ChainGuard's dependency-inference thresholds. Without such an analysis, the drop from 69.0% to 22.5% in Table 4 may reflect a specific design match rather than a robust property of context-aware scanning.","section":"ChainGuard Defense Evaluation; Table 4"}],"minor_comments":[{"comment":"Appendices A and B are referenced in 'Implementation details' but are not included in the submitted manuscript; please include them or remove the reference.","section":"Implementation details"},{"comment":"Figure 1 contains the typo 'Payload-decompostion'; also the diagram labels 'iterative refinement' three times without indicating the loop's termination condition.","section":"Figure 1"},{"comment":"The abstract and introduction state that the authors conducted 'an empirical study of existing skill scanners' that found the individual-inspection blind spot, but no methods or results for that study are reported; either add a subsection or cite a prior study.","section":"Abstract and Introduction"},{"comment":"The attacker LLM is denoted Gθ but θ is never defined, and the sentence 'the program writes its body into a separate SKILL.md file' is ambiguous about who writes the file.","section":"In-chain Attack Generation"},{"comment":"The term 'CISCO Skill Scanner' should be 'Cisco Skill Scanner' to match the reference list, and scanner names such as 'SkillSpector' should be checked for consistent capitalization.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely of interest to the agent-security community, but the main quantitative claims currently rest on a circular evaluation. I would condition acceptance on a reworked evaluation with held-out scanners, validation-based hyperparameter selection, and uncertainty quantification. I also note that the manuscript references two appendices that are missing, which complicates reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ColluSkill/ChainGuard is a solid, well-scoped attack paper. The threat it targets—malicious intent split across multiple skill files that pass individual scans but compose into a harmful workflow—is real, and the paper demonstrates it convincingly against six currently deployed scanners. The new piece is the combination of LLM-based chain planning with iterative scanner-feedback refinement; the ablation in Table 3 cleanly shows that naive splitting alone gets only 36.7%, ordered planning brings it to 68.2%, and refinement to 96.0%. The runtime activation numbers on three coding agents add useful evidence that the chain actually executes.\n\nThe main soft spot is the evaluation protocol, and it matches the stress-test concern. The attack queries the same six scanners to rewrite flagged sub-skills, then reports success against those same scanners. That is a legitimate black-box threat model, but it means the 96% is a measure of success against five specific static detectors, not a generalizable property of \"skill scanners\" as a class. There is no held-out scanner, no scanner-update simulation, and no transferability test. The broader claim that existing defenses have a cross-skill blind spot is supported only for this fixed set. A defender who watermarks queries, rate-limits the refinement loop, or simply updates the scanner could break the attack. The authors should be asked for held-out scanner evaluation and at least one adaptive-defense simulation.\n\nChainGuard's evaluation is thin in a different way: it is only compared to itself without installed-skill context (Table 4), and the 22.5% residual ASR is not benchmarked against any other defense. That is acceptable for a first demonstration, but the defense claim is weaker than the attack claim. The post hoc selection of chain length (3) and refinement budget (7 iterations) from the same evaluation curves in Figure 3 is another reason to treat the headline as an upper bound. Also, the appendices and code are not in the manuscript, so reproducibility cannot be checked.\n\nNone of this undermines the central result: the attack works on six real scanners, and the chain-level threat is credible. The paper cites prior work (SCRBench) and honestly positions itself as an extension. I would send it to review, asking for the missing transferability analysis, defense baselines, and code or a clear reproducibility appendix.","headline":"The cross-skill composition threat is real and the paper demonstrates it against six current scanners, but the 96% ASR is an upper bound because the attack is tuned and tested on the same scanners.","tokens_in":13964,"tokens_out":3742,"would_cite":true,"duration_ms":29366,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Splitting a malicious intent across three innocuous-looking agent skills bypasses current skill scanners 96% of the time.","keywords":["agent skills","LLM agents","skill scanners","multi-skill composition","chain-level attack","adversarial evasion","security scanning"],"falsifier":"Take chains generated by ColluSkill after refinement on the six scanners, freeze them, and run them against a held-out seventh scanner or an updated version of one of the six; if the average attack success rate collapses toward the 36.7% naive-split level, the headline result is an artifact of feedback-loop overfitting rather than generalizable evasion.","tokens_in":13032,"feed_emoji":"🛡️","tokens_out":13947,"duration_ms":99340,"temperature":0.7,"pith_summary":"Agent skills are becoming an installable extension mechanism for LLM agents, and this paper argues that the security scanners built to vet them inspect the wrong unit. The claim is that a single harmful intent can be decomposed into sub-skills that each look locally plausible but recover the full attack when executed in order, and the paper demonstrates this with ColluSkill, a chain-planning and scanner-feedback refinement attack that evades six representative skill scanners on 96.0% of test chains. The paper also claims the blind spot is fixable: ChainGuard, which scans a candidate skill together with skills already installed in the agent environment, cuts the attack success rate to 22.5% while still passing 99.5% of benign workflows. If the reported numbers hold, skill-ecosystem security should shift from single-skill inspection to workflow-level analysis.","feed_headline":"Skill chains slip past agent scanners 96% of the time","feed_subtitle":"A harmful workflow split into local-looking sub-skills fools isolated scanners; a context-aware scanner cuts success to 22.5%.","key_machinery":"The central object is the interdependence chain, a conceptual three-role structure $C_{\\mathrm{concept}}=(Z,E)$ with $Z=\\{z_1,z_2,z_3\\}$ and $E=\\{(z_1\\to z_2),(z_2\\to z_3)\\}$, instantiated as three ordered sub-payloads connected by artifact passing, context references, and execution handoffs. This object makes the scanner view ($\\max_i D(s_i)=0$) and the workflow view ($R_{\\mathrm{chain}}(W)\\ge\\tau_{\\mathrm{risk}}$) diverge, which is exactly the gap the attack exploits. The second load-bearing mechanism is scanner-feedback refinement: at each iteration only flagged sub-skills are rewritten from compressed scanner output, so local suspicious signals shrink while chain-level semantics are preserved. On the defense side, the equivalent mechanism is ChainGuard's candidate-centered dependency path, reconstructed by comparing a candidate skill against all installed skills in both producer and consumer directions and checking whether their composition forms a complete harmful workflow.","core_discovery":"ColluSkill's central claim is that the risk of a malicious agent workflow can be relocated from any single skill to the ordered relation between skills, and that current scanners cannot follow it there. Given an original payload $p$, an attacker LLM first infers an ordered payload plan $(p_1,p_2,p_3)$ and then a skill plan, writing each sub-payload into its own SKILL.md file $s_i$. The chain is defined as $C_{\\mathrm{inst}}=(\\{p_1,p_2,p_3\\},\\{(p_1\\to p_2),(p_2\\to p_3)\\})$, so later sub-payloads consume artifacts or context produced by earlier ones. ColluSkill submits each sub-skill to the target scanner set; if any scanner flags $s_i^{(t)}$, the attacker LLM rewrites only that sub-skill using compressed scanner feedback, so $\\max_i D(s_i)=0$ while the composed workflow still carries chain-level risk $R_{\\mathrm{chain}}(W)\\ge\\tau_{\\mathrm{risk}}$. The paper reports that this yields a 96.0% average attack success rate across six scanners, with chain planning alone contributing 68.2% and scanner-feedback refinement the rest, and that the resulting chains activate successfully on three coding agents. The paired defense claim is that a context-aware scanner reconstructing candidate-centered dependencies and capability-splitting relations against the installed-skill context reduces the attack success rate to 22.5% while admitting 99.5% of benign workflows.","pith_inferences":["The 96.0% figure measures evasion of the six queried scanners under iterative feedback; a held-out test against updated or unqueried scanners would show how much of the success is feedback-loop overfitting versus intrinsic stealth.","Because the refinement loop consumes only binary flags, defenders could poison the oracle by randomizing, rate-limiting, or watermarking scanner verdicts, making the rewrite signal unreliable.","ChainGuard reconstructs installation-time dependency paths; extending it to temporal and concurrent execution ordering could catch interleaved chains that never exist as a contiguous installation sequence.","The 3-step optimum suggests an arms race over chain length and context budgets, potentially pushing attackers toward stateful or cross-session composition that no single installation context contains."],"forward_implications":["Single-skill scanners cannot cover agent-skill security: any risk that exists only in ordered composition passes through them.","Chain planning is what makes evasion reliable: naive payload splitting reaches only 36.7% average attack success, ordered chain planning reaches 68.2%, and scanner-feedback refinement lifts it to 96.0%.","Installation-time defense is viable: blocking any one sub-skill of a chain prevents the workflow, and context-aware scanning keeps benign workflow pass rates at 99.5%.","The attack executes on real coding agents, with chain activation success between 58.5% and 92.5% across agents and model backbones, so the blind spot is practical, not just a scanner artifact.","Even under the proposed defense, 22.5% of attack chains still pass, so multi-skill composition remains an unsolved problem."],"supporting_citations":[{"why":"SkillJect, the strongest single-skill attack baseline, whose closed-loop hidden-payload approach ColluSkill extends to chains.","marker":"(Jia et al. 2026a)"},{"why":"Skill-Inject, the prompt-injection baseline that establishes skill files as an attack channel and is outperformed by ColluSkill.","marker":"(Schmotz et al. 2026)"},{"why":"SCRBench, the prior multi-skill composition benchmark whose 34.8% average ASR is the main multi-skill comparison point.","marker":"(Xie et al. 2026)"},{"why":"One of the six target scanners, representing the combined rule-plus-LLM-plus-dataflow inspection that ColluSkill must bypass.","marker":"(Cisco AI Defense 2026)"},{"why":"SkillFortify, one of the six target scanners, providing a formal supply-chain analysis baseline.","marker":"(Bhardwaj 2026)"},{"why":"Auditor, the LLM-based audit-template scanner against which ColluSkill is measured.","marker":"(Ying et al. 2026)"},{"why":"One of the six target scanners that ColluSkill must evade.","marker":"(SlowMist 2026)"},{"why":"Skill Vetter, an installation-time skill scanner among the six targets.","marker":"(fedrov 2025)"},{"why":"SkillSpector, one of the six target scanners in the evaluation.","marker":"(NVIDIA 2026)"},{"why":"Source of the interdependence-chain concept that structures the multi-skill decomposition into an ordered workflow.","marker":"(Elias 1978)"}],"fun_headline_variants":["Cross-skill workflows evade agent scanners at 96%","Chain-aware scanner blocks 96% of collusive skill attacks","Isolated skill scanners miss 96% of cross-skill attacks","Skill-splitting attacks bypass isolated scanners; chain defense works"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the attacker can keep submitting sub-skills to the exact six scanners and read their binary flags, so the reported 96.0% success rate is evasion of those specific scanners under iterative feedback rather than proof of generalizable stealth.","fun_headline_variants_meta":{"raw":{"variants":["Cross-skill workflows evade agent scanners at 96%","Chain-aware scanner blocks 96% of collusive skill attacks","Isolated skill scanners miss 96% of cross-skill attacks","Skill-splitting attacks bypass isolated scanners; chain defense works"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000498,"raw_usage":{"total_tokens":2548,"prompt_tokens":1161,"completion_tokens":1387,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":777,"completion_tokens_details":{"reasoning_tokens":1316}},"tokens_in":777,"tokens_out":1387,"duration_ms":10053,"temperature":1.0,"reasoning_tokens":1316,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:47:56.805132+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take chains generated by ColluSkill after refinement on the six scanners, freeze them, and run them against a held-out seventh scanner or an updated version of one of the six; if the average attack success rate collapses toward the 36.7% naive-split level, the headline result is an artifact of feedback-loop overfitting rather than generalizable evasion.","supporting_citations":[],"review_version":1}