{"id":"00b5def1-dd64-422f-bf98-3bccec1222f6","arxiv_id":"2608.03485","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An AI-skill safety scanner that runs skills with and without decoy resources and compares behavior to detect hidden, conditional malicious actions.","lead":"The paper describes SkillSentry, a safety checker that runs AI-agent skills in simulated environments filled with fake files and credentials to see if a skill does things its description does not mention. It reports catching nearly all malicious skills in benchmark tests while flagging far fewer safe ones than existing scanners.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single no-skill control run per seed is statistically unsupported; LLM stochasticity could drive the differential and all headline metrics.","rationale":"Reading the paper in good faith, the central claim is that the scanner's causal attribution is reliable. The method in §3.3 computes a differential between one skill-enabled and one no-skill execution. Because the base agent is an LLM, its behavior is stochastic, and a single control run cannot separate sampling variation from skill-caused effects. The paper gives no evidence of repeated runs, seed sweeps, or confidence bounds; the phrase 'For each test and random seed' does not state how many seeds were used, and the results do not report variance. The case studies show replay only for confirmed positive findings, not for controls. This concern is load-bearing: the entire verdict pipeline consumes the differential, so both the recall and the false-positive rate are contingent on the assumption that one control run is representative. The reader's weakest_assumption identifies exactly this issue, and I agree. The concrete test I propose would measure verdict stability across seeds and produce a confidence interval for FPR, directly testing the assumption. Since the reader already made a conditional acceptance based on this concern, my stress test does not change the verdict.","tokens_in":13350,"tokens_out":4843,"duration_ms":48762,"concrete_test":"Re-run the four-execution causal bundle on a stratified sample of 50 clean and 50 harmful skills from SkillTrustBench with 10 distinct seeds (e.g., 0–9) per skill, each with fresh temperature>0 sampling. Count how many skills flip verdict (Safe/Unsafe) across seeds, and compute the standard error of SkillSentry's FPR and Recall. If any clean skill is flagged in ≥1 seed but the single reported run was 'safe,' or if the FPR 95% CI includes >5%, the single-run differential is not reliable and the headline FPR/Recall need re-estimation with repeated controls.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 defines the causal bundle as one skill-enabled and one no-skill execution per world. The baseline differential retains any effect that 'appears only with the skill.' With a single control run, this is not a causal test: LLM agents are stochastic, and the paper does not report temperature, seed distribution, or repeated runs. A no-skill run that randomly omits a side effect—or a skill run that randomly performs one—creates a spurious differential. Every verdict (Safe/Unsafe/Unresolved) is a function of these differentials, so the reported 4.15% FPR and 96.12% Recall on SkillTrustBench and 99.50% Recall on HarmfulSkillBench inherit this unsupported assumption. The ablation 'Without Baseline Differential' (Table 3) shows the component matters, but still uses a single control, so it does not resolve the sampling issue. Without error bars or repeated seeds, the central claim that the scanner 'detects nearly all harmful skills' while keeping FPR below 5% is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SkillSentry is a pre-deployment dynamic safety-testing framework for LLM-agent skills. It infers a capability contract from the skill's advertised function, uses source analysis to generate test plans, constructs ordinary and honey worlds with controlled decoy resources, and compares skill-enabled vs. no-skill executions to attribute observed effects to the skill. The paper evaluates against seven static/semantic scanner configurations on three standard benchmarks, an evasion suite, and two injection benchmarks, reporting high recall/F1 and low FPR, with ablations attributing gains to each component. The code is released.","tokens_in":13582,"tokens_out":3492,"duration_ms":42521,"significance":"If the reported results are reliable, SkillSentry makes a useful contribution: it advances skill safety testing from artifact inspection toward execution-grounded, causally attributed auditing. The controlled-experiment framing (paired runs, matched worlds, completed-effect verification, explicit 'unresolved' outcomes) is a sound design idea, and the ablation study gives partial evidence that each component matters. The public code release and the use of replayable execution evidence are strengths. However, the evaluation's statistical grounding is currently too weak to support the headline claims, and the baseline comparison omits the most relevant dynamic-detection work cited in the paper itself.","major_comments":[{"comment":"The causal-attribution method rests on a single no-skill control run per world per seed, yet LLM agents are stochastic. Section 3.3 mentions a 'random seed' but no number of seeds, temperature, or repeated executions is reported; all headline metrics are point estimates. A single control run can produce a spurious skill/no-skill differential simply because one of the two trajectories omitted or performed an effect by chance, and every verdict inherits this. The 'Without Baseline Differential' ablation in Table 3 does not fix this, since it still uses one control per condition. Please run repeated seeds/runs and report distributions, confidence intervals, or at least the number of repetitions; without this, the claims of 99.50% Recall and sub-5% FPR are not statistically supported.","section":"§3.3, Tables 1–3"},{"comment":"The evaluation compares against seven configurations from three scanner families (SkillSpector, Cisco Skill Scanner, Skill Vetter), all of which are static or one-shot semantic reviewers. Yet the related work cites Ji et al. 2026, 'Cloak and Detonate,' which is a dynamic detection method for agent skill malware. Omitting the closest dynamic baseline makes the 'strongest baselines' comparison incomplete and weakens the claim of substantial improvement. Please add this baseline (or a comparable dynamic detector) to Tables 1–2, or explicitly justify its exclusion with a concrete reason.","section":"§4.1, Related Work §2.2/§2.3"},{"comment":"The same model family (DeepSeek-V4-Pro) is used for test generation and evidence judging. The authors argue that semantic critic and dynamic verifier have distinct roles, and Figure 4 shows matched generator-judge pairs outperform cross-model pairs. However, this does not address the risk that a false premise from the generator propagates to a judge with the same inductive biases. A concrete test would be to evaluate a sample of verdicts with a judge from a different model family or with human adjudication, and report agreement. This is a correctness-risk concern rather than a circularity claim, but it directly affects the reliability of the reported verdicts.","section":"§3.4, Figure 4"}],"minor_comments":[{"comment":"The phrase '80.07% for the strongest baselines' is ambiguous because the strongest baseline differs per attack. Specify which baseline is used for each attack and whether the average is over the per-attack best baseline or the same configuration.","section":"§4.3"},{"comment":"No dataset sizes are given for HarmfulSkillBench, SkillTrustBench, or MalSkillBench. Reporting sample sizes per benchmark would help interpret the precision/recall/FPR percentages, especially for HarmfulSkillBench where only Recall is reported.","section":"§4.1"},{"comment":"The abstract states '96.26% average F1 on standard benchmarks'; this number appears to be the average of the two mixed benchmarks. Clarify that HarmfulSkillBench is excluded because it has no clean samples.","section":"Abstract and §4.2"},{"comment":"The phrase 'random seed' appears only in this section and is never operationalized in the experiments. Either specify the number of seeds used or remove the phrase to avoid implying variability.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the paper is well written, but the single-run evaluation and the omitted dynamic baseline are load-bearing. I would not reject on these grounds because the statistical issue is fixable with additional runs and the dynamic-baseline comparison can be added. However, if the authors cannot provide repeated-seed results showing stable differentials, the central claims should be substantially weakened. I also encourage the editor to consider whether the paper's scope overlaps with the concurrently cited dynamic-detection work and whether a direct comparison is necessary for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nYou should know two things about SkillSentry. First, it's a genuinely useful design: a dynamic scanner that builds contract-adaptive honey worlds and compares skill-enabled runs with matched no-skill controls, grounding every finding in source code and verified execution traces. Second, the headline numbers—99.50% recall, FPR below 5%—rest on a single run per condition per seed, which is not enough to separate LLM stochasticity from skill-caused effects. That is a load-bearing weakness, not a cosmetic one.\n\nWhat's new is the combination. Static localization, sandboxed execution, decoys, and differential comparison all exist, but the paper is the first to tie them into an audit workflow with a capability contract, adaptive honey resources, and a four-execution bundle. The ablation study is informative: removing the no-skill differential raises FPR from 3.24% to 10%, removing the dynamic verifier to 34.09%. That shows the components matter. The paper also handles unresolved cases honestly, reports efficiency costs, and releases code. Credit where it's due.\n\nThe soft spots. The causal bundle in Section 3.3 runs one no-skill control per seed. LLM agents are stochastic; a single control can't establish what 'appears only with the skill.' The paper doesn't report temperature, seed distribution, or repeated runs. Without error bars, the 4.15% FPR and 96.12% recall on SkillTrustBench are point estimates from a lucky or unlucky draw. The ablation 'Without Baseline Differential' shows the component matters, but it still uses a single control. That's a fixable flaw—run N seeds, report variance—but as it stands the central claim isn't established. Second, the paper cites 'Cloak and Detonate' (Ji et al. 2026) but doesn't include it as a baseline in the evasion experiments. That's a conspicuous omission for a paper claiming robustness against scanner evasion. Third, the capability contract is inferred by the same model family that judges evidence. They split critic and verifier roles, which helps, but a shared backbone still carries a self-consistency risk.\n\nOverall, the design is sound and the evaluation is more transparent than most in this space, but the statistical foundation is too weak for the strength of the claims. This should go to review and be accepted only if the authors add repeated-seed runs, confidence intervals, and the missing dynamic baseline.\n\nTake it to reading group? Probably worth a discussion.","headline":"SkillSentry is a genuinely useful dynamic scanner with a well-designed attribution pipeline, but the headline numbers rest on a single no-skill control run per seed, so the recall/FPR claims are not yet established.","tokens_in":14051,"tokens_out":2473,"would_cite":true,"duration_ms":28499,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a pre-deployment audit that runs each agent skill in matched environments, with and without the skill enabled, and adds minimal task-irrelevant decoy resources, detects 99.50% of harmful skills while keeping false posi","keywords":["agent skills","LLM agents","dynamic safety testing","honey worlds","capability drift","causal attribution","skill injection","scanner evasion"],"falsifier":"Re-run SkillSentry on SkillTrustBench many times with different seeds—say 20 seeds per skill—and record whether each skill gets the same verdict. If verdicts flip between runs because the no-skill control occasionally performs the flagged action, or because the skill-enabled run occasionally fails to trigger it, the single-bundle attribution assumption fails. A more targeted version: construct skills that do nothing, run the bundle repeatedly, and count how often a completed 'skill-only' effect still appears; any nontrivial rate would falsify the claim that the differential isolates skill caus","tokens_in":13277,"feed_emoji":"🧪","tokens_out":7778,"duration_ms":77535,"temperature":0.7,"pith_summary":"SkillSentry's central claim is that harmful agent skills are best detected by running a controlled experiment around each skill rather than by inspecting its code. The auditor infers a capability contract from the skill's advertised function, builds two matched simulated environments—one ordinary and one containing a minimal task-irrelevant decoy resource—and executes the same task once with the skill enabled and once without it. A violation is reported only when a completed tool effect appears solely in the skill-enabled run, is grounded in the package's own source, and falls outside the advertised capability. On benchmarks this design reports 99.50% recall on the harmful-only set and 96.08% and 96.43% F1 on mixed sets with false-positive rates below 5%; under four semantics-preserving evasion attacks it keeps 92.95% average F1 versus 80.07% for the strongest baseline. If the approach holds, skill scanning becomes evidence-grounded behavioral testing, with the principal cost being computation and API tokens rather than detection accuracy.","feed_headline":"Decoy worlds catch 99.5% of harmful agent skills","feed_subtitle":"Running each skill with and without it enabled flags only effects the skill itself causes—false positives stay below 5%.","key_machinery":"The central object is the adaptive honey world and the four-execution causal bundle. A honey world is an LLM-simulated environment seeded with the smallest task-irrelevant decoy resource a suspected behavior needs to fire; the causal bundle runs the same task in ordinary and honey worlds, each once with the skill and once without, so that the difference isolates skill-caused effects. This machinery makes latent code paths reachable and attributes observed effects to the skill rather than to the task or base agent.","core_discovery":"The discovery is that a skill scanner can treat each suspicious behavior as a testable hypothesis and decide safety by executing the skill under controlled conditions. SkillSentry derives a capability contract from the skill's advertised function, converts source-supported candidate behaviors into test plans, and creates an adaptive honey world whose only difference from an ordinary world is a uniquely marked, task-irrelevant decoy resource that gives hidden behavior an opportunity to fire. Each test runs a four-execution causal bundle—ordinary and honey worlds, each with and without the skill—so effects caused by the task or base agent are subtracted out by the no-skill control. The retaine","pith_inferences":["The reported numbers come from one execution per seed; because LLM agents are stochastic, the causal bundle could be sensitive to seed choice. A natural extension is to repeat each bundle across several seeds and report variance or confidence intervals for recall and FPR.","The detection ceiling is set by the test generator's ability to infer plausible behaviors from source; a skill whose trigger the generator cannot imagine from the package content would stay invisible even in a honey world. This points to combining source-guided planning with fuzzing over environment states.","The decoy approach could generalize beyond skills to other plugin- or tool-based agent extensions, such as MCP servers, by treating each external component as the component under test and applying the same differential-execution design.","Because the verdict comes from replayed observable effects, the framework's outputs are audit artifacts rather than binary judgments; this could support downstream review processes, policy logging, and adversarial retesting as skills update."],"forward_implications":["A scanner that requires a completed, skill-caused, out-of-contract effect can report high recall without drowning in false positives: FPR falls below 5% on both mixed benchmarks while recall stays near or above 94%.","Rewriting a harmful skill's instructions or code layout does not hide it, because the test triggers the capability itself; SkillSentry keeps 92.95% average F1 against four semantics-preserving attacks, versus 80.07% for the strongest baseline.","The same audit transfers to capability injection, where a clean skill is modified to carry hidden behavior: 86.32% F1 with 3.94% FPR on Skill-Inject and SkillJect.","The pipeline is backbone-independent: every tested LLM keeps HarmfulSkillBench recall at or above 97%, though contract interpretation is the most model-sensitive stage.","Using the method costs more than static scanning—119 seconds per skill in the sampled run—but the extra computation is what produces replayable evidence and the low false-alarm rate."],"supporting_citations":[{"why":"Supplies HarmfulSkillBench, the harmful-only benchmark used for the recall headline.","marker":"Jiang et al. 2026"},{"why":"Provides SkillTrustBench, a mixed clean/harmful benchmark on which F1 and FPR are measured.","marker":"Tencent Zhuque Lab 2026"},{"why":"Provides MalSkillBench, the second mixed benchmark used for F1 and FPR comparison.","marker":"Guo et al. 2026a"},{"why":"Defines POISE, a position-aware evasion attack the method is tested against.","marker":"Hao et al. 2026a"},{"why":"Contributes SkillCloak-Structural and SkillCloak-SFS, evasion transformations used in the robustness table.","marker":"Ji et al. 2026"},{"why":"Provides the VulMask-style transformation used as the fourth evasion attack.","marker":"Lin and Yu 2026b"},{"why":"Supplies SkillSpector, the scanner family used as a baseline in the detection and evasion tables.","marker":"Paz et al. 2026"},{"why":"Supplies the Cisco Skill Scanner configurations used as baselines.","marker":"Cisco AI Defense 2026"},{"why":"Supplies Skill Vetter, the strongest baseline overall, used for comparison and case studies.","marker":"spclaudehome 2026"},{"why":"Introduces honeywords, the decoy-token concept that adaptive honey worlds generalize.","marker":"Juels and Rivest 2013"}],"fun_headline_variants":["Honey worlds unmask skills that turn harmful mid-task","Decoy resources expose hidden agent-skill risks at 96% F1","Skill scanner uses decoy worlds to catch 99.5% of attacks","Adaptive decoys isolate skill-only harm in agent tests","Test suspicious skills in fake worlds to confirm danger"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The causal claim rests on a single no-skill control run per seed: if LLM-agent stochasticity makes that one control run differ by chance, effects could be misattributed to the skill and the reported recall and false-positive rates would not be stable.","fun_headline_variants_meta":{"raw":{"variants":["Honey worlds unmask skills that turn harmful mid-task","Decoy resources expose hidden agent-skill risks at 96% F1","Skill scanner uses decoy worlds to catch 99.5% of attacks","Adaptive decoys isolate skill-only harm in agent tests","Test suspicious skills in fake worlds to confirm danger"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000597,"raw_usage":{"total_tokens":2631,"prompt_tokens":750,"completion_tokens":1881,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":1804}},"tokens_in":494,"tokens_out":1881,"duration_ms":13766,"temperature":1.0,"reasoning_tokens":1804,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T18:07:13.083525+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run SkillSentry on SkillTrustBench many times with different seeds—say 20 seeds per skill—and record whether each skill gets the same verdict. If verdicts flip between runs because the no-skill control occasionally performs the flagged action, or because the skill-enabled run occasionally fails to trigger it, the single-bundle attribution assumption fails. A more targeted version: construct skills that do nothing, run the bundle repeatedly, and count how often a completed 'skill-only' effect still appears; any nontrivial rate would falsify the claim that the differential isolates skill caus","supporting_citations":[],"review_version":1}