{"id":"cf741e1b-8ccb-4a6a-b2b4-c955fc9b48ca","arxiv_id":"2608.11888","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Topically relevant skills can make LLM agents implement required elements incorrectly, omit them, or add expensive verification work, causing functional failures and efficiency regressions.","lead":"This study shows that agent skills, the reusable instruction packages loaded into LLM agents, can cause task failures and large cost increases. The authors built a paired-comparison framework that attributes failures to specific skills, and found 307 confirmed cases across two benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run paired comparisons without repetition or a chance baseline cannot support the 307 attributed failures; sampling noise may dominate the headline counts.","rationale":"The reader's weakest-assumption analysis correctly identifies the absence of repeated runs and variance estimates as the most load-bearing threat to the central claim. The paper's design is otherwise sensible: deterministic verifiers, paired comparisons, and trajectory-level evidence are appropriate for studying skill-induced failures. However, the causal attribution 'skill caused the failure' requires that the observed outcome difference is not due to stochastic variation in a single LLM trajectory. Since the study selects extreme pairs from a very large comparison space, the risk of false attribution is amplified. A concrete negative-control experiment and repeated-seed replication would either validate the counts or reveal that a substantial portion of the 307 cases are sampling artifacts. This concern does not contradict the paper's plausibility or its useful taxonomy; it means the empirical support is conditional on additional variance control. The reader's verdict of CONDITIONAL remains appropriate, so I recommend no change to the verdict. I agree with the reader's identification of the weakest assumption, and I would emphasize that the issue is not merely missing seeds but the absence of any chance baseline for the selection process itself.","tokens_in":16025,"tokens_out":4046,"duration_ms":43449,"concrete_test":"Rerun a stratified random sample of at least 200 target/reference pairs from Table II with 10 independent seeds per condition, holding all else fixed. For each pair, record the proportion of seeds where the target run fails and the reference run passes, and the proportion where the cost ratios exceed the paper's threshold. Apply a binomial test or require that at least 8 of 10 seeds reproduce the original outcome before counting a case as skill-induced. Also run no-skill versus no-skill and same-skill versus same-skill pairs as negative controls to estimate the chance FAIL/PASS rate. If the negative-control rate is non-negligible, or if fewer than 80% of the originally labeled cases replicate, the 307 count and the 68.8% and 62.6% shares are not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim depends on the differential paired design in Section III-A, which 'varies only the skill setup' between target and reference runs. However, each run appears to be executed once: the paper states that 'PASS and FAIL denote the deterministic verifier outcome of a single run' and no repeated seeds or variance estimates are reported anywhere. Because LLM agent trajectories are stochastic, a single no-skill PASS and a single skill FAIL do not by themselves establish that the skill caused the failure; the difference could be ordinary sampling variation. This is especially consequential because the study searches a large space of 20,664 potential paired comparisons (Table I) and then selects extreme cases where the target fails and the reference passes, or where cost ratios exceed 2.0. Under random variation alone, such extreme pairs are expected in a large comparison space. The paper's efficiency-regression threshold (min(r_tok, r_time)>1.0 and max(r_tok, r_time)>2.0) can also be triggered by token/time noise. The 'confirmed' 307 cases and the headline percentages (68.8% Task-Implementation Fault, 62.6% Excessive Procedure) inherit this attribution risk. The paper's threats-to-validity section acknowledges 'ordinary agent variance' but does not measure it or include a negative control condition, such as no-skill versus no-skill or same-skill versus same-skill repeated runs. Without such a control, the causal interpretation is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an empirical study of skill-induced failures in LLM agents. The authors propose a differential analysis framework that attributes task failures and efficiency regressions to loaded skills by comparing a target skill-guided run against a no-skill or semantically matched reference run on the same task. They instantiate the framework on SkillsBench and SWE-Skills-Bench, reporting 307 confirmed skill-induced failures, of which 125 are functional failures and 182 are efficiency regressions. From these cases, the authors derive root-cause taxonomies, finding that Task-Implementation Fault accounts for 68.8% of functional failures and Excessive Procedure accounts for 62.6% of efficiency regressions. They also build SKILLTRIAGE, an LLM-based tool that normalizes paired cases, extracts differential evidence, and produces attribution reports, reporting 88.8% subcategory agreement for functional failures and 72.5% for efficiency regressions against manual labels.","tokens_in":16226,"tokens_out":4410,"duration_ms":42707,"significance":"If the causal attribution is sound, this study makes a valuable contribution to the emerging literature on LLM agent skills. It moves beyond aggregate pass-rate comparisons to a mechanism-level analysis of how skill contents and induced trajectories cause failures and cost regressions. The differential design is a logical and appropriate way to isolate skill effects, and the manual labeling with group consensus is a strength. The taxonomy is detailed and the tooling direction is timely. However, the significance is currently tempered by the lack of repeated runs, the absence of a released dataset or code, and the internal-consistency nature of the SKILLTRIAGE evaluation. The central claim that skills can cause failures is plausible and the qualitative examples are suggestive, but the quantitative headline counts are not yet separated from sampling variation.","major_comments":[{"comment":"The causal attribution in this paper rests on paired target/reference runs in which only the skill setup is varied, yet each condition appears to be executed exactly once: Section III-A states that \"PASS and FAIL denote the deterministic verifier outcome of a single run,\" and no repeated seeds or variance estimates are reported anywhere. Because LLM agent trajectories are stochastic, a single no-skill PASS and a single skill FAIL do not by themselves establish that the skill caused the failure. The problem is amplified by the search over 20,664 potential paired comparisons (Table I) and the later selection of extreme cases where the target fails while the reference passes or cost ratios exceed 2.0; under sampling noise, such extreme pairs are expected in a large comparison space. The efficiency regression threshold in Section III-D (min(r_tok,r_time)>1.0 and max(r_tok,r_time)>2.0) can also be triggered by token/time noise. The threats-to-validity paragraph in Section VII-B acknowledges \"ordinary agent variance\" but does not measure it and does not include a negative control such as no-skill versus no-skill or same-skill versus same-skill repeated runs. Without repeated runs or a noise baseline, the headline counts (125 functional failures, 182 regressions) and the percentages in Findings 1 and 4 cannot be separated from sampling variation.","section":"III-A and III-D"},{"comment":"The evaluation of SKILLTRIAGE is an internal consistency check rather than an external validation. The taxonomy used as ground truth (Tables III and IV) was induced from the same 125 functional failures and 182 efficiency regressions that are later used to test the tool, and SKILLTRIAGE's evidence extraction is explicitly built from those taxonomy definitions (Section VI-A). The paper's claim in Section VII-B that SKILLTRIAGE \"provides an independent consistency check\" is therefore overstated: the reported 88.8% subcategory agreement for functional failures and 72.5% for efficiency regressions measure how well the tool reproduces the human labels on the very data used to construct the taxonomy. A held-out set of newly collected cases, or at least a cross-validation split, is needed to support the claim that the tool generalizes.","section":"VI-C and VII-B"},{"comment":"The dataset refinement step that reduces 665 labeled candidates to 307 analysis cases is not sufficiently specified for reproducibility. Section III-D says only that \"We remove candidates with insufficient evidence, likely verifier-induced false positives, and duplicate same-task/same-skill effects,\" without operational definitions, per-reason counts, or examples. Because the final counts are central to every finding, the paper should report how many candidates were excluded for each reason, with representative examples, and describe the consensus procedure in enough detail that the filtering could be re-run.","section":"III-D and Table II"}],"minor_comments":[{"comment":"The abstract contains a typo: \"thede factomechanism\" should read \"the de facto mechanism.\"","section":"Abstract"},{"comment":"The semantic similarity threshold (0.7), the use of all-MiniLM-L6-v2, and the top-5 candidate selection are free parameters; please report a sensitivity analysis or at least justify these exact choices.","section":"III-C"},{"comment":"Figure 2's color coding (orange and yellow boxes) is mentioned in the text but not explained in the caption; please add a legend or caption description.","section":"III-A and Figure 2"},{"comment":"The sentence in Section III-C explaining that the self-generated-skill condition is \"included in the original comparison space but not treated as a variant in the expanded public-skill comparison space\" is confusing; please clarify what this means for the reported counts.","section":"III-C"},{"comment":"The rule for assigning a case to Context Bloat versus Excessive Procedure when both effects are present (\"we assign the subcategory according to the larger cost driver observed in the trajectory\") is subjective; please provide a more operational decision rule or inter-rater agreement for this boundary.","section":"V-A"},{"comment":"For an empirical study of this kind, releasing the paired trajectories, skill files, verifier outputs, and labeling code would greatly aid reproducibility and allow independent verification of the 307 attributed failures.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a timely and relevant problem, and the differential testing design is a good starting point. However, the single-run paired comparisons are a serious threat to the central quantitative claims, and the SKILLTRIAGE evaluation lacks external validity. I would like to see the authors address the variance/noise issue, ideally with repeated runs or a negative-control condition, and provide a more transparent dataset-refinement breakdown. The paper's scope fits the journal, but I would not advise acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper gives the first systematic attempt I've seen to attribute LLM-agent failures to loaded skills, and the taxonomy it produces is genuinely useful. But the central numbers—125 functional failures, 182 regressions—come from paired runs executed once per condition, with no repeated seeds and no chance baseline. The causal claim is therefore weaker than the paper presents.\n\nWhat's new: the differential paired-comparison design (target vs. no-skill or semantically matched skill), the root-cause taxonomy (Task-Implementation Fault vs. Artifact Misplacement vs. Environment Mismatch; Excessive Procedure vs. Context Bloat vs. Dependency Resolution), and SkillTriage. The manual audit is careful: deterministic verifiers, group consensus, exclusion of ambiguous and verifier-narrow cases. The headline findings are plausible and practically relevant—seemingly relevant skills cause most functional failures, and cost regressions are dominated by excessive verification rather than prompt length alone. That matches my intuition about skill-guided agents.\n\nWhere it gets soft. Section III-A says PASS/FAIL is the deterministic verifier outcome of a single run. There is no report of repeated runs, variance, or a negative control like no-skill vs. no-skill or same-skill vs. same-skill. The study searches 20,664 potential paired comparisons and then selects extreme FAIL/PASS or cost-ratio pairs with one metric more than doubling. Under sampling noise alone you expect some extreme pairs. Without a chance baseline or multiple seeds, the 307 confirmed cases are over-stated. The threats-to-validity section acknowledges ordinary agent variance but does not measure it.\n\nSecond, the taxonomy was induced from the audited cases and then used as ground truth for SkillTriage. That makes the 88.8% and 72.5% agreement values an internal consistency check, not an external validation. The tool may be useful, but those numbers do not demonstrate independent attribution.\n\nThird, the 665-to-307 candidate exclusion is described only by summary criteria. There are no per-category exclusion counts, and no released data or code. For an empirical paper making these claims, that is a real reproducibility gap.\n\nNone of this kills the core message—skills can be harmful and the failure modes are worth cataloguing. But the paper should report repeated runs or a noise baseline, release the dataset, and present SkillTriage as a consistency check rather than validation. I would send this to peer review, not desk reject, because the problem is important and the taxonomy will likely be a reference point. A competent referee should ask for the missing control and the data.","headline":"A thoughtful empirical study of skill-induced agent failures with a plausible taxonomy, but the headline 307 cases rest on single-run paired comparisons, so the causal counts are not yet established.","tokens_in":16803,"tokens_out":2915,"would_cite":true,"duration_ms":29540,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agent skills can be harmful: even topically relevant skills frequently induce failures and cost regressions in LLM agents.","keywords":["agent skills","skill-induced failures","LLM agents","differential testing","root-cause taxonomy","functional failures","efficiency regressions","cost-aware skill reuse"],"falsifier":"Take a random sample of the 125 functional-failure pairs and re-run each condition, say 20 times, with the same model, verifier, and skill setup. If the no-skill condition and the skill condition each produce a mix of PASS and FAIL across seeds, or if the FAIL/PASS contrast reverses in a substantial share of seeds, the skill-induced attribution for those cases is not stable and the 307-count conclusion would need revision.","tokens_in":15777,"feed_emoji":"⚠️","tokens_out":6246,"duration_ms":59814,"temperature":0.7,"pith_summary":"The paper claims that loading a reusable agent skill into an LLM agent's context can cause the agent to fail tasks or become substantially more expensive, even when the skill is topically relevant to the task. It establishes this with paired runs that keep the task, verifier, model, and environment fixed and vary only the skill setup, yielding 307 confirmed skill-induced failures: 125 functional failures and 182 efficiency regressions. The dominant functional failure mechanism is Task-Implementation Fault, where a seemingly relevant skill leads the agent to fill a required element incorrectly (46 cases) or omit it (36 cases), together 82 of 125 cases. The dominant cost mechanism is Excessive Procedure, where skills turn optional exploration, construction, and verification into mandatory work, accounting for 114 of 182 regressions, with excessive verification alone at 67 cases. If these attributions hold, skill reuse needs compatibility checks, cost prediction, and guarded task constraints rather than blind trust in topical relevance.","feed_headline":"Agent skills backfire in 307 test cases, mostly on-topic","feed_subtitle":"Paired runs show relevant skills distort required implementations and turn optional checks into costly mandatory work.","key_machinery":"The mechanism is differential paired execution, adapted from differential testing. For every audited skill, the paper runs the same task under the skill and under a reference setup, either no skill or a semantically matched public skill, while holding task, verifier, agent framework, model, and repository state fixed. A functional failure is recorded when the skill run fails and the reference run passes; an efficiency regression is recorded when both pass but token use and execution time both increase and at least one more than doubles. This contrast turns the loaded skill into the only plausible causal variable. The companion machinery is a pair of root-cause taxonomies: functional failures split into Applicability Mismatch, Environment Mismatch, Task-Implementation Fault, and Artifact Misplacement; efficiency regressions split into Context Bloat, Excessive Procedure, and Dependency Resolution. The triage tool operationalizes these taxonomies as differential-evidence checklists for automated attribution.","core_discovery":"The central discovery is that skills harm agents mainly by being too trusted in the wrong way, not by being obviously irrelevant. In the functional-failure taxonomy only 2 of 125 cases are Applicability Mismatch; 86 cases are Task-Implementation Fault, where the agent correctly recognizes the topic but implements a task-required element with the wrong API, value, scaling, or format, or leaves that element absent. A further 24 cases are Artifact Misplacement, where the agent writes a plausible artifact at a repository-convention location rather than the task-specified path, and 13 are Environment Mismatch. On the efficiency side, the paper shows regression cost is not explained by prompt length alone: when context overhead appears it is almost always the mandatory skill-body text (43 of 46), but the larger class is Excessive Procedure (114 of 182), dominated by excessive verification (67) and heavy implementation pipelines (30). The paper also demonstrates that a taxonomy-guided triage tool can reproduce the manual category for 117/125 functional failures and 145/182 efficiency regressions, showing the classification is operationalizable.","pith_inferences":["The single-run design means the reported counts are likely an upper bound; re-running with multiple seeds would probably shrink the 307 total, though the rough shape of the taxonomies could survive.","The same differential-pair method could be applied to other context injections, such as retrieved documentation, system prompts, or example packs, to test whether implementation distortion is unique to skills or generic to added guidance.","A cheap screening experiment suggested by the paper: take a skill with Excessive Verification and replace its exhaustive checklist with a conditional 'verify only if the change is risky' instruction; if cost drops without new failures, it confirms the mandatory-work mechanism."],"forward_implications":["Agent platforms should treat task-specified paths, integration points, and environment state as guarded constraints, because Artifact Misplacement and Environment Mismatch together account for 37 of 125 functional failures.","Skill authors should separate mandatory task requirements from reusable examples and defaults, and lazy-load long skill bodies, since mandatory body text causes 43 of 46 context-overhead regressions.","Agents need budget-aware verification policies: excessive verification is the single largest regression source, so validation checklists become mandatory work in 67 of 182 cases.","Skill selection should estimate induced procedural cost, not just prompt length, because Excessive Procedure accounts for 114 of 182 regressions, more than double Context Bloat.","Automated triage at 88.8% exact-subcategory agreement for functional failures and 72.5% for efficiency regressions makes continuous screening of skill updates feasible."],"supporting_citations":[{"why":"Supplies the 84-task benchmark, its deterministic verifiers, and the no-skill/curated-skill comparison settings that the study's functional-failure and regression pairs are built from.","marker":"[1]"},{"why":"Supplies the 490 repository-based software-engineering task instances and curated-skill settings that form the second benchmark in the paired analysis.","marker":"[2]"},{"why":"Provides the differential-testing idea that motivates the target/reference pseudo-oracle design used to attribute failures to skills.","marker":"[7]"},{"why":"Defines agent skills as SKILL.md packages loaded into context, the object whose harmful effects the paper studies.","marker":"[8]"},{"why":"Identifies the agent runtime used for every paired execution in the study.","marker":"[9]"},{"why":"Identifies the language model used for every paired execution, fixing the model across target and reference runs.","marker":"[10]"}],"fun_headline_variants":["Most agent-skill failures stem from relevant skills","Skill-induced faults: relevance isn't safety","On-topic skills cause 86 implementation failures","Excessive verification drives skill cost regressions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attribution assumes that a single PASS in the reference run and a single FAIL or single costly run in the skill run reflect the skill's causal effect rather than ordinary sampling variation in the language model, because the paper does not report repeated runs or variance estimates.","fun_headline_variants_meta":{"raw":{"variants":["Most agent-skill failures stem from relevant skills","Skill-induced faults: relevance isn't safety","On-topic skills cause 86 implementation failures","Excessive verification drives skill cost regressions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1482,"prompt_tokens":1072,"completion_tokens":410,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":688,"completion_tokens_details":{"reasoning_tokens":353}},"tokens_in":688,"tokens_out":410,"duration_ms":5002,"temperature":1.0,"reasoning_tokens":353,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:22:40.468030+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 125 functional-failure pairs and re-run each condition, say 20 times, with the same model, verifier, and skill setup. If the no-skill condition and the skill condition each produce a mix of PASS and FAIL across seeds, or if the FAIL/PASS contrast reverses in a substantial share of seeds, the skill-induced attribution for those cases is not stable and the 307-count conclusion would need revision.","supporting_citations":[{"cited_title":"OpenCode: The open source coding agent,","cited_arxiv_id":null,"evidence_quote":"Identifies the agent runtime used for every paired execution in the study."},{"cited_title":"Introducing claude opus 4.6,","cited_arxiv_id":null,"evidence_quote":"Identifies the language model used for every paired execution, fixing the model across target and reference runs."},{"cited_title":"Differential testing for software,","cited_arxiv_id":null,"evidence_quote":"Provides the differential-testing idea that motivates the target/reference pseudo-oracle design used to attribute failures to skills."},{"cited_title":"Equipping agents for the real world with agent skills,","cited_arxiv_id":null,"evidence_quote":"Defines agent skills as SKILL.md packages loaded into context, the object whose harmful effects the paper studies."}],"review_version":1}