{"id":"cdd5e450-d60f-4eec-ab3c-fb14ced8e326","arxiv_id":"2608.08037","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SkillSmith automatically constructs and evolves skills for a frozen local agent using cloud-agent exploration and local failure feedback, lifting Qwen3.6-27B pass rates from 36.3% to 78.6% on AppWorld-Normal.","lead":"This paper presents SkillSmith, a system that automatically builds and refines skill files for a local AI agent, using a cloud AI to explore tasks and fix failures. The claimed payoff is that a small local model can nearly match large cloud models on everyday agent tasks, with lower cost and better privacy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The generated Skill may encode training-task-specific answers rather than transferable environment knowledge, because EXECUTEREFLECT feeds the Cloud Agent evaluator feedback and asks for a 'more efficient solution path' per task; no leakage audit or ablation removing this channel is reported.","rationale":"The reader's conditional verdict is appropriate, and my concern reinforces it rather than overturning it. I agree with the reader that the Cloud Agent's reports are an unverified source, but I would sharpen the failure mode: the report template includes evaluator feedback and a 'more efficient solution path,' which can leak expected outcomes into the Skill. This is more damaging than hallucination because it is consistent with high training and test performance while undermining the claim that the Skill contains transferable environment knowledge. The paper's own motivation (P1) warns that directly prompting the Cloud Agent can hallucinate rules; the Explore-and-Reflect step is meant to ground the Skill in execution, but without an audit it also grounds it in task answers. The ablation in Table V shows Creation is the dominant contributor, which is exactly where the leakage channel operates. The evolution stage only sees failures on training tasks and cannot remove an incorrect-but-untriggered rule or a task-specific snippet that helps on both train and test. A leakage audit or an outcome-free rerun would settle the question. Because the paper reports no code, data, or generated Skill artifacts, this check cannot currently be performed by reviewers, which is another reason to keep the verdict conditional. If the check later passes, the central claim stands; if it fails, the magnitude of the reported gains and the transferability claim need to be revised.","tokens_in":15782,"tokens_out":9617,"duration_ms":104779,"concrete_test":"Release the generated SKILL.md and all reference files, and run a named-entity and literal-value overlap audit against the training task instructions and expected outcomes (e.g., email addresses, account names, dollar amounts, dates, phone numbers). If any unique training-task-specific values or exact operation sequences appear in the Skill, remove those snippets and re-evaluate the test set with the frozen Local Agent and unchanged evolution settings. If test PR drops materially after removal, the reported gains are partly attributable to memorized training-task answers; if it does not drop, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SkillSmith distills reusable environment knowledge (rules, procedures) into a Skill that transfers to unseen tasks. The most load-bearing unexamined assumption is that the Reflection Reports produced in Sec. III-C are a source of general knowledge rather than a channel for memorizing training-task solutions. Algorithm 1 line 2 calls EXECUTEREFLECT(G_c, D_train); per Sec. III-C, for each training task the Cloud Agent receives the task instruction, its execution trace, the evaluator feedback, and is prompted to summarize 'a more efficient solution path.' The evaluator feedback for AppWorld/WorkBench can include the expected outcome y* (Eq. 2), so the Cloud Agent is effectively shown the answer while writing the report. The downstream CLUSTER/DISTILL/MERGE prompts (Fig. 5) filter by frequency and cross-cluster generality, but they impose no explicit check that SKILL.md and reference files are free of task-specific identifiers (entity names, amounts, email addresses, exact step sequences from training tasks). If such content survives, test-set pass rates are inflated because test tasks are drawn from the same environment with overlapping entities and procedures, and the cross-backbone transfer results (Table VI) would reflect memorized task solutions rather than environment knowledge. No inspection of the generated Skill, no leakage audit, and no ablation with outcome-free feedback is reported, so this alternative explanation is not ruled out. This is the inverse of the reader's hallucination concern: the risk is not that the Cloud Agent's reports are wrong, but that they are too specific and non-transferable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"SkillSmith is a Cloud–Local collaboration framework that automatically constructs a Skill—a main Markdown manual with progressive-disclosure reference files—from Cloud Agent exploration of training tasks, then iteratively evolves it using Local Agent execution feedback, while keeping the Local Agent backbone frozen. The paper motivates the design with a failure analysis attributing most Local Agent failures to missing environment knowledge, formalizes offline Skill learning as black-box optimization over natural-language Skills, and evaluates on AppWorld and WorkBench. The reported results show large pass-rate gains over prompt-, memory-, and skill-based baselines, a reduction in average actions per task, lower online API cost than Cloud Agent serving, and transfer to other SLM backbones.","tokens_in":16076,"tokens_out":6023,"duration_ms":61869,"significance":"The contribution is potentially significant for practical local deployment of LLM agents. The framework addresses a real gap: existing skill-based methods either require expert-authored initial Skills or are not verified and corrected from execution feedback. The paper includes a stage-level ablation showing Creation dominates the gain, an offline-cost analysis from API logs, and a cross-backbone transfer experiment, all of which are useful. The main strength is the explicit role-separated pipeline with concrete prompts and a clear cost model. However, the evidence that the generated Skill encodes reusable environment knowledge rather than memorized training-task answers is incomplete, and the absence of variance estimates weakens the quantitative claims. If the leakage concern is resolved, this would be a strong contribution.","major_comments":[{"comment":"The most load-bearing unexamined assumption is the leakage channel from evaluator feedback into the Skill. In Explore and Reflect, each Cloud Agent trajectory is summarized into a Reflection Report that includes 'a more efficient solution path' after the agent has received evaluator feedback containing the expected outcome y* (Eq. 2). The downstream CLUSTER/DISTILL/MERGE prompts (Fig. 5) filter by frequency and cross-cluster generality but impose no explicit check that SKILL.md and the reference files are free of task-specific identifiers, amounts, or exact step sequences from training tasks. Since the test set is drawn from the same environment, memorized training-task content could inflate the held-out pass rates in Table III. Please add (i) an audit of the generated Skill for task-specific content, (ii) an ablation that removes outcome information from reflection (e.g., success/failure only, without y*), and (iii) a comparison of the Skill's high-frequency steps against the actual training task instructions to show they encode environment procedures rather than particular solutions.","section":"Sec. III-C, Algorithm 1, Eq. (2)"},{"comment":"All headline results are single numbers with no error bars, confidence intervals, or significance tests. The Local Agent and Cloud Agent are stochastic systems, so the reported margins (e.g., 11.3-26.2 points over the strongest baseline in Table III, and cross-backbone deltas in Table VI) may reflect run-to-run variation. Please report means and variances over multiple independent runs, or justify determinism of the evaluation protocol, and include paired significance tests where comparisons are made on the same task sets.","section":"Table III and Table VI"},{"comment":"The diagnostic claim that 91.7% of Local Agent failures are caused by missing environment rules or operation procedures is based on only 36 sampled trajectories. This sample is too small to support a precise attribution, and the manual classification has no reported agreement or confidence interval. Please report the sampling procedure, a larger sample, and at least a bootstrap confidence interval for the 91.7% figure, or otherwise temper the claim.","section":"Table II"},{"comment":"The Trace2Skill baseline is initialized from 'a initial skill derived from the description of datasets,' which is likely weaker than the intended initialization of a skill-evolution method. Since Trace2Skill is the strongest baseline on WorkBench (62.8 vs. 89.0) and one of the closest on AppWorld-Challenge, this setup may understate a key competitor. Please initialize Trace2Skill with the same automatic Skill Creation output as SkillSmith, or with an expert-authored initial Skill, and report results for both.","section":"Sec. IV-A, Trace2Skill"},{"comment":"The final Skill is selected by the best empirical pass rate on the training set (Algorithm 1, lines 17-19), and the evolution objective in Eq. (6) is also the training pass rate. This creates a risk of overfitting to D_train, which is especially relevant given the leakage channel in Major Comment 1. Please use a held-out validation split for model selection, or report the gap between training and test pass rates across Skill versions, to demonstrate that the reported test gains are not artifacts of selection on training tasks.","section":"Sec. III-B, Algorithm 1"}],"minor_comments":[{"comment":"The abstract and introduction contain typos including 'depolying', 'depolyment', and 'adotpts'; please proofread the manuscript.","section":"Abstract and Sec. I"},{"comment":"Table I reports Qwen3.6-27B versus Claude Opus 4.5 values from the Qwen technical report; since these are not independently verified, please state this clearly and avoid over-claiming 'comparable' from a single third-party source.","section":"Table I"},{"comment":"The y-axis of Figure 8 starts at 55, which visually exaggerates the evolution gains; use a full axis or an explicit axis break.","section":"Figure 8"},{"comment":"The implementation details state that Skill Evolution 'runs for at five rounds'; specify R=5 and report whether the training/test split is fixed across all baselines.","section":"Sec. IV-A"},{"comment":"Eq. (1) defines Skill S=(M,R,C), but the budget L_max and reference file cap of 250 lines are introduced later; please define the feasibility set S_Lmax explicitly and state whether the cap applies to each reference file or the total reference content.","section":"Sec. II-C, Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my take on the SkillSmith paper. The core idea is genuinely new: instead of assuming an expert-written initial skill, SkillSmith has a cloud agent explore training tasks, reflect on outcomes, cluster those reflections, and distill them into a skill that a frozen local SLM then loads. That removes a real bottleneck. The cost analysis is useful—one-time offline spending versus per-task cloud calls—and the cross-backbone transfer experiment, even if preliminary, is the right question to ask. The related work is well covered and the positioning against Trace2Skill and EvoSkill is accurate.\n\nBut I would not trust the headline numbers yet. Every result in Table III is a single run, with no error bars or significance tests. A 42-point gain is large, but we have no sense of variance. The diagnostic claim that 91.7% of failures come from missing environment knowledge rests on 36 manually inspected trajectories. And the Trace2Skill baseline is initialized from a dataset description, which is not the setting that method was designed for; calling that 'unfair' understates it.\n\nThe bigger problem is the leakage channel that the stress-test note identifies. The reflection reports are written after the cloud agent sees evaluator feedback, which includes the expected outcome. The prompt asks for a 'more efficient solution path' per task. Clustering and distillation might filter some task-specific detail, but there is no audit of the final skill and no ablation that removes outcome feedback. If the skill encodes training-task-specific strings—entity names, amounts, step sequences—test accuracy will be inflated, because the test tasks come from the same environment with overlapping entities. The paper's held-out split does not rule this out; it is the contents of the skill, not the evaluation protocol, that need scrutiny. This is the load-bearing issue, and the paper does not address it.\n\nMy bottom line: this is a promising framework and a paper worth arguing with, but the evidence as presented is not sufficient to support the claims. I would send it to peer review—the novelty justifies referee time—but the authors need to release code and data, report repeated runs, fix the baseline, and run a leakage audit or a no-feedback ablation. Until then, I would not cite the numbers.","headline":"Promising but unproven: SkillSmith's automatic skill construction is a real step forward, but the evidence is single-run and the skill may be memorizing training-task answers rather than generalizable knowledge.","tokens_in":16600,"tokens_out":3994,"would_cite":false,"duration_ms":38095,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Local agents can match cloud-agent task effectiveness when missing environment knowledge is supplied as automatically constructed Skills.","keywords":["agent skills","local deployment","environment knowledge","skill evolution","small language models","cloud-local collaboration","progressive disclosure","task effectiveness"],"falsifier":"A decisive test: inject one fabricated environment rule into the Cloud Agent's exploration prompt before Skill Creation, then check whether the final Skill propagates that rule and whether the Local Agent follows it on tasks where the rule is false.","tokens_in":15588,"feed_emoji":"🤖","tokens_out":8132,"duration_ms":77677,"temperature":0.7,"pith_summary":"SkillSmith claims that the gap between locally deployed agent models and cloud-hosted frontier models is mostly a knowledge gap, not a reasoning gap: small local models already match large cloud models on tool use and instruction following, but lack environment-specific rules and operation procedures. The paper shows that this missing knowledge can be packaged into a Skill document and supplied to a frozen local model without fine-tuning. In the proposed two-stage pipeline, a cloud agent first explores representative tasks and reflects on them to create an initial Skill, then local execution failures are clustered and analyzed to evolve the Skill in bounded edits. With such a Skill, a 27-billion-parameter local model reaches 78.6% pass rate on AppWorld-Normal, 74.9% on AppWorld-Challenge, and 89.0% on WorkBench, cutting average actions per task from 36.1 to 9.9 and transferring to other local backbones without rebuilding. If true, this makes private, low-cost local deployment viable for daily agent tasks that previously required cloud LLM calls.","feed_headline":"Auto-built skills lift local agents to cloud-level tasks","feed_subtitle":"A 27B local model goes from 36% to 79% pass rate using a reusable Skill built once by a cloud agent.","key_machinery":"The load-bearing object is the Skill, defined as S=(M,R,C): a main Markdown manual, a set of reference files holding environment rules and procedures, and loading conditions that decide when each reference enters context. This progressive disclosure keeps context cost proportional to the current task rather than to the whole knowledge base. Around this object, SkillSmith builds a two-stage black-box optimization: Skill Creation uses three cloud-agent roles (cluster, distill, merge) to turn reflection reports into a length-bounded initial Skill, and Skill Evolution uses local failure reports, a cluster-attribute-merge loop, and a length budget to refine it. The cloud agent acts as a text optimizer that treats execution feedback as a gradient-like signal, following the TextGrad-style view of LLM feedback as a surrogate gradient.","core_discovery":"The central discovery is that non-parametric environment knowledge, delivered as a progressive-disclosure Skill, repairs most local-agent failures. The paper first diagnoses that 91.7% of sampled local-agent failures trace to missing environment rules or operation procedures, not to weak core capabilities, and that manually injecting such knowledge resolves 26 of 33 knowledge-related failures. SkillSmith then automates that injection: a cloud agent executes training tasks, writes reflection reports, and a cluster-distill-merge procedure produces an initial Skill; subsequent rounds compress local failures into reports, attribute each failure cluster to Skill loading, Skill content, or near-miss execution, and apply bounded revisions. The result is a Skill that lifts the frozen Qwen3.6-27B local agent from 36.3% to 78.6% pass rate on AppWorld-Normal, from 28.1% to 74.9% on AppWorld-Challenge, and from 51.1% to 89.0% on WorkBench, outperforming the strongest non-parametric baseline by 11.3 to 26.2 points and generalizing to other SLM backbones without re-running Skill construction.","pith_inferences":["Beyond the paper's experiments, the same Skill document could in principle be reused across agent frameworks that support skill-style progressive disclosure, not just the one local framework tested here; cross-framework transfer would test whether the Skill encodes environment knowledge rather than framework-specific prompt formatting.","Once an initial Skill exists, the evolution loop's cloud dependency could be replaced by a local SLM doing the clustering and bounded edits, which would eliminate all cloud calls but likely trade away some evolution quality.","Because Skill quality is limited by what the cloud teacher knows and reflects, auditing generated Skills against the environment's ground truth (for example, by testing each stated rule against the simulator) could push local-agent performance past the cloud agent's own."],"forward_implications":["A local 27B model with a SkillSmith Skill reaches 78.6%, 74.9%, and 89.0% pass rates on AppWorld-Normal, AppWorld-Challenge, and WorkBench, versus 36.3%, 28.1%, and 51.1% without the Skill.","The Skill cuts average actions per AppWorld-Normal task from 36.1 to 9.9 and cumulative context tokens from 450.7k to 200.6k, so the added reading cost is outweighed by fewer failed steps.","SkillSmith's one-time offline cost of about $88 on AppWorld and $77 on WorkBench replaces per-task cloud LLM calls; after roughly 560 AppWorld tasks or 1,400 WorkBench tasks, it is cheaper than serving every task through a cloud agent.","Created Skills transfer to other SLM backbones without rebuilding, with pass-rate gains up to +48.2 points on one 31-billion-parameter backbone and +30.8 points on another."],"supporting_citations":[{"why":"Defines the Agent Skill as (M,R,C) with progressive disclosure, which is the carrier SkillSmith adopts for context-efficient knowledge.","marker":"[10]"},{"why":"ExpeL is a memory-based baseline that distills insights and trajectories, and it is the strongest non-parametric baseline on AppWorld for the main comparison.","marker":"[8]"},{"why":"AWM abstracts successful trajectories into workflow memories and serves as the second memory-based baseline.","marker":"[9]"},{"why":"Trace2Skill evolves skills from execution traces, is the strongest baseline on WorkBench, and motivates the requirement for automatic initial skill creation.","marker":"[11]"},{"why":"EvoSkill shows that skills can be evolved from rollout feedback, which supports the evolution-stage design.","marker":"[12]"},{"why":"AppWorld supplies the Normal and Challenge task sets, training tasks, and evaluator feedback used in creation, evolution, and testing.","marker":"[31]"},{"why":"WorkBench supplies the workplace task set and evaluator used to measure the 89.0% pass rate.","marker":"[32]"},{"why":"The SLM backbone defines the Local Agent whose core capabilities and missing environment knowledge are compared, and whose pass rates are reported.","marker":"[15]"},{"why":"TextGrad supplies the view of LLM feedback as a gradient-like signal that justifies using the cloud agent as a text optimizer for Skills.","marker":"[28]"}],"fun_headline_variants":["Auto skills let small local agents match cloud performance","SkillSmith: skills auto-built from cloud lift frozen local agents","Non-parametric skills give local agents cloud-level task ability","Auto skill evolution closes the local-agent performance gap","SkillSmith auto-builds reusable skills to upgrade local agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline trusts the Cloud Agent's exploration and self-reflection reports as accurate source material; if the Cloud Agent hallucinates an environment rule or misses an implicit procedure, the generated Skill silently encodes that error.","fun_headline_variants_meta":{"raw":{"variants":["Auto skills let small local agents match cloud performance","SkillSmith: skills auto-built from cloud lift frozen local agents","Non-parametric skills give local agents cloud-level task ability","Auto skill evolution closes the local-agent performance gap","SkillSmith auto-builds reusable skills to upgrade local agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000784,"raw_usage":{"total_tokens":3508,"prompt_tokens":1040,"completion_tokens":2468,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":2390}},"tokens_in":656,"tokens_out":2468,"duration_ms":19252,"temperature":1.0,"reasoning_tokens":2390,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:32:29.055838+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test: inject one fabricated environment rule into the Cloud Agent's exploration prompt before Skill Creation, then check whether the final Skill propagates that rule and whether the Local Agent follows it on tasks where the rule is false.","supporting_citations":[{"cited_title":"Equipping agents for the real world with agent skills,","cited_arxiv_id":null,"evidence_quote":"Defines the Agent Skill as (M,R,C) with progressive disclosure, which is the carrier SkillSmith adopts for context-efficient knowledge."},{"cited_title":"ExpeL: LLM agents are experiential learners,","cited_arxiv_id":null,"evidence_quote":"ExpeL is a memory-based baseline that distills insights and trajectories, and it is the strongest non-parametric baseline on AppWorld for the main comparison."},{"cited_title":"Agent workflow memory,","cited_arxiv_id":null,"evidence_quote":"AWM abstracts successful trajectories into workflow memories and serves as the second memory-based baseline."},{"cited_title":"AppWorld: A controllable world of apps and people for benchmarking interactive coding agents,","cited_arxiv_id":null,"evidence_quote":"AppWorld supplies the Normal and Challenge task sets, training tasks, and evaluator feedback used in creation, evolution, and testing."},{"cited_title":"WorkBench: A benchmark dataset for agents in a realistic workplace setting,","cited_arxiv_id":null,"evidence_quote":"WorkBench supplies the workplace task set and evaluator used to measure the 89.0% pass rate."},{"cited_title":"Qwen3.6-27B model card,","cited_arxiv_id":null,"evidence_quote":"The SLM backbone defines the Local Agent whose core capabilities and missing environment knowledge are compared, and whose pass rates are reported."}],"review_version":1}