{"id":"cfcf45ef-99e2-4ff4-a0be-8fe4f49aec0e","arxiv_id":"2608.05810","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Unconditional skill accumulation in self-evolving agents is non-monotone, and a pre-commit verification gate with per-skill and combinatorial checks prevents the collapse.","lead":"Self-improving AI agents can get worse as they accumulate skills, because flawed skills contaminate the ones written later. The authors test a gating system that checks each new skill alone and in combination, and it beats unconstrained accumulation on a terminal-task benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (5)'s regime assumption—that descendants inherit flawed reasoning—is never measured, and Figure 4's 'irrecoverable' residual tests only two cleanup policies, so the pre-commit necessity claim overreaches.","rationale":"The reader's conditional verdict is appropriate, and the weakest assumption identified—unverified inheritance of flawed reasoning behind the irreversibility claim—is the same concern that is most load-bearing here. The paper's empirical contribution, VaG improving monotonically to 72% with a smaller pool, stands independently of the strong irreversibility narrative; even if post-hoc remediation could recover some of the drop, pre-commit gating may still be preferable for cost and reliability. However, the title, abstract, and conclusion make 'structurally irreversible' and 'pre-commit necessity' central claims. Those claims require either a direct causal trace of inherited flawed reasoning (measurement of descendant behavior with and without the source in context) or a search over a broader class of post-hoc remediation operations than two deletion policies. Figure 4's residual 5.6pp gap is presented as proof of irrecoverability, but it is only one observation from one split with one seed, and the oracle cleanup policy itself may suppress recovery by removing useful lineage members. The proposed test would settle whether the 'irrecoverable' label is empirically warranted or whether the honest claim is limited to 'the two cleanup policies we tried recover little.' The reader already flagged this, and my read does not move the verdict. The paper is otherwise valuable: the non-monotonicity observation, the three-level contamination taxonomy, and the complementary-critics design are useful and coherent, and the confidence intervals are reported honestly. The main correction needed is to temper the irreversibility language or back it with the missing measurements.","tokens_in":11882,"tokens_out":4405,"duration_ms":48391,"concrete_test":"On the same collapsed R5 pool, run four additional post-hoc remediations and compare on Test-25 (and Event-50 for comparability with Figure 4): (1) delete every skill that fails singleton A-B replay on Holdout; (2) apply Gate 2's marginal-gain greedy selection to choose a best Hot subset from the R5 pool; (3) regenerate descendants of the 8 source skills with sources excluded from the distillation context and swap them in; (4) edit descendants to remove any reference to or phrasing traceable to their recorded source. If any policy recovers more than Oracle cleanup's 6.7pp (or reaches 62% on Event-50), the 'irrecoverable' conclusion is empirically false. Separately, for each recorded descendant, measure pass@1 with and without its source skill in the context; if removing the source does not change descendant behavior, the Eq. (5) regime is not satisfied.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central move is from 'source-only rollback recovers only 1.7pp' to 'contamination is structurally irreversible, so pre-commit gating is a structural necessity.' This move has two unverified links. First, Eq. (5) is explicitly conditional: source-only rollback is dominated by full-lineage cleanup only 'in that regime' where descendants have inherited flawed reasoning. The paper never shows that regime holds for the 8 identified sources—it provides one anecdote (the git-conflict skill) but no systematic measurement that a descendant reproduces a flaw specifically inherited from its source. Second, the empirical signature in Figure 4 tests exactly two remediation policies (delete 8 source skills; delete sources plus recorded descendants). The residual 5.6pp gap to the R3 peak is labeled 'irrecoverable even under Oracle cleanup,' but no alternative post-hoc operation was tried: editing descendants, re-distilling descendants without sources in context, deleting every R5 skill that fails singleton A-B replay, or running the paper's own marginal-gain selection over the R5 pool. Moreover, oracle cleanup removes the entire lineage, including any descendants that are net-beneficial in combination; the remaining gap could be a removal artifact rather than inherited contamination. Because the 'pre-commit necessity' framing depends on irreversibility being stronger than 'two hand-picked cleanups fail to recover the peak,' the central theoretical claim is underdetermined by the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies self-evolving LLM agents that accumulate natural-language skills in their context. It reports that unconditional skill accumulation on Terminal-Bench 2 is non-monotone: pass@1 rises from 48% at R1 to 62% at R3, then falls to 50% at R5 as the pool grows from 35 to 179 skills. The paper formalizes a 'capability-contamination phase transition,' argues that contamination is structurally irreversible because descendants of defective skills inherit flawed reasoning, and proposes Verifier-as-Gatekeeper (VaG), a two-gate pipeline that filters skills individually and via marginal-gain subset selection before they enter the runtime context. Experiments report that VaG improves monotonically to 72% pass@1 with a 37-skill pool, and that the frozen pool transfers positively to four other backbones and a second benchmark. The central claims of non-monotonicity and irreversibility are currently supported by a single evolution run and by a rollback analysis testing only two cleanup policies.","tokens_in":12220,"tokens_out":9540,"duration_ms":86823,"significance":"If the non-monotonic capability trajectory and the structural irreversibility of skill contamination hold, the paper identifies an important failure mode in self-evolving agents and a sensible remedy in pre-commit gating. The VaG design is clearly presented, the ablation structure is clean, and the use of a deterministic verifier and cross-model/cross-benchmark transfer are strengths. The paper is also honest about the overlap of its Wilson confidence intervals. However, the primary evidence for the phenomenon is a single 5-round run, and the irreversibility conclusion is inferred from only two cleanup policies without direct measurement of the proposed inheritance mechanism. The 'structural necessity' framing therefore goes beyond what the data currently establish. The issues are addressable with additional seeds and targeted experiments, so the paper is a candidate for major revision rather than rejection.","major_comments":[{"comment":"The central phenomenon of non-monotone skill accumulation is reported from a single evolution run with one fixed seed on the 50-task Event split. The paper acknowledges that the 95% Wilson CIs are approximately 30pp wide and that per-round point tests are underpowered, but then asserts that a direction-of-trend difference '50 tasks cannot produce by chance.' With only one trajectory there is no sampling distribution over runs, so this assertion is unsupported; a single lucky R3 and unlucky R5 could produce the same shape. Please provide multiple independent seeds (at least five) or a per-task paired test (e.g., bootstrap over the 50 tasks of the peak-round versus final-round pass rate) to establish that the inverted-U shape is not noise. The current evidence is insufficient for the paper's headline claim that 'unconditional skill accumulation is non-monotone.'","section":"Experiments; Table 1, Figure 3"},{"comment":"The conclusion that contamination is 'structurally irreversible' and that pre-commit gating is a 'structural necessity' rests on two unverified links. First, Eq. (5) holds only in the regime where descendants have inherited the source's flawed reasoning, but the paper never measures this inheritance; the sole supporting evidence is one anecdote about the git-conflict skill. Second, Figure 4 tests exactly two cleanup policies (source-only deletion and oracle full-lineage deletion); the 5.6pp residual labeled 'irrecoverable even under Oracle cleanup' has not been shown to be unrecoverable by other post-hoc operations, such as editing descendants, re-distilling descendants with the sources removed from context, or applying the paper's own marginal-gain selection to the R5 pool. Moreover, oracle cleanup removes the entire lineage, which may include net-beneficial descendants, so the residual gap could be a removal artifact. Please either provide systematic evidence of inherited flawed reasoning in descendants (e.g., trace a specific defective logic from source to descendant behavior) or soften the irreversibility claim to 'the two tested cleanup policies fail to fully recover the peak.' The current overreach is load-bearing because the entire motivation for VaG depends on post-hoc remediation being impossible in principle, not merely ineffective in two configurations.","section":"§Irreversibility of Contamination (Eq. 5); §Post-hoc Rollback Analysis (Figure 4)"},{"comment":"The 'capability-contamination phase transition' is formalized as k* = argmax P(k), which is a definition of the observed peak rather than a prediction. A formalization should yield testable content; as written, Eq. (4) merely labels the empirically highest round. To make the phase transition a substantive claim, the paper could specify a null model (e.g., a random walk or a model where each skill has i.i.d. utility) and show that the observed single-peaked trajectory is unlikely under it, or derive a prediction for k* as a function of pool composition or task difficulty. Without such a test, the 'formalize' contribution is descriptive.","section":"§Problem Formalization (Eq. 4)"},{"comment":"The ablations are interpreted as demonstrating that the three critics are complementary and non-substitutable, and that the marginal-gain gate removes combinatorial contamination (8pp drop when removed). However, these are single-run differences on 50 tasks with a reported CI width of about 30pp; the observed 4-10pp differences are within that noise band. Please provide per-task paired analyses or multiple seeds for the ablation configurations before drawing conclusions about non-substitutability. This also applies to the claim that 'each critic intercepts a largely disjoint class of harmful skills,' which would be better supported by reporting the overlap of rejected skills across the three critics.","section":"§Ablation Studies (Table 2)"}],"minor_comments":[{"comment":"The text says source-only rollback recovers 'only 2pp (50%→52%)' while Figure 4 reports 1.7pp; please reconcile the numbers for consistency.","section":"§Main Results"},{"comment":"The cost bound 'a single greedy pass costs at most |W|−1 joint replays' is incorrect for a greedy algorithm that selects the candidate with the largest estimated marginal gain at each step; such a pass requires O(|W|²) joint evaluations. The error does not affect the experimental conclusions given |W| ≤ 15, but the stated bound should be corrected.","section":"§Pre-commit Gating (Gate 2)"},{"comment":"The distillation process π that converts execution trajectories into skills is not described in the main text beyond 'short natural-language skills' and a reference to the Supplementary Material; please provide the distillation prompt and selection criteria, since the non-monotonicity phenomenon may be sensitive to the distiller's behavior.","section":"§Implementation details"},{"comment":"The paper does not state how many independent evolution runs were performed; please specify the number of seeds and, if only one, say so explicitly and temper the claims accordingly.","section":"§Experiments"},{"comment":"The phrase 'the empirical signature of irreversibility' and the statement that contamination is 'structurally irreversible' are stronger than what Figure 4 shows; consider replacing with 'consistent with irreversibility' or 'not fully recoverable by the tested cleanup policies.'","section":"§Abstract and §Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central scientific claim is important but currently rests on a single seed and an underspecified irreversibility argument. The authors are transparent about statistical limitations, which is commendable, but the trajectory-shape defense is not a substitute for replication. I recommend major revision with emphasis on additional seeds and direct measurement of skill-lineage inheritance. The paper fits the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest read: the paper earns its place as a phenomenon paper. The observation that self-evolving agents get worse as the skill pool grows past a point, on a real verifier-checked benchmark, is new and worth knowing. The three-level contamination taxonomy is useful framing, and VaG is a sensible engineering response: cheap structural check, behavioral replay, semantic review, then greedy joint selection. The ablation is the best part—each critic and the joint gate measurably changes outcome, which supports the claim that the components are not redundant. Cross-backbone transfer of the frozen pool is a nice bonus.\n\nThe soft spots are real and concentrated in the 'structural irreversibility' argument. The stress-test note lands: Eq. (5) is explicitly conditional on descendants having inherited flawed reasoning, and the paper never measures that inheritance. One anecdote about a git-conflict skill is not a systematic trace. The 'irrecoverable' residual is based on exactly two cleanup policies; a pool-level greedy selection or re-distillation without sources might recover more. Calling the residual 'irrecoverable' overreaches. The phase transition k* is also defined as argmax of the observed curve, so it is a description of the peak, not a prediction. None of this kills the main practical claim—that pre-commit gating beats letting everything in—but it means the necessity narrative is stronger than the evidence.\n\nStatistics are thin: one run, one seed, 50 tasks on the distillation split, with Wilson intervals the authors themselves say overlap per round. The qualitative trend claim is plausible, but with a single trajectory it is not a statistical test. Main results should be reported on the held-out Test split as well. No code or data either.\n\nCitation pattern looks fine—Voyager, ExpeL, AHE, ACE are the right comparators and the paper is clear about how it differs.\n\nWho this is for: anyone building skill libraries for self-evolving agents. The phenomenon and the VaG mechanism deserve a serious referee. I would send it to review, but the revision should soften the irreversibility language, add seeds or a trend test, report the Test-split trajectory, trace at least one full contamination chain, and release the artifacts.","headline":"A genuine phenomenon and a sensible gate, but the irreversibility claim is load-bearing and unproven; the paper deserves review with revisions.","tokens_in":12741,"tokens_out":2168,"would_cite":true,"duration_ms":23223,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Unconditional skill accumulation in self-evolving agents is non-monotonic and structurally hard to undo, so skills must be vetted before entering runtime context.","keywords":["self-evolving agents","skill distillation","skill contamination","capability-contamination tipping point","pre-commit gating","marginal-gain subset selection","combinatorial contamination","LLM agent transfer"],"falsifier":"Run a controlled lineage-tracing experiment: inject one skill carrying a known false instruction into a live pool, distill one or two more rounds, then test descendants on tasks where the false instruction would matter, and also attempt post-hoc removal of source plus descendants. If descendants never reproduce the false step, or if a post-hoc operation restores nearly all of the lost pass@1, the structural irreversibility claim collapses.","tokens_in":11652,"feed_emoji":"🛡️","tokens_out":10504,"duration_ms":92559,"temperature":0.7,"pith_summary":"Self-evolving agents accumulate skills by distilling execution trajectories, but the paper argues that admitting every distilled skill unconditionally is actively harmful: performance rises to a critical pool size, then collapses as defective skills contaminate later distillation. On Terminal-Bench 2, ungated evolution peaks at 62% pass@1 at round 3 and falls to 50% by round 5, and removing the identified 8 culprit skills afterward recovers only 1.7 of the 12.3-percentage-point drop. The paper explains this as structural irreversibility: later skills are distilled with earlier skills as reference context, so flawed reasoning passes into descendants that do not name their source. From this it derives Verifier-as-Gatekeeper (VaG), a pre-commit admission pipeline in which each skill must pass structural, behavioral, and semantic checks and then a marginal-gain joint selection before entering runtime context. VaG improves monotonically to 72% pass@1 with a 37-skill pool and transfers to four other backbones and a second benchmark without re-evolution.","feed_headline":"Gating skills first reaches 72%; unchecked growth collapses","feed_subtitle":"On Terminal-Bench 2, unfiltered accumulation fell from 62% to 50%; vetting skills first hit 72% with a pool five times smaller.","key_machinery":"The load-bearing object is the admission operator $G$ in Eq. (1): ungated evolution sets $G=\\mathrm{identity}$, while VaG replaces it with a progressive trust hierarchy $\\mathcal{L}=\\{\\mathrm{Cold},\\mathrm{Warm},\\mathrm{Hot}\\}$. The first gate (Cold to Warm) requires a skill to pass all three heterogeneous critics—a deterministic schema check, a single-skill A-B behavioral replay on held-out tasks, and one LLM semantic review—so a harmful skill must fail to fool all three to enter. The second gate (Warm to Hot) is a marginal-gain greedy selection over measured joint held-out performance $f(H)$, adding a candidate only when it strictly improves the set, which targets combinatorial contamination that per-skill checks cannot see. The conceptual machinery behind the motivation is the inequality $R(\\mathcal{M}_r \\setminus \\{s\\}) < R(\\mathcal{M}_r \\setminus (\\{s\\} \\cup \\mathrm{desc}(s)))$ (Eq. 5), which formalizes why post-hoc rollback is structurally weaker than full-lineage cleanup.","core_discovery":"The paper's central claim is a phase transition in self-evolving agents: the capability curve $P(k)$ over pool size is not monotone but single-peaked, with a critical size $k^*$ beyond which the marginal value of adding skills turns negative. It offers a three-level contamination taxonomy—individual, combinatorial, and systemic—and an irreversibility argument: because round-$r$ skills are generated conditioned on the then-live pool, a defective source skill has descendants that inherit its flawed reasoning, so source-only removal is strictly dominated by full-lineage cleanup (Eq. 5). Empirically, source removal recovers only 1.7pp of the 12.3pp peak-to-R5 drop on Terminal-Bench 2, and even oracle full-lineage cleanup leaves 5.6pp unrecovered, which the paper reads as evidence that no post-hoc remediation can fully undo contamination. The constructive consequence is VaG: a three-tier Cold/Warm/Hot trust hierarchy with three heterogeneous critics at the first gate and marginal-gain subset selection at the second; gated evolution improves every round to 72% pass@1 with a pool roughly five times smaller, and its frozen pool transfers positively to other backbones and a second benchmark.","pith_inferences":["If contamination inheritance is as broad as the paper assumes, the same single-peaked risk should appear in any growing-context mechanism, such as tool-document stores, retrieval packs, or memory queues, so the gate design generalizes beyond skill libraries.","A stress test the paper does not run is exhaustive selection on small Warm pools; comparing it with the marginal-gain greedy would quantify the gap the heuristic leaves.","Recording provenance at distillation time would make full-lineage cleanup practical; re-running the rollback analysis with real provenance could either shrink the 5.6 percentage points the oracle still loses or confirm them as structurally irrecoverable."],"forward_implications":["Self-evolving agents that admit skills unconditionally should expect capability to peak and then decline as the pool grows past a critical size, rather than keep improving.","Skill admission must be treated as a pre-commit decision, because post-hoc removal of a culprit skill cannot undo contamination carried by descendant skills.","Pre-commit gating can outperform even oracle early stopping, since a smaller, cleaner pool beats the best round of unchecked accumulation.","Reliable self-evolution needs joint or combinatorial verification, not just per-skill checks, because skills that are individually harmless can conflict when injected together.","Gated skill pools can transfer across backbones and benchmarks without re-evolution, indicating that they encode reusable, model-agnostic engineering knowledge."],"supporting_citations":[{"why":"Supplies Terminal-Bench 2, the verifier-checked benchmark whose Event/Holdout/Test split drives all main results.","marker":"(Merrill et al. 2026)"},{"why":"Voyager is the archetype of unconditional skill admission; the paper contrasts its store-anything approach with pre-commit gating.","marker":"(Wang et al. 2023)"},{"why":"ACE scales context-level evolution but inherits unconditional admission, serving as a comparison point for scale context.","marker":"(Zhang et al. 2025)"},{"why":"AHE evolves the full harness with deferred aggregate evaluation, motivating the paper's shift to per-skill pre-commit verification.","marker":"(Lin et al. 2026)"},{"why":"MetaClaw uses dual-timescale meta-learning with weight access and no admission gate, framing the practical constraints VaG addresses.","marker":"(Xia et al. 2026)"},{"why":"Provides the classical submodular greedy approximation that motivates the marginal-gain subset-selection heuristic at Gate 2.","marker":"(Nemhauser, Wolsey, and Fisher 1978)"},{"why":"InterCode NL2Bash is the second benchmark used to test whether frozen gated skills transfer across benchmarks.","marker":"(Yang et al. 2023)"}],"fun_headline_variants":["Too many skills break agents; gate them early","Skill contamination: pre-commit gating saves agents","Unchecked growth collapses; gating hits 72% pass@1","Capability phase transition: pre-gate skills to avoid backfire","5x smaller pool, 72% pass@1 via pre-commit gating"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that skills distilled after a defective skill enters the context inherit that skill's flawed reasoning, an inheritance the paper asserts rather than measures.","fun_headline_variants_meta":{"raw":{"variants":["Too many skills break agents; gate them early","Skill contamination: pre-commit gating saves agents","Unchecked growth collapses; gating hits 72% pass@1","Capability phase transition: pre-gate skills to avoid backfire","5x smaller pool, 72% pass@1 via pre-commit gating"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000248,"raw_usage":{"total_tokens":1632,"prompt_tokens":1117,"completion_tokens":515,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":733,"completion_tokens_details":{"reasoning_tokens":426}},"tokens_in":733,"tokens_out":515,"duration_ms":4911,"temperature":1.0,"reasoning_tokens":426,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T23:04:47.103448+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled lineage-tracing experiment: inject one skill carrying a known false instruction into a live pool, distill one or two more rounds, then test descendants on tasks where the false instruction would matter, and also attempt post-hoc removal of source plus descendants. If descendants never reproduce the false step, or if a post-hoc operation restores nearly all of the lost pass@1, the structural irreversibility claim collapses.","supporting_citations":[],"review_version":1}