{"id":"ed57c1c0-ffbd-46b0-9a44-04a11598bcf3","arxiv_id":"2607.13683","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A gated, statistically-checked self-evolution loop improves frozen agents' harnesses by +9 to +15.5 points on sealed tests across six benchmarks, retaining 86-147% of the training gain.","lead":"This paper builds a loop in which a stronger language model proposes edits to an agent's prompt-and-runtime 'harness' while deterministic code — validity, activation, and paired statistical checks plus a test set sealed from evolution — decides which edits truly helped. The credited sealed-test gains, +9 to +15.5 points across six benchmarks with a frozen model, are worth reading because agent self-improvement results are usually unauditable and this paper is a concrete attem","discovery_kind":"new_method","skeptic_critique":null,"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a self-evolving harness framework for frozen LLM agents. A stronger evolver diagnoses failures and proposes patches, while deterministic code owns sampling, measurement, and paired per-task significance testing. Candidate patches populate a categorical MAP-Elites archive keyed on (where × why) pathology cells. The authors report that the train-selected harness improves over vanilla on a sealed test across six credited benchmarks (+9 to +15.5pp, retention 86–147%), with SWE-bench reported as preliminary. They also claim a cross-model pathology-to-patch matching law on AppWorld across Qwen and Gemini models.","tokens_in":18537,"tokens_out":6419,"duration_ms":130503,"significance":"If the sealed-test guarantees are trustworthy, the paper makes a strong methodological contribution: it demonstrates a disciplined loop in which the proposing model is separated from the crediting machinery, every credited gain passes a paired 2σ gate on a withheld set, and the archive keyed on pathology rather than task identity is a plausible anti-overfitting inductive bias. The honest reporting of SWE-bench, the documented rejection of a wrong hypothesis, and the release of per-task differences are commendable and increase confidence in the experimental core. The cross-model dissociation, if confirmed, is a valuable result for the harness-optimization literature. However, the central claim is currently undermined by unresolved questions about sealed-test integrity and by an internal contradiction in the number of times the test set was used.","major_comments":[{"comment":"The sealed-test guarantee is procedural rather than mechanical. The evolver (Claude Opus 4.8 via Claude Code) operates in the same git-tracked environment that contains the test tasks, and no access log for the test set is provided. The manuscript itself states 'Enforcing the withholding mechanically is future work.' Because the headline conclusion is that the gains generalize, the possibility that the evolving agent observed the test set during evolution makes the credited sealed-test numbers uninterpretable as currently presented. This needs to be fixed, either by sandboxing the test set with access logging or by rerunning the main experiments under mechanical separation.","section":"§3.4 and §3.2"},{"comment":"The protocol says the sealed test is 'scored once' and that 'each credited number is a single sealed comparison' (Algorithm 1, §4.3). Yet §4.6 reports that on GDPval 'a recovery+VF variant reached the highest sealed score (+11.5pp)' despite being ranked below on train. This implies that more than one candidate harness was scored on the sealed test. If true, the 'scored once' claim is false and the argument that no multiple-comparison correction is needed collapses; if false, the statement needs clarification. Please disclose exactly how many harnesses were evaluated on the sealed test and whether the reported +9.2pp is the single pre-registered comparison or a post-hoc selection.","section":"§4.6 vs §3.4/§4.3"},{"comment":"The abstract claims 'Across seven domains ... its credited gains there are +9 to +15.5pp and retain 86–147% of the training gain.' Table 1 shows SWE-bench with +5.1pp, z=0.78, retention 37%, explicitly not credited. Thus the credited-gain claim covers six domains, not seven. The abstract and §1 should be revised to avoid overclaiming: six credited benchmarks plus one preliminary result.","section":"Abstract and Table 1"},{"comment":"The six credited benchmarks are said to use 'disjoint train/sealed-test splits' and SWE-bench's split is 'disjoint by construction,' but the construction is not documented. For each benchmark, specify how the split was made (e.g., provided by EvoAgentBench, random with seed, temporal split) and report any overlap check. Without this, the held-out status of the test set cannot be independently audited.","section":"§4.1"}],"minor_comments":[{"comment":"The definition of pass@1 with K=3 attempts is unusual (per-task success over K attempts, not the standard first-sample pass@1). Please flag this at first use to avoid confusion with standard pass@1 notation.","section":"§3.4"},{"comment":"The 'thinking-off' ablation is described as 'blanket thinking-off toggle' but the table footnote says 'thinking disabled only after a detected runaway.' Please make the condition unambiguous in the table caption.","section":"§4.4 / Table 2"},{"comment":"The lemma refers to a 'posterior mean' without stating the prior. If a Beta prior is assumed, give the prior parameters; otherwise clarify that the variance reduction is computed for the empirical Bernoulli variance.","section":"App. B, Lemma 1"},{"comment":"The sentence 'researcher degrees of freedom are spent on train' is true only if the sealed test is never used for any model selection. The GDPval discrepancy in §4.6 makes this sentence potentially misleading; after fully resolving the test-usage question, revisit this wording.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong methodological core: paired per-task significance, validity/activation gates, and honest reporting of a preliminary result are all commendable. However, the sealed-test integrity is load-bearing and currently rests on procedure rather than mechanism, with no access logs provided. Even more concrete is the §4.6 statement that a non-selected variant reached the highest sealed score, which directly contradicts the 'scored once' protocol. These issues are fixable by disclosure, sandboxing, or rerunning the central experiments, but they must be resolved before the claims can be accepted. If the authors can provide a mechanically sealed test evaluation or equivalent assurance, I would be favorably inclined toward the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one for the propose/credit split and the discipline around it. The evolver proposes patches; deterministic code owns sampling, the three gates, and the paired 2-sigma test; the sealed test is scored once. Six of seven benchmarks clear the bar, retention is 86-147%, and SWE-bench is honestly reported as preliminary instead of stretching the bar. They even document a wrong hypothesis being rejected by the gate. That is genuinely good methodology and a useful template.\n\nThe new thing is not any single ingredient: the related work already has harness evolution, held-out discipline from VeRO, and cause/status/mechanism clustering close to their where-times-why split. What's new is the combination: a categorical quality-diversity archive keyed on pathology, with cross-cell recombination, plus the paired-2-sigma gate and retention scalar. Their claim that the winning patch tracks the model's dominant pathology, not its size or family, is plausible and supported by the cross-model dissociation.\n\nNow the soft spots, in proportion. The biggest is that the sealed-test guarantee is enforced by procedure, not mechanism. Section 3.4 says outright: 'Enforcing the withholding mechanically is future work.' The evolver runs in the same git-tracked environment that holds the test tasks, and no access log is provided. If the test set was observed, every credited number is uninterpretable. That is not a hit job - it's the paper's own caveat - but it means 'trustworthy by construction' is too strong. Second, there's no code or data, no per-task ledgers, and the train/sealed split construction is unspecified. Those are addressable, and for a paper whose whole pitch is auditability, shipping them should be a condition of acceptance.\n\nThe central argument holds up: I found no load-bearing error in the protocol or math. The novelty section is appropriately modest. The main missing analyses are a real ablation of the GSME archive and any comparison against one of the cited prior systems like Self-Harness or AHE - without those, the QD archive's contribution is asserted more than shown.\n\nWho's this for? People building self-evolving agent pipelines and anyone designing evaluation protocols for agent optimization. It deserves a serious referee, but with a clear ask: enforce test withholding mechanically, ship the artifacts, and add the baseline comparison. I'd engage with it.","headline":"A carefully disciplined self-evolution loop with a load-bearing audit gap: the sealed-test guarantee is procedural, not mechanical, but the protocol is honest enough to deserve a serious referee.","tokens_in":19074,"tokens_out":1817,"would_cite":true,"duration_ms":541467,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A self-evolving loop improves a frozen model's harness by +9 to +15.5 percentage points on held-out tests, with every credited gain gated by a paired 2-sigma significance test on a sealed test set.","keywords":["self-evolving agents","harness optimization","gated semantic MAP-Elites","quality-diversity archive","paired significance testing","sealed test generalization","frozen language models","pathology-to-patch matching"],"falsifier":"Re-run the evolution loop with the sealed test placed in a separate container with no read access and every filesystem read by the evolver logged; reproduce the six credited sealed gains and the 86–147% retention under mechanical withholding. Alternatively, audit the evolution tree's git history and the evolver's command history for any read of the held-out task files — a single such read would falsify the 'sealed' property and make the generalization claims uninterpretable.","tokens_in":18377,"feed_emoji":"🧬","tokens_out":7259,"duration_ms":66994,"temperature":0.7,"pith_summary":"This paper tries to establish that the harness around a frozen language model — the prompts, injected knowledge, control loop, and configuration — is a reliable and auditable optimization lever. It argues the hard part is not generating better edits but deciding which one truly helped, since self-generated feedback is noisy and single-run gains are often artifacts. The authors build a loop that separates proposing from crediting: a language model diagnoses failures and writes patches, while deterministic code owns sampling, measurement, and significance testing, crediting only paired 2-sigma gains on a sealed test scored once after evolution. Across six benchmark domains with a frozen open-weight model, the train-selected harness achieves credited held-out gains of +9 to +15.5 percentage points, retaining 86–147% of the training gain, and the winning patch tracks the model's dominant failure pathology rather than its size or family. If correct, this makes harness self-evolution a dependable way to raise deployed performance without touching weights, and locates the transferable contribution in the diagnose-and-credit loop itself, not in any specific harness.","feed_headline":"Frozen-agent harnesses self-improve by up to 15.5 points","feed_subtitle":"Every credited gain clears a paired 2-sigma test on a sealed set; retention hits 86–147%.","key_machinery":"The central object is the Gated Semantic MAP-Elites (GSME) archive combined with a strict three-gate credit pipeline. Each candidate harness edit is placed in a categorical archive cell indexed by (where × why): the harness surface it touches (prompt, knowledge, runtime, config) and the diagnosed failure pathology it addresses (e.g., thinking-runaway, premature finalization). Credit is decided only by deterministic code: a validity gate re-runs infrastructure failures, an activation gate requires the mechanism to have fired, and a paired 2σ significance test on per-task differences decides acceptance. The sealed test, withheld from the evolver and scored once after evolution, is the arbiter","core_discovery":"The paper's central claim is that a frozen model's harness can be improved automatically by a self-evolution loop whose every credited improvement is real, in the specific sense of clearing a paired 2σ significance bar on a sealed test withheld from all decisions. The loop's GSME archive organizes candidate edits by the (where × why) pathology they address — the harness surface touched and the diagnosed failure mode — rather than by the tasks they fix, which is the anti-overfitting bias. The claimed empirical result is credited held-out gains of +9 to +15.5 percentage points on six domains, with 86–147% retention of training gain, and a pathology-to-patch matching law: the winning patch chan","pith_inferences":["If the pathology-to-patch law extends beyond the two families tested, harness evolution could become a two-step procedure: first diagnose the model's dominant failure mode cheaply, then fetch the matched patch from a library — turning an expensive per-model evolution run into a lookup.","The anti-overfitting design could transfer to other self-improvement loops with verifiable per-task rewards (skill acquisition, memory editing): key the archive by the failure type rather than the task identity, and gate credit with paired significance.","The paper's own caveat that mechanical sealing is future work implies a direct audit: instrument the evolver's filesystem access and re-run with the test set physically inaccessible; the retention numbers are only convincing if they survive that check.","The loop's reliance on per-task verifiers means the method inherits any verifier gameability; in weak-verifier domains, an extension would need a meta-verifier or reward-model calibration before the same guarantees apply."],"forward_implications":["If the central claim holds, harness optimization is a dependable lever for frozen, mid-capability models: credited held-out gains of +9 to +15.5 percentage points, with no overfitting collapse, across tasks the evolving agent never saw.","The winning patch is portable exactly as far as the pathology it targets is shared: the same careless-to-submit-verify match recurs across two model families, while a patch mismatched to a model's pathology is near-zero in effect.","What reliably transfers is the diagnose-and-credit loop, not any specific harness; a new model still needs its own diagnosis even if a pathology recurs.","A deterministic 'mean improves' rule would have credited phantom gains, including a mechanism whose activation beacon never fired; the paired significance and activation gates are necessary, not cosmetic.","The evolved harness expands the set of solvable tasks, since best-of-3 pass also rises on every credited domain, not just per-attempt hit rate."],"fun_headline_variants":["Frozen-agent harnesses gain 9–15.5 points via verified self-evolution","Sealed-test-verified harness evolution: +15.5 pp on frozen models","Harness co-evolution: frozen LLMs gain 9–15.5 pp, verified on sealed tests","Self-evolving harness beats overfitting: +15.5 pp on frozen agents","Gated archive certifies harness gains: 9–15.5 pp on sealed tests"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The sealed test is never read by the evolving agent is enforced only by procedure, not mechanism — the evolver runs in the same environment that holds the test tasks, so any observed test signal would invalidate every credited gain and retention ratio; the paper flags mechanical enforcement as future work.","fun_headline_variants_meta":{"raw":{"variants":["Frozen-agent harnesses gain 9–15.5 points via verified self-evolution","Sealed-test-verified harness evolution: +15.5 pp on frozen models","Harness co-evolution: frozen LLMs gain 9–15.5 pp, verified on sealed tests","Self-evolving harness beats overfitting: +15.5 pp on frozen agents","Gated archive certifies harness gains: 9–15.5 pp on sealed tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000688,"raw_usage":{"total_tokens":3000,"prompt_tokens":835,"completion_tokens":2165,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":2050}},"tokens_in":579,"tokens_out":2165,"duration_ms":17545,"temperature":1.0,"reasoning_tokens":2050,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T04:03:31.005335+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evolution loop with the sealed test placed in a separate container with no read access and every filesystem read by the evolver logged; reproduce the six credited sealed gains and the 86–147% retention under mechanical withholding. Alternatively, audit the evolution tree's git history and the evolver's command history for any read of the held-out task files — a single such read would falsify the 'sealed' property and make the generalization claims uninterpretable.","supporting_citations":[],"review_version":1}