{"id":"3dbff3d6-c51b-4b14-852f-1ca8beb6a659","arxiv_id":"2608.09537","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A continual world-model optimization framework in which retrieved strategies are hypotheses until verified under a frozen target-side evaluator; on three world-model families it cuts search and GPU cost by roughly two-thirds and lowers negative transfer to 0.06.","lead":"VERDI is a system that stops treating past optimization successes as directly reusable recipes: it measures each world model with shared probes, retrieves past strategies only as ranked hypotheses, and admits them as knowledge only after target-side validation. The paper reports 68% lower search cost, 69% lower GPU cost, and a drop in negative transfer from 0.34 to 0.06 across Ctrl-World, Cosmos, and RoboCoin.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Negative-transfer rate in Table 4 is never defined, so the headline 0.34→0.06 reduction cannot be independently checked; per-fold counts are required.","rationale":"The reader's weakest_assumption is the validity of the IRG fingerprint geometry as a transfer predictor. That concern is real, but the paper's collision-detection and probe-evolution machinery are explicitly designed to absorb IRG collisions as learning costs, so the geometry concern alone does not demonstrate a failure; it is a limitation the design partially addresses. The more load-bearing issue is that the headline negative-transfer result cannot be interpreted without a definition and per-fold counts. This is not an internal contradiction, but it blocks independent verification, which is exactly the basis for the reader's CONDITIONAL verdict. The reader's rationale does flag the undefined negative-transfer metric, but their stated weakest_assumption is the IRG geometry, so agreement is partial. I keep the verdict unchanged: the concern is fixable by releasing the ledger and defining the metric, not evidence of a demonstrated error.","tokens_in":24569,"tokens_out":9899,"duration_ms":91576,"concrete_test":"Obtain the per-fold LOBO ledger and a written operational definition of negative transfer. Recompute Table 4 by counting, for each fold and arm: the number of retrieved skills tested on the target, the number whose measured target-side effect was significantly negative, and the trials to the first strict-gate positive. Require the same denominator for unconditional and certified reuse. Then apply a bootstrap over the three seeded folds and check whether the certified-reuse negative-transfer rate remains near 0.06; if the definition changes the comparison or the certified rate exceeds 0.10 under the natural definition, the central claim is not supported as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative evidence for the paper's principle 'retrieval is not transfer' is the certified-reuse row of Table 4: 3.6 trials, 96 GPU-h, and negative transfer 0.06 versus 0.34 for unconditional reuse. But the paper nowhere defines the negative-transfer rate. Section 4.1 defines trials, C_first+, and overhead precisely, and Section 4.5 reports 'negative-transfer rate' in Table 4 without a formula, numerator, denominator, or per-fold counts. Without such a definition, the 0.34→0.06 gap could be an artifact of counting different populations in the two rows: unconditional reuse transfers unvalidated skills, while certified reuse screens every candidate before admitting it, so the rows may not be measuring the same object. The missing per-fold detail is also load-bearing because all numbers are means over three seeds and the LOBO folds are few; a single fold can dominate the aggregate. The abstract elevates this metric to a headline result, so as written the central empirical claim is unverifiable. This is an external-validity/measurement concern rather than an internal inconsistency; the architecture itself is coherent and the control-plane ablations are unusually honest.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VERDI, a continual framework for world-model optimization whose central principle is 'retrieval is not transfer': prior optimization strategies retrieved from an archive are treated only as target-side hypotheses, and they become reusable knowledge only after experimental validation under a frozen verifier. The framework couples an Optimization Fingerprint (a probe-based response Jacobian), an Interventional Repair Geometry (IRG) over those fingerprints, a compatibility certificate that licenses transfer-prioritized trials, and a continual evidence memory. Experiments cover within-backbone repairs on Ctrl-World, Cosmos, and RoboCoin, plus a leave-one-backbone-out transfer study in which certified reuse is claimed to reach the first strict-gate positive in 3.6 trials and 96 GPU-hours versus 11.2 trials and 312 GPU-hours for cold start, and to reduce negative transfer from 0.34 to 0.06 while achieving 0.83 sign agreement. The appendices provide extensive ablations, including certificate leave-one-term-out, planted-fault audits, screening recall, and a selector-source ablation that honestly reports that the selector is not the source of the gains.","tokens_in":24779,"tokens_out":4121,"duration_ms":41298,"significance":"If the empirical claims hold, the paper makes a useful contribution to autonomous research-agent evaluation: it replaces opportunistic reuse with an evidence-licensed reuse protocol, and it makes several design choices that raise the credibility of the evaluation. Particularly strong points are the frozen verifier per campaign, paired trials with common seeds, the frozen per-fold archive in the leave-one-backbone-out protocol, the explicit reporting of null and harmful results, and the control-plane audits under planted faults. The explicit negative result that the online selector is not the source of the gains, and the finding that ungated probe growth does not help, are valuable calibration signals for the community. The central quantitative claim, however, depends on a metric, the negative-transfer rate, that is never defined in the manuscript; until that is fixed and per-fold data are reported, the headline reduction from 0.34 to 0.06 cannot be independently checked.","major_comments":[{"comment":"The negative-transfer rate is never defined. Section 4.1 defines trials, C_first+, and overhead, but no formula, numerator, denominator, or counting rule is given for the 'Neg. transfer' column in Table 4 or for the same column in Tables 7 and 14. The abstract's headline claim ('negative transfer from 0.34 to 0.06') is exactly this undefined quantity, so the central empirical claim is unverifiable as written. This is not a cosmetic omission: the unconditional-reuse and certified-reuse rows may be measuring different populations, since certified reuse screens every candidate before admitting it while unconditional reuse does not. Please define the rate precisely (per campaign? per trial? per admitted trial? how are abstentions counted?), report per-fold and per-seed counts, and give confidence intervals or raw seed-level values. With only a few LOBO folds and three seeds, a single fold can dominate the aggregate, and the current point estimates do not allow the reader to assess that risk.","section":"§4.5, Table 4"},{"comment":"The leave-one-backbone-out procedure is underspecified at the level needed to audit the central transfer result. The text says Table 4 'aggregates the rotated held-out folds with fresh target IDs' and that 'the two named source backbones are representative, not exhaustive,' but it never lists the folds or reports per-fold trials, GPU-hours, and negative-transfer values. Appendix Table 5 inventories shared prior archives, but it does not give the fold structure. Because the speedup claim (3.6 vs. 11.2 trials, 96 vs. 312 GPU-h) is the paper's principal quantitative evidence for the 'retrieval is not transfer' principle, the reader needs to know exactly which backbones were held out in each fold and whether the result is stable across folds. Please include a fold-by-fold table and state how source campaigns were selected for each fold.","section":"§4.5, LOBO protocol"}],"minor_comments":[{"comment":"The related-work section contains duplicated text: the sentence about SWE-agent and Agent Laboratory appears twice, and the sentence beginning 'Benchmarks such as CORE-Bench and PaperBench also expose...' also appears twice. Please deduplicate.","section":"§2, Related Work"},{"comment":"After the abstract, the text contains the fragment 'Reflect/Continue to the next step' followed by unrelated strings such as '销售财务市场营销HR0'. This appears to be a formatting artifact and should be removed before resubmission.","section":"Abstract / front matter"},{"comment":"The compatibility certificate is stated twice, as Eq. (7) and as Eq. (15), with identical content. Please keep one formal definition and cross-reference the other, or state explicitly that the second is a restatement for the appendix.","section":"§3.3 and §A.4"},{"comment":"The system name is rendered inconsistently as 'VERDI', 'Verdi', and 'VerdiWM'. Please use a single convention throughout the text and figures.","section":"Throughout"},{"comment":"The phrase 'as their diversity increases' is grammatically awkward and appears to be an editing remnant; please revise.","section":"§4.1, Settings and metrics"}],"recommendation":"major_revision","confidential_remarks":"The paper's methodology and appendices are unusually careful, and the honest negative results (selector not the source of gains, ungated probe growth not helpful) suggest the authors are not trying to hide unfavorable findings. The blocking issue is narrow but central: the negative-transfer rate is undefined and the LOBO fold structure is not auditable. I would not reject on these grounds, since the concern is fixable by adding definitions and per-fold tables. If those are provided, the central claim may become verifiable; as written, the headline numbers cannot be independently checked."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the important thing: this is a serious systems paper with a sensible central principle and better-than-usual internal auditing. The core idea—that a strategy validated on one world model is at most a hypothesis for another until re-verified—is correct and worth building on. The fingerprint + IRG distance is a plausible way to relate models by behavior rather than architecture, and the collision-driven probe evolution is a nice way to close the loop. Credit where due: the paper reports several negative results plainly (the selector isn't the source of gains, ungated probe growth doesn't help, the control plane trades false conclusions for missed genuine effects). The planted-fault audit in Appendix B.5 is a genuinely useful methodology. If I were refereeing, I'd push for that to be highlighted more.\n\nThe soft spot is exactly what the stress-test note flags: the headline negative transfer reduction from 0.34 to 0.06 is never defined. No formula, no numerator/denominator, no per-fold counts. Since that number is in the abstract, the central empirical claim is unverifiable as written. It could easily be measuring different populations in the two rows—certified reuse screens before admission; unconditional reuse doesn't. Add to that no code, no confidence intervals, and a development suite (ACWM-Phys) built by the authors, and external validity is thin. The paper itself is honest about several of these limits, but that doesn't fix the missing definition.\n\nIs the architecture coherent? Yes. The internal ablations are consistent with the story, and the leave-one-backbone-out protocol is a reasonable attempt at breaking circularity. But the evaluation is still self-run, and the number of folds is small.\n\nWho is this for? People working on autonomous research agents, world model optimization, and continual learning. The principle is worth discussing even if the numbers don't fully land.\n\nRecommendation: send it to peer review, but require a clear definition of negative transfer and per-fold data before acceptance. If the code and definitions come out, the core claim is replicable. As it stands, it's a strong framework paper with an unverifiable headline.","headline":"A serious framework paper with a sensible principle and unusually honest ablations, but the headline negative-transfer number is undefined and the evaluation is not independently checkable as written.","tokens_in":25430,"tokens_out":2208,"would_cite":false,"duration_ms":19256,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes that a repair validated on one world model is only an optimization hypothesis for another, and that target-side verification under a frozen evaluator is what converts a retrieved strategy into transferable knowledge.","keywords":["continual world model optimization","retrieval is not transfer","optimization fingerprint","interventional repair geometry","transfer certificate","probe evolution","evidence-licensed optimization","world model"],"falsifier":"Find two world models whose IRG distance is below the certificate's neighbor threshold yet for which a validated skill has opposite signed effects on the target after confirmation; a systematic collection of such collisions at a rate well above the certificate's stated 0.06 negative transfer, on held-out backbones not used to build the archive, would show that fingerprint proximity does not license the transfer claim.","tokens_in":1652,"feed_emoji":"🔄","tokens_out":9368,"duration_ms":116328,"temperature":0.7,"pith_summary":"The paper argues that reusing an optimization strategy across world models is never justified by the source model alone: a strategy validated on one model is only a hypothesis for another, and it becomes transferable knowledge only when the target model itself confirms it. To make that principle operational, the paper builds VERDI, a continual optimization loop that measures each world model's response to shared inference-time probes, retrieves prior repairs by geometric proximity, and refuses to admit any retrieved repair as reusable evidence until a frozen target-side verifier has confirmed it. The payoff claimed is concrete: on held-out cross-backbone transfers, certified reuse reaches the first strict-gate positive in 3.6 trials (96 GPU-hours) versus 11.2 trials (312 GPU-hours) cold start, reduces negative transfer from 0.34 to 0.06, and predicts transfer sign with 83% accuracy. If correct, the framework turns world-model optimization from a sequence of isolated campaigns into a continually accumulating knowledge base whose failures refine its own diagnostic language.","feed_headline":"Repair recipes don't transfer until verified on the target","feed_subtitle":"Certified reuse cuts negative transfer from 0.34 to 0.06 while staying as fast as unconditional reuse.","key_machinery":"The central object is the Optimization Fingerprint $φ_x = (J_x, G_x, Σ_x)$, where $J_x$ is the paired finite-difference response Jacobian of the world model under shared inference-time probes, $G_x$ is the induced weighted repair metric, and $Σ_x$ is bootstrap uncertainty. The Interventional Repair Geometry (IRG) is the map of these fingerprints with an uncertainty-weighted distance $d_{IRG}$; the transfer-prioritization certificate (a product of six binary checks: compilability, support overlap, evidence sufficiency, response alignment, sign agreement, and lower-bounded gain over cold start) decides only whether a candidate may be tested, never that it transfers. Counterexample-driven probe evolution completes the loop by adding new probe directions only when IRG collisions show the current basis aliases distinct repair states and only when nested replay shows the addition reduces selection regret.","core_discovery":"The paper's central discovery is the dissociation between retrieval and transfer. VERDI characterizes each world model by a fingerprint built from paired, dose-calibrated, inference-only probe responses (a response Jacobian, a weighted metric, and bootstrap uncertainty). Retrieval ranks candidate repairs by distance in the Interventional Repair Geometry (IRG), but every candidate must pass a six-term compatibility certificate and then be validated under the target's frozen verifier before it is written into memory; positive, null, harmful, and interaction outcomes are all stored. The system also treats failed transfers as evidence that the probe basis is underspecified: when two nearby fingerprints yield opposing validated effects for the same repair, that collision triggers a regret-gated probe evolution that refines the geometry itself. The claim is therefore not just that verification prevents harm, but that the diagnostic representation is learnable and improves across campaigns.","pith_inferences":["If IRG distance is treated as a metric over optimization behavior, the same machinery could rank not just repairs but whole optimization goals, letting one campaign decide which objectives are safe to pursue jointly.","The fingerprint's dependence on a fixed probe library suggests a natural stress test: repeatedly re-learning probes from scratch and comparing convergence would reveal whether the representation self-improves or merely grows.","Collision-driven probe evolution is analogous to active learning on the geometry; one could make the regret gate adaptive per campaign rather than global.","The 0.83 sign agreement predicts transfer sign on held-out pairs; a natural extension would be calibrated probabilities, so abstention rates can be set by a user's tolerance for harmful transfers."],"forward_implications":["World-model optimization can accumulate: a single verified repair, with its boundary annotations, becomes reusable evidence for future models without rediscovery.","Negative transfer is filterable: the certificate can cut harmful reuse by more than fivefold while keeping the speed of unconditional reuse.","Failures are productive: every IRG collision is a work order for a new diagnostic probe, so the system's representation of model behavior improves with experience.","Effect records are paired change vectors, not scores: a repair that improves one metric while violating a protected coordinate is stored as a boundary, so trade-offs are visible before any transfer decision.","Transfer is licensed, not assumed: retrieved strategies only change candidate order; the frozen target-side verifier remains the sole authority for accepting a claim."],"supporting_citations":[{"why":"Supplies the first main backbone, Ctrl-World, whose physical-consistency failure motivates the latent-motion-prior repair.","marker":"(Guo et al., 2025)"},{"why":"Supplies the Cosmos world model family used in the cross-backbone transfer experiments and the REP/self-forcing repairs.","marker":"(NVIDIA et al., 2025)"},{"why":"Defines the WorldArena validity-gate and protected-metric structure that the frozen verifier instantiates.","marker":"(Shang et al., 2026a)"},{"why":"Is the closest prior agentic loop; VERDI extends its verify-before-improve idea to cross-model transfer with explicit abstention.","marker":"(Xie et al., 2026)"},{"why":"Provides the DINOv2 representation-guidance repair primitive used in the Cosmos-Predict 2 campaign.","marker":"(Oquab et al., 2024)"}],"fun_headline_variants":["Retrieval isn't transfer—VERDI verifies before reusing fixes","VERDI: Validate repairs on target; retrieval only ranks hypotheses","Negative transfer drops to 0.06 with VERDI's target-side validation","Retrieval ≠ transfer: VERDI's certificate makes reuse evidence-based","Test-before-reuse: VERDI cuts search cost 68%, negative transfer 0.06"],"cache_read_input_tokens":27392,"weakest_assumption_plain":"The argument rests on the premise that two world models which respond similarly to a fixed set of inference-time probes will also respond similarly to optimization repairs; if the probe basis misses a transfer-relevant distinction, retrieval becomes actively misleading.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval isn't transfer—VERDI verifies before reusing fixes","VERDI: Validate repairs on target; retrieval only ranks hypotheses","Negative transfer drops to 0.06 with VERDI's target-side validation","Retrieval ≠ transfer: VERDI's certificate makes reuse evidence-based","Test-before-reuse: VERDI cuts search cost 68%, negative transfer 0.06"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00098,"raw_usage":{"total_tokens":4162,"prompt_tokens":947,"completion_tokens":3215,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":3113}},"tokens_in":563,"tokens_out":3215,"duration_ms":22129,"temperature":1.0,"reasoning_tokens":3113,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:28:24.208192+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find two world models whose IRG distance is below the certificate's neighbor threshold yet for which a validated skill has opposite signed effects on the target after confirmation; a systematic collection of such collisions at a rate well above the certificate's stated 0.06 negative transfer, on held-out backbones not used to build the archive, would show that fingerprint proximity does not license the transfer claim.","supporting_citations":[],"review_version":1}