{"id":"7d6221db-a8f2-4759-88d4-e079a252a2cf","arxiv_id":"2607.25593","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Legacy robot demonstrations become useful only after the upgraded robot crosses a task-dependent competence threshold, yielding an inverted-U benefit pattern.","lead":"A robotics study finds that old demonstration data from a previous hardware generation does not help an upgraded robot until the new robot has already learned the task somewhat, after which it suddenly becomes very useful, then fades again. The finding gives engineers a practical rule for when to keep collecting new data versus reusing legacy data during hardware upgrades.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Phase I→II jump in flower insertion compares 4.31 h early data with a 15.60 h quality-refined batch; the threshold claim is confounded by target-data quantity/quality, and the watering sweep's three budgets are too coarse to establish a sharp transition.","rationale":"I agree with the reader's conditional verdict. The concern is load-bearing because the abstract's flagship numbers are the 10→10 vs 23.3→86.7 contrast; if that contrast is confounded, the headline 'emergent transfer' loses its most striking real-robot evidence. The watering experiment is a genuine controlled validation and prevents me from moving to REJECT: it varies only new-hardware data budget and reproduces the three phases at 0.5/1.5/8 h. However, it is a single task, measured on two sub-stages, and the three budgets are too coarse to localize a threshold. The theory does not close the gap: Assumptions 1 and 4 (monotone coupling, η(1−SR)) are not measured; Appendix C.4 concedes per-source gradient alignment was never recorded; C.5 concedes the H(T)-τ relation is not a validated scaling law. The dense budget sweep on the original task is the minimal experiment that would either rescue or refute the central claim.","tokens_in":18067,"tokens_out":6377,"duration_ms":63155,"concrete_test":"Run a single quality-controlled Gen-2 flower-insertion dataset and split it into equal-quality subsets of, e.g., 2, 4, 6, 8, 12, and 16 h. For each budget, train a standalone policy and a co-trained policy using the same 17.10 h Gen-1 legacy set, same sampling ratio and training budget. Measure ΔSR with n=60 trials per condition. Fit ΔSR as a function of standalone SR with both a step/threshold model and a smooth polynomial/GP model. If ΔSR rises continuously (e.g., at least 3 of the 6 points fall in the 0–20% SR region with intermediate positive gains) or the best-fit threshold has a wide credible interval (>15 SR points), the sharp phase-transition claim fails. If ΔSR is near zero below ~20% and jumps by >40 points between adjacent budgets, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing evidence for the Phase I→II transition is the flower-insertion contrast in Tables 2 and 3: at 10.0% standalone SR co-training gives ΔSR=0, while at 23.3% it gives +63.4. But these two points are not comparable: the 10% condition uses 4.31 h of 'early' Gen-2 data, while the 23.3% condition uses a separately collected 15.60 h 'quality-refined' batch (Table 1). The refined batch differs in operator training, per-trajectory quality filtering, and pose tolerance (§3), so the jump in ΔSR may be caused by the target dataset being larger/cleaner rather than by the target policy crossing a competence threshold τ(T). The held-out watering sweep (Table 5) does control dataset quality, but it has only three budgets (0.5/1.5/8 h), with no intermediate baseline around the predicted threshold; it shows a peaked curve, not a sharp sign change. Because Definition 1 (Eq. 2) defines τ(T) as the infimum SR at which expected gain becomes positive, a nonzero gain above any observed sign-change point is guaranteed by construction, and Eq. 5 is a piecewise fit unless τ is predicted independently. Thus the central 'emergent/grokking-like' claim rests on a two-point, confounded comparison plus a three-point sweep.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies cross-configuration robot learning on a wheeled humanoid platform with two hardware generations. It compares policies trained only on new-hardware (Gen-2) demonstrations with policies co-trained on legacy Gen-1 plus Gen-2 data, and examines how the co-training gain ΔSR = SR_co-train − SR_single varies with the standalone success rate SR. The authors report a three-phase pattern: no measurable gain at low SR, large gains at intermediate SR, and diminishing returns near saturation. They introduce a task-dependent transfer threshold τ(T) and an inverted-U gain law motivated by gradient alignment and residual policy uncertainty, and propose a phase-aware data collection rule: collect new-hardware data until standalone competence crosses the threshold, then co-train with legacy data. Validation on a held-out mobile dual-arm watering task used three new-data budgets (0.5/1.5/8 h) and observed no gain at 0.5 h, +38–40 percentage-point gains at 1.5 h, and no significant gain at 8 h. The paper reports Fisher exact tests and Wilson intervals for all key comparisons.","tokens_in":18500,"tokens_out":6064,"duration_ms":58353,"significance":"If the three-phase pattern holds, it has immediate practical value for hardware iteration: rather than always mixing legacy data, a practitioner should first bring the new configuration to a minimal competence level. The held-out watering experiment is a genuine strength because it varies only the new-hardware data budget while holding legacy data, sampling ratio, optimization, and evaluation fixed, and the statistical reporting is solid. However, the central claim of an emergent, threshold-like transition is not yet robustly established: the flagship flower-insertion contrast is confounded by dataset quantity and quality, the threshold is defined from the same sign-change it is used to explain, and the theory relies on unmeasured assumptions. The paper nevertheless makes a useful empirical observation that co-training benefit is non-monotonic in standalone competence, and it demonstrates a practically relevant data-allocation rule.","major_comments":[{"comment":"The Phase I→II evidence for flower insertion is confounded. Phase I uses 4.31 h of early Gen-2 data; Phase II uses a separately collected 15.60 h quality-refined batch with stricter operator training, per-trajectory quality filtering, and tighter pose tolerances (§3). The gain jump from 0 to +63.4 percentage points could be caused by the target dataset being larger and cleaner, not by the target policy crossing a competence threshold. To support a sharp threshold, the authors should vary the amount of new-hardware data within a single collection protocol (or otherwise control for dataset quality and size) and include intermediate baselines.","section":"§4.1–4.2, Tables 1–3"},{"comment":"The threshold τ(T) in Eq. (2) is defined as inf{SR : E[ΔSR|SR] > 0}, and it then appears as the switch in the inverted-U law of Eq. (5). Consequently, the existence of a sign change above τ is true by construction; a piecewise fit of the form [κ(1−SR)−δ(SR)]·1[SR>τ] will trivially describe any data with a positive-gain region unless τ is predicted independently. The predictive content must come from the complexity–threshold relation τ ≈ α + βH(T), but §C.5 admits that 'these observations are consistent with the proposed monotonic relation but are insufficient to establish a precise scaling law.' The paper should distinguish the fitted description from an independent prediction.","section":"§5, Definition 1 and Eq. (5)"},{"comment":"The theoretical account rests on Assumption 1 (stage decodability ρ_c(T;θ) is monotonically coupled to standalone SR) and Assumption 4 (residual uncertainty H_within = η(1−SR)). Neither is measured. Appendix C.4 concedes that 'direct per-source gradient alignment was not recorded,' so the loss trajectories are only an indirect diagnostic. The statement that gradient alignment 'explains' the phase structure is therefore too strong; it is a plausible post-hoc model. Please either measure the assumed quantities (e.g., probe stage decodability or estimate remaining uncertainty) or explicitly present the theory as a qualitative framework with no direct empirical support.","section":"§5, Assumptions 1 and 4; Appendix C.4"},{"comment":"The held-out watering sweep has only three new-data budgets (0.5/1.5/8 h), with no points between 0.5 and 1.5 h or just above the predicted threshold. A smooth inverted-U function with gradual onset would produce the same three observations; the data cannot distinguish a sharp threshold from a gradual improvement, which is the central claim. Moreover, the positive-gain phase is represented by a single budget (1.5 h), albeit with two sub-stages. Please add an intermediate budget (e.g., 1.0 or 2–3 h) and ideally repeat the sweep to establish reliability.","section":"§6.1, Table 5"},{"comment":"The Phase III saturation comparison for pen insertion is also confounded. The 'Early + refined' condition (32.21 h) is compared to the 'Refined' condition (13.58 h); the change in gain from +26.6 to +8.3 could reflect the larger, mixed dataset rather than the higher standalone success rate alone. Since this is the main evidence for diminishing returns in Phase III, the data-composition difference should be controlled (e.g., use the same union dataset with random subsets of differing size, or compare same-hours refined-only conditions).","section":"Appendix B.2, Table 7"}],"minor_comments":[{"comment":"The '±22% envelope' is not defined. Is it a confidence band, a model-fit residual, or an ad hoc uncertainty range? Please specify how it was computed.","section":"Figure 3"},{"comment":"The term 'grokking-like' is potentially misleading: grokking is a training-time phenomenon, whereas here the transition is across data-budget/standalone-competence levels. Consider using a more neutral term such as 'regime-dependent transfer' or clarify the analogy.","section":"Abstract, Introduction"},{"comment":"After Assumption 5, the text contains garbled strings like '/uni00000013/...'. This appears to be a PDF-extraction artifact. Please ensure the submitted PDF renders the proofs cleanly.","section":"Appendix C"},{"comment":"The phase boundaries are listed as '<15–20%', '20–75%', and '>75%', but Table 3 reports a Phase II result at 71.7% and Phase III at 85%. A fuzzy boundary is fine, but it should be stated as approximate rather than categorical.","section":"Table 4"},{"comment":"The symbol ε_dom is described as 'irreducible configuration-specific mismatch' but is not formally defined in terms of the data distributions or gradients. Please clarify its meaning and units.","section":"Eq. (4)"}],"recommendation":"major_revision","confidential_remarks":"The paper contains a potentially valuable empirical observation, and the held-out watering control is a step in the right direction. However, the headline claim of an 'emergent transfer threshold' is not currently supported by the evidence: the main two-point comparison is confounded by dataset quantity/quality, the threshold is definitionally tied to the sign change, and the theory rests on unmeasured assumptions. I would require either new experiments that control dataset quality and span more budgets, or a substantial softening of the claims (e.g., 'regime-dependent co-training benefit' instead of 'emergent threshold'). If the authors choose the softening path, the paper could still be a solid empirical contribution; in its current form it overstates what the data show."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a genuinely new empirical claim: co-training gain from legacy demonstrations follows an inverted-U with respect to the new hardware's standalone success rate, with no benefit at low competence, a sharp payoff after some threshold, and diminishing returns near saturation. If that pattern holds up, it's a practical rule for robot-learning teams deciding whether to spend hours on new-hardware data or reuse legacy demos.\n\nWhat the paper does well: the held-out watering task. There they vary only the new-hardware data budget (0.5h, 1.5h, 8h) while holding the legacy set and all other procedure fixed, and they get Phase I, II, III results consistent with the prediction. That is real evidence for the qualitative shape. They also report Fisher exact tests and Wilson intervals, which is more than many robotics papers bother with.\n\nThe soft spot is the flagship flower-insertion transition. The 10.0%→10.0% case uses 4.31h of early Gen-2 data; the 23.3%→86.7% case uses a separately collected 15.60h quality-refined batch. The two conditions differ in hours and quality, not just in baseline competence, so the sharp jump is confounded. The watering sweep controls this, but with only three budgets it cannot localize a sharp threshold. The theory is more framing than derivation: Assumptions 1 and 4 (stage decodability monotone in SR, residual uncertainty η(1−SR)) are unmeasured, and Appendix C.4 concedes per-source gradient alignment was never recorded. Definition 1 defines τ(T) as the observed sign-change location, so Equation 5 is partly restatement unless τ is predicted independently. The complexity-based prediction τ≈α+βH(T) is suggestive but not validated.\n\nThe paper earns credit for honesty: it states these limitations itself, calls the scaling law a theoretical prediction rather than a law, and does not oversell the negative-transfer angle. The central question is important, the held-out validation is a genuine attempt at prediction, and the three-phase pattern is plausible even if not nailed down. I'd send it to peer review, with a referee asked to push on a cleaner threshold experiment—more data points around the transition with controlled dataset quality—and a clearer account of what predicts τ before it is observed.","headline":"A practically useful three-phase claim about when legacy robot data starts to help, with real held-out support but a confounded flagship transition and a partly circular threshold definition.","tokens_in":18979,"tokens_out":3914,"would_cite":true,"duration_ms":36424,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Legacy demonstration data from older hardware helps an upgraded robot only after the new configuration reaches a task-dependent competence threshold, with gains rising sharply past the threshold before diminishing near saturation.","keywords":["robot learning","cross-configuration transfer","legacy demonstration data","co-training","transfer threshold","inverted-U gain","behavior cloning","hardware iteration"],"falsifier":"Collect new-hardware demonstration batches for the same Gen-2 flower-insertion task at several intermediate budgets between the 4.3h (10.0% baseline) and 15.6h (23.3% baseline) points, co-train each with the same legacy set, and plot ΔSR against baseline; a smooth upward slope rather than a near-zero-then-jump curve would falsify the phase transition. Independently, recording per-source gradient alignment during such co-training would test whether expected alignment is actually non-positive below the threshold.","tokens_in":17910,"feed_emoji":"🤖","tokens_out":6462,"duration_ms":60183,"temperature":0.7,"pith_summary":"This paper asks when previously collected demonstration data from an older robot configuration begins to benefit a policy for upgraded hardware, after cameras and grippers change. The authors claim that the gain from co-training old and new data is not monotonic: it stays at zero until the new-hardware policy reaches a task-dependent transfer threshold, then jumps sharply before diminishing near saturation. They establish this three-phase pattern across real-robot insertion and grasping tasks, with flower insertion improving from 23.3% to 86.7% once the threshold is crossed, while the same co-training leaves a 10% baseline unchanged. They also provide a theoretical mechanism based on gradient alignment and residual policy uncertainty, plus a phase-aware data collection rule that saved 6.5 hours of new data on a held-out mobile watering task. If correct, the result changes when engineers should merge legacy data: not immediately, only after the new robot has learned enough task structure.","feed_headline":"Legacy data helps only after a new robot clears a skill threshold","feed_subtitle":"Real-robot tests show old data gives zero gain at first, jumps to +63 points after the threshold, then fades.","key_machinery":"The transfer threshold τ(T), defined as the smallest standalone success rate above which co-training with legacy data yields positive expected gain. It is coupled to a stage-decodability order parameter ρ(T;θ): the probability that the policy's internal representation can be decoded into the task's latent stages (approach, grasp, insert). The argument runs through two theorems: a threshold transition in expected gradient alignment ⟨g_S, g_T⟩ (non-positive below τ, positive above) and an inverted-U gain law E[ΔSR|SR] = [κ(1−SR)−δ(SR)]·1[SR>τ(T)]. The stage-decodability proxy connects the unobservable representation to the observable success rate, and residual uncertainty η(1−SR) drives the de","core_discovery":"The central discovery is an emergent transition in cross-configuration co-training: the benefit of adding legacy demonstrations to new-hardware training is governed by the upgraded configuration's standalone success rate. For any task there is a transfer threshold τ(T); below it, co-training yields no measurable gain, above it the gain rises sharply, then declines near saturation. The supporting evidence is the flower-insertion contrast (10.0%→10.0% below threshold vs 23.3%→86.7% above) and the pen-insertion saturation (85.0%→93.3%). The authors model the phenomenon with a stage-decodability order parameter: once the policy's internal representation can reliably separate task stages, legacy","pith_inferences":["If the phase transition is a general property of cross-domain co-training rather than a quirk of this vision-language-action setup, similar inverted-U gains should appear under other embodiment or dataset shifts (e.g., sim-to-real, new action spaces), making the transfer threshold a candidate design principle for generalist robot datasets.","A testable extension: measure per-source gradient alignment during co-training (which the paper did not record) to verify that alignment actually flips sign at τ(T); if alignment stays positive below the threshold, the mechanism would need replacement even if the empirical pattern holds.","The inverted-U suggests an optimal data-mixing schedule: instead of co-training from the start, one could first train on new-hardware data alone until the baseline crosses τ(T), then introduce legacy data — a curriculum for data mixing.","The threshold's dependence on task complexity H(T) implies that easy, coarse-tolerant tasks (like watering) become legacy-ready much sooner than precise insertion tasks, enabling task-level prioritization of scarce new-hardware collection effort."],"forward_implications":["Engineers should not blindly mix legacy and new-hardware data: below the threshold, legacy data yields nothing, so collection effort should go to raising the new configuration's standalone success.","Once the threshold is crossed, co-training can deliver large gains, with flower insertion improving by 63.4 percentage points (23.3%→86.7%).","Data quality affects how quickly a task crosses phases but does not bypass the threshold: refined pen-insertion data reached a higher baseline with fewer hours, and the phase then predicted the co-training gain.","The phase-aware rule cut required new-hardware collection time from 8 hours to 1.5 hours on the held-out watering task, where co-training at 1.5h produced gains of 38–40 points.","The model predicts that τ(T) grows with task complexity H(T)=L(T)log(1/ε(T)), so longer or tighter-tolerance tasks need higher standalone baselines before legacy data helps."],"fun_headline_variants":["Legacy data: zero gain, then +63 points after a skill threshold","Three-phase transfer: old data does nothing, then spikes, then fades","Emergent transfer: legacy data pays off only past a competence threshold","Old demo data: useless below threshold, then a sharp jump in gains","Legacy data only helps after new robots clear a skill threshold"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The sharp transition claim rests on assuming that the new robot's standalone success rate reliably tracks an unmeasured internal quantity — how cleanly its representation separates task stages — so that the threshold is real rather than a side effect of how the two flower-insertion datasets were collected.","fun_headline_variants_meta":{"raw":{"variants":["Legacy data: zero gain, then +63 points after a skill threshold","Three-phase transfer: old data does nothing, then spikes, then fades","Emergent transfer: legacy data pays off only past a competence threshold","Old demo data: useless below threshold, then a sharp jump in gains","Legacy data only helps after new robots clear a skill threshold"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000751,"raw_usage":{"total_tokens":3208,"prompt_tokens":804,"completion_tokens":2404,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":2309}},"tokens_in":548,"tokens_out":2404,"duration_ms":18867,"temperature":1.0,"reasoning_tokens":2309,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T01:58:20.559864+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect new-hardware demonstration batches for the same Gen-2 flower-insertion task at several intermediate budgets between the 4.3h (10.0% baseline) and 15.6h (23.3% baseline) points, co-train each with the same legacy set, and plot ΔSR against baseline; a smooth upward slope rather than a near-zero-then-jump curve would falsify the phase transition. Independently, recording per-source gradient alignment during such co-training would test whether expected alignment is actually non-positive below the threshold.","supporting_citations":[],"review_version":1}