{"id":"182efad4-0526-4f62-81dc-684e6c9335e6","arxiv_id":"2506.13536","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Camera pose and spatial arrangement are the most important dimensions for both collecting diverse robot data and retrieving useful subsets, and aligned retrieval from DROID beats full-dataset co-training on real tasks.","lead":"This paper uses a procedural simulator to test which types of variety in robot demonstration data matter most for downstream policies, and finds camera pose and object position are key while textures matter little. It also shows that retrieving aligned subsets from the DROID dataset improves real-robot success rates compared with training on the full dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Full-DROID co-training baseline (0% on all six tasks) is likely an optimization artifact; the central 'retrieval beats full training' conclusion depends on an unablated baseline.","rationale":"The reader's conditional verdict and weakest assumption align with the main risk. The simulation study is large and internally consistent, with controlled DV manipulations and multiple policy backbones (BC-RNN, BC-Transformer, Diffusion Policy), which supports the collector-side claims about camera pose and spatial arrangement being important. The retriever-side claim, however, is the paper's headline and is only as strong as the full-DROID baseline. That baseline's uniform 0% is a red flag: in behavior cloning with 50% target sampling, a policy that simply ignores co-training data should approximately reproduce target-only behavior, not fall to 0%. The absence of training curves, epoch ablations, and matched-size random baselines makes the 'too much diversity' explanation an assertion rather than a demonstrated mechanism. I also note the abstract's 'up to 70%' is inconsistent with Fig. 5's absolute gaps, which further weakens the quantitative claim. The proposed experiment would settle the concern: if longer training or a random subset of equal size achieves non-trivial success, the paper's central comparison is invalid and the conclusion should be re-scoped to 'retrieval helps under a fixed, undertrained full-data baseline.' If the 0% persists under all tested recipes and a matched-size random baseline also fails, the diversity-instability story is supported. Because the current evidence does not resolve this, the existing CONDITIONAL verdict should remain; my read does not change it.","tokens_in":22196,"tokens_out":5216,"duration_ms":52042,"concrete_test":"Run a single controlled experiment on the real-robot retriever suite. Arm A: train the full-DROID baseline for 600 epochs (and, if compute allows, 1200) with the identical Diffusion Policy recipe, recording per-task success and training loss. Arm B: train a matched-size random baseline sampled uniformly from all DROID demos, with the same number of demonstrations as the best retrieved set for each task (e.g., the obj/skill+objSpat set), same omega and epochs. If Arm A or Arm B achieves non-zero success on any task, the Fig. 5 zero-collapse is not attributable to dataset diversity per se, and all reported retrieval gains must be recomputed against the stronger baseline. Report per-task 95% confidence intervals over the 20 rollouts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central real-world claim is that retrieval on DROID outperforms co-training on the entire dataset by up to 70% (Abstract, Sec. 5.4.2). The comparison rests entirely on Fig. 5, where every full-DROID policy scores 0% while the same target-only policies score 5-60%. A 0% result across six diverse tasks, when the target-only policy succeeds, is not explained by 'too much diversity in co-training datasets can potentially lead to unstable training' alone. The paper never ablates the training recipe on the full dataset: it reports one training duration (300 epochs, Appendix E), one re-balancing weight (omega=0.5, Appendix E.1), and no learning curves, losses, or checkpoint evaluations for full DROID. Because the full dataset is orders of magnitude larger and more heterogeneous than the retrieved subsets, the 0% could equally come from insufficient optimization, action-space/observation mismatch, or unstable training under a fixed learning rate and batch size. With N_C >> N_T, omega=0.5 sampling already forces half of each batch from the target, so a complete collapse to 0% is especially suspicious. No matched-size random-subset baseline is provided, so 'retrieval outperforms full training' is not separated from 'small training sets are easier to optimize.' Additionally, the abstract's 'up to 70%' does not match Fig. 5, where the largest gap is 85 percentage points (serve snack: 85 vs. 0). This is the load-bearing weakness: if the full-DROID baseline can be made to succeed with more epochs, tuning, or a comparable-size random subset, the headline retrieval advantage shrinks or reverses, while only the simulation-side DV ranking remains.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies how the composition of robot manipulation demonstration datasets affects downstream imitation learning. The authors build a procedural data-generation framework, MimicLabs, that synthesizes nearly one million trajectories across more than three thousand task instances in eight simulated environments, with controllable variation along camera pose, object and table texture, object and receptacle spatial arrangement, and motion primitives. Using this framework, they run 'collector' experiments that vary one dimension of variation (DV) at a time to determine which types of diversity matter during data collection, and 'retriever' experiments that compare alignment-based subset selection against training on full variation. Their central simulation findings are that camera pose and spatial arrangement are the most important DVs for both diversity and alignment, while object texture has minimal impact. They then transfer these insights to a real robot: collector experiments on three tasks reproduce the simulation trends, and retriever experiments on DROID show that retrieving demonstrations with aligned camera pose or object location can improve success rates over target-only training. The headline claim is that these retrieval strategies 'consistently outperform existing training strategies by up to 70%', where the existing strategy is co-training on the entire DROID dataset.","tokens_in":22537,"tokens_out":10502,"duration_ms":94492,"significance":"If the central claims hold, the paper would give dataset collectors actionable guidance (prioritize camera-pose and spatial diversity; do not spend effort on texture variation) and give practitioners a cheap way to improve downstream performance (retrieve aligned subsets from large datasets like DROID rather than co-training on everything). The procedural generator itself, with roughly one million controlled trajectories, is a substantial contribution, and the authors make a genuine effort to validate simulation insights on a real robot whose setup did not participate in DROID collection, which reduces data-pollution concerns. The counterfactual retriever experiments (removing the relevant skill/object) are a well-designed probe of the value of skill alignment. However, the quantitative headline is currently under-supported: the full-DROID baseline collapses to 0% on all six real tasks with no diagnosis, the 'up to 70%' figure does not match the reported numbers, and none of the central comparisons include variance estimates or multiple seeds. These issues bear directly on the paper's main conclusion about retrieval versus full-dataset training.","major_comments":[{"comment":"The full-DROID co-training baseline achieves 0% success on all six real-world tasks, while the same target-only policies reach 5–60%. The paper attributes this to 'too much diversity' causing unstable training, but no evidence rules out a training artifact. Only one training recipe is reported (300 epochs, ω=0.5, LR 1e-4, batch 128; Appendix E), with no learning curves, no checkpoint analysis, no hyperparameter search, and no matched-size random-subset baseline. Because the claim that 'retrieval beats full-dataset training' rests entirely on this comparison, the authors should add ablations of the training recipe on full DROID and a random subset of DROID of size comparable to the retrieved sets. As is, the 0% baseline is consistent with optimization failure rather than dataset diversity.","section":"§5.4.2, Fig. 5"},{"comment":"The claim that retrieval strategies 'outperform existing training strategies by up to 70%' is not supported by Fig. 5. The largest gain in Fig. 5 is 80 percentage points over the target-only policy (serve snack: 5%→85%) and 85 points over the full-DROID baseline; no task shows a 70% relative or absolute improvement over target-only. The authors need to specify the exact comparison underlying the 70% figure or correct it.","section":"Abstract and §5.4.2"},{"comment":"The central quantitative comparisons are made without variance estimates or multiple seeds. Real-world policies are evaluated on 20 rollouts and simulation policies on 30 rollouts, but no standard errors or confidence intervals are given. For 20 rollouts, a 50% success rate has a standard error of about 11 percentage points, so differences of 5–15 points among retrieval strategies (e.g., wipe board: 55 vs 65; put marker: 20 vs 35) are not clearly significant. The authors should report multiple seeds or confidence intervals before making fine-grained claims such as 'a 25% increase in performance when co-training with aligned camera poses.'","section":"§5.4.2 and Appendix E"},{"comment":"The simulation results do not show the collapse that the real-world full-DROID baseline exhibits. In Table 3, the 'all' strategy (full combinatorial variation) achieves 90%, 73.3%, 43.3%, 33.3%, and 36.7% success on the five benchmark tasks, respectively, rather than 0%. The paper's explanation that 'visuomotor policies are inherently prone to unstable training in the presence of heterogeneous data' is therefore not supported by the authors' own simulation data, and the real-world 0% result likely reflects a DROID-specific artifact. The authors should reconcile this discrepancy or provide a concrete analysis of the full-DROID training runs (e.g., loss curves, action statistics).","section":"§5.3 vs §5.4.2"}],"minor_comments":[{"comment":"The statement '500 epochs (250k gradient steps)' is inconsistent with the reported batch size of 32 and dataset sizes of about 1,000 demos; 500 epochs would correspond to roughly 15k steps. Please verify the number.","section":"Appendix E"},{"comment":"The metadata pipeline (syntactic parsing of language instructions, LLaVA color detection, and gripper-based object localization) is central to the DROID retrieval results, but no validation or error-rate analysis is reported; a small human-annotated evaluation of the object/color/location labels would substantially strengthen confidence in the retrieval.","section":"Appendix J"},{"comment":"The column layout is garbled for the collector experiments (e.g., the target-only column and the co-training columns are not clearly separated), making it difficult to verify the claims about baseline versus camPose diversity. Please reformat these tables.","section":"Tables 1 and 9"},{"comment":"The claim that 'empirically, we found no difference in performance between models trained for 300 epochs and those trained for 600' is presented without supporting data; please provide the comparison or qualify the statement.","section":"Section 5.4 and Appendix E"},{"comment":"The DROID retrieval thresholds (e.g., 20 cm in X/Y and 10 cm in Z for camera pose, and the 60×60×30 cm cuboid for object spatial) are chosen without sensitivity analysis; since these thresholds define the retrieved subsets, a brief sensitivity check would make the results more robust.","section":"Section 5.4.2"},{"comment":"The phrase '1000s of hours' is informal; consider 'thousands of hours.'","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper has a solid simulation study and a valuable real-robot transfer, but the two load-bearing issues (unablated 0% full-DROID baseline and the unsupported 'up to 70%' figure) need to be resolved before publication. I would not reject, but the authors must add diagnostic experiments and correct the quantitative claims. The paper fits the scope of the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2506.13536. First, the simulation-side study is the real contribution: a large, controlled dataset-composition experiment with a sensible ranking of which dimensions of variation matter. Camera pose and spatial arrangement come out as the high-value DVs; object texture barely matters. That is actionable and, as far as I can tell, new as a systematic result. Second, the real-world 'retrieval beats full DROID' headline is not supported by the evidence as presented, because the full-DROID baseline collapses to 0% on all six tasks with no ablation of the training recipe.\n\nWhat the paper does well: it builds a procedural generator on BDDL + MimicGen that can produce ~1M trajectories with independent control over camera pose, textures, and spatial layouts, then runs a large battery of collector- and retriever-perspective experiments in simulation with two policy classes. The DV ranking is consistent across tasks and policies, and the real-world collector experiments line up with it. That is a useful result for anyone planning data collection or retrieval pipelines.\n\nThe soft spots are mostly in the DROID retriever experiment. The full-DROID co-training baseline gets 0% everywhere while the target-only policy gets 5–60%. That looks like an optimization artifact, not a statement about data diversity. The paper mentions unstable training but never shows learning curves, losses, or a matched-size random-subset baseline, and it uses one training duration and one rebalancing weight. Without those, 'retrieval beats full training' is not separated from 'small training sets are easier to optimize.' The abstract's 'up to 70%' also doesn't match the table, where the raw gap is 85 percentage points on serve snack. And the paper reports no variance across seeds, with 20 real-robot rollouts per task and peak checkpoint selection in simulation. These are fixable in revision, but they are load-bearing for the strongest claim.\n\nBottom line: this is a solid simulation study with an overclaimed real-world coda. It deserves a serious referee and a reading group slot. If I were reviewing, I'd ask for baseline ablations on the full DROID co-training, variance or seeds, and a corrected abstract number, and I'd expect those to change the strength but probably not the direction of the simulation-side findings.","headline":"Solid simulation-side DV ranking; the DROID retriever claim rests on an unexplained 0% baseline that needs ablations before it convinces.","tokens_in":23107,"tokens_out":2940,"would_cite":true,"duration_ms":28257,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Camera pose and spatial arrangement, not object texture, are the decisive dimensions when learning robot policies from large demonstration datasets.","keywords":["robot manipulation","imitation learning","dataset composition","co-training","dataset retrieval","dimensions of variation","simulation-to-real transfer","DROID"],"falsifier":"Train the same Diffusion Policy on the entire DROID dataset using the exact recipe and epoch budget used for the retrieved subsets, with action-space normalization and batch weighting matched, and measure whether success on the six real tasks rises materially above 0%; if full-data co-training then matches or beats the retrieved subsets, the paper's central retrieval claim is refuted.","tokens_in":22021,"feed_emoji":"🤖","tokens_out":5110,"duration_ms":43438,"temperature":0.7,"pith_summary":"This paper asks what actually matters when training robot manipulation policies from large demonstration datasets: which kinds of variety should collectors add, and which demonstrations should practitioners retrieve. Using a procedural generator that produces nearly a million simulated trajectories with controlled composition, the authors find that diversity and alignment in camera poses and in object and receptacle spatial arrangements drive downstream success, while object texture has little effect. In real-world tests on seven tasks, retrieval guided by these insights consistently beats co-training on the entire DROID dataset by up to 70%.","feed_headline":"Retrieval beats full robot datasets by up to 70 percent","feed_subtitle":"A controlled composition study shows camera and spatial alignment matter most for imitation learning.","key_machinery":"The load-bearing machinery is the MimicLabs dataset generator, which combines BDDL task specifications with MimicGen source-demonstration decomposition to procedurally instantiate over 3,000 task instances and roughly one million trajectories in eight simulated labs. This lets the authors independently control dimensions of variation (DVs) such as camera pose, object and table texture, and object and receptacle spatial arrangement, and construct target and co-training distribution pairs that are aligned or misaligned along any single DV. These datasets then serve as a controlled testbed for collector-side and retriever-side questions, and the same DV axes are used to query real-world datasets such as DROID via language parsing, object-color annotation, and gripper-based position heuristics.","core_discovery":"The paper's central claim is that the utility of a large multi-task robot dataset is governed by a small set of measurable dimensions of variation, with camera pose and spatial arrangement as the decisive ones for both collection diversity and retrieval alignment. In simulation, co-training that retrieves demonstrations aligned with the target camera and object locations outperforms co-training on the full combinatorially varied dataset, sometimes while using one-tenth to one-hundredth of the data. On a real robot, every full-DROID co-training baseline failed all six evaluated tasks, whereas the best aligned retrievals succeeded on 65–85% of rollouts, a gap the authors attribute to instability from excessive diversity in the full dataset.","pith_inferences":["A plausible extension would test whether exactly matching camera extrinsics outperforms the coarse 20 cm binning used in the DROID retrieval, since binning may discard fine viewpoint information that matters for visuomotor policies.","The 0% full-DROID baseline could partly reflect optimization instability rather than diversity alone, because the paper does not ablate the training recipe (epoch count, action-space normalization, loss weighting) on the full dataset; the claimed gap is therefore strongest when that recipe is held fixed.","The DV formalism suggests that dataset metadata standards (language instructions, camera extrinsics, object identities, manipulation timesteps) are themselves a form of dataset quality, since retrieval quality is bounded by annotation quality."],"forward_implications":["Dataset collectors should spend effort on widening camera pose coverage and object and receptacle spatial reset ranges, not on texture variety.","Practitioners should retrieve subsets aligned with the target camera and spatial distribution rather than co-train on the full available dataset, since aligned retrieval beats full co-training even with far fewer demos.","Retrieval results in simulation transfer qualitatively to real robots, so simulation can serve as a low-cost testbed for dataset composition decisions before large real-world collection.","When alignment along camera pose is impossible, retaining high diversity in camera pose still helps, while low-diversity misalignment is the worst case.","Quality often beats quantity in co-training, with a few thousand aligned demonstrations sometimes outperforming hundreds of thousands of heterogeneous ones."],"supporting_citations":[{"why":"Provides DROID, the real-world large-scale dataset used to validate retrieval strategies and to form the full-dataset co-training baseline.","marker":"(Khazatsky et al., 2024)"},{"why":"MimicGen supplies the demonstration-synthesis machinery that scales a small set of human demos into roughly one million trajectories.","marker":"(Mandlekar et al., 2023)"},{"why":"LIBERO's BDDL parsing framework is the basis for procedural task specification with controllable dimensions of variation.","marker":"(Liu et al., 2023a)"},{"why":"Diffusion Policy is the real-world policy class whose success rates measure the retrieval and co-training comparisons.","marker":"(Chi et al., 2023)"},{"why":"BC-RNN is the simulation policy backbone and the prior 'what matters' study this work explicitly extends.","marker":"(Mandlekar et al., 2021)"},{"why":"Motivates the ω-weighted target/co-training batch sampler that prevents the large co-training set from dominating the behavior-cloning objective.","marker":"(Hejna et al., 2024)"}],"fun_headline_variants":["Aligned retrieval beats full robot datasets by 70%","Camera and spatial alignment matter most in robot demo data","Retrieval beats full dataset in robot manipulation learning","Matching camera and layout beats using all robot data","Less data, better results: align camera and spatial setup"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that retrieval beats full-dataset training rests on the assumption that the full-DROID baseline's 0% success is caused by too much data diversity, not by a fixable training problem such as mismatched action space or insufficient epochs.","fun_headline_variants_meta":{"raw":{"variants":["Aligned retrieval beats full robot datasets by 70%","Camera and spatial alignment matter most in robot demo data","Retrieval beats full dataset in robot manipulation learning","Matching camera and layout beats using all robot data","Less data, better results: align camera and spatial setup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00051,"raw_usage":{"total_tokens":2481,"prompt_tokens":944,"completion_tokens":1537,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":1459}},"tokens_in":560,"tokens_out":1537,"duration_ms":11546,"temperature":1.0,"reasoning_tokens":1459,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:58:25.526199+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same Diffusion Policy on the entire DROID dataset using the exact recipe and epoch budget used for the retrieved subsets, with action-space normalization and batch weighting matched, and measure whether success on the six real tasks rises materially above 0%; if full-data co-training then matches or beats the retrieved subsets, the paper's central retrieval claim is refuted.","supporting_citations":[],"review_version":2}