Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

What Matters in Learning from Large-Scale Datasets for Robot Manipulation

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Camera pose and spatial arrangement, not object texture, are the decisive dimensions when learning robot policies from large demonstration datasets.

desk verdict Solid simulation-side DV ranking; the DROID retriever claim rests on an unexplained 0% baseline that needs ablations before it convinces. read the letter →

arxiv 2506.13536 v1 pith:IMJPHJWW submitted 2025-06-16 cs.RO cs.LG

Vaibhav Saxena , Matthew Bronars , Nadun Ranawaka Arachchige , Kuancheng Wang , Woo Chul Shin , Soroush Nasiriany , Ajay Mandlekar , Danfei Xu This is my paper · ORCID
classification cs.ROcs.LG
keywords robotmanipulationimitationlearningdatasetcompositionco-trainingretrievaldimensionsofvariationsimulation-to-realtransferDROID
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what actually matters when training robot manipulation policies from large demonstration datasets: which kinds of variety should collectors add, and which demonstrations should practitioners retrieve. Using a procedural generator that produces nearly a million simulated trajectories with controlled composition, the authors find that diversity and alignment in camera poses and in object and receptacle spatial arrangements drive downstream success, while object texture has little effect. In real-world tests on seven tasks, retrieval guided by these insights consistently beats co-training on the entire DROID dataset by up to 70%.

What carries the argument

The load-bearing machinery is the MimicLabs dataset generator, which combines BDDL task specifications with MimicGen source-demonstration decomposition to procedurally instantiate over 3,000 task instances and roughly one million trajectories in eight simulated labs. This lets the authors independently control dimensions of variation (DVs) such as camera pose, object and table texture, and object and receptacle spatial arrangement, and construct target and co-training distribution pairs that are aligned or misaligned along any single DV. These datasets then serve as a controlled testbed for collector-side and retriever-side questions, and the same DV axes are used to query real-world datasets such as DROID via language parsing, object-color annotation, and gripper-based position heuristics.

What would settle it

Train the same Diffusion Policy on the entire DROID dataset using the exact recipe and epoch budget used for the retrieved subsets, with action-space normalization and batch weighting matched, and measure whether success on the six real tasks rises materially above 0%; if full-data co-training then matches or beats the retrieved subsets, the paper's central retrieval claim is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that the utility of a large multi-task robot dataset is governed by a small set of measurable dimensions of variation, with camera pose and spatial arrangement as the decisive ones for both collection diversity and retrieval alignment. In simulation, co-training that retrieves demonstrations aligned with the target camera and object locations outperforms co-training on the full combinatorially varied dataset, sometimes while using one-tenth to one-hundredth of the data. On a real robot, every full-DROID co-training baseline failed all six evaluated tasks, whereas the best aligned retrievals succeeded on 65–85% of rollouts, a gap the authors attribute to instability from excessive diversity in the full dataset.

Load-bearing premise

The claim that retrieval beats full-dataset training rests on the assumption that the full-DROID baseline's 0% success is caused by too much data diversity, not by a fixable training problem such as mismatched action space or insufficient epochs.

Editorial extensions

If this is right

  • Dataset collectors should spend effort on widening camera pose coverage and object and receptacle spatial reset ranges, not on texture variety.
  • Practitioners should retrieve subsets aligned with the target camera and spatial distribution rather than co-train on the full available dataset, since aligned retrieval beats full co-training even with far fewer demos.
  • Retrieval results in simulation transfer qualitatively to real robots, so simulation can serve as a low-cost testbed for dataset composition decisions before large real-world collection.
  • When alignment along camera pose is impossible, retaining high diversity in camera pose still helps, while low-diversity misalignment is the worst case.
  • Quality often beats quantity in co-training, with a few thousand aligned demonstrations sometimes outperforming hundreds of thousands of heterogeneous ones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A plausible extension would test whether exactly matching camera extrinsics outperforms the coarse 20 cm binning used in the DROID retrieval, since binning may discard fine viewpoint information that matters for visuomotor policies.
  • The 0% full-DROID baseline could partly reflect optimization instability rather than diversity alone, because the paper does not ablate the training recipe (epoch count, action-space normalization, loss weighting) on the full dataset; the claimed gap is therefore strongest when that recipe is held fixed.
  • The DV formalism suggests that dataset metadata standards (language instructions, camera extrinsics, object identities, manipulation timesteps) are themselves a form of dataset quality, since retrieval quality is bounded by annotation quality.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper studies how the composition of robot manipulation demonstration datasets affects downstream imitation learning. The authors build a procedural data-generation framework, MimicLabs, that synthesizes nearly one million trajectories across more than three thousand task instances in eight simulated environments, with controllable variation along camera pose, object and table texture, object and receptacle spatial arrangement, and motion primitives. Using this framework, they run 'collector' experiments that vary one dimension of variation (DV) at a time to determine which types of diversity matter during data collection, and 'retriever' experiments that compare alignment-based subset selection against training on full variation. Their central simulation findings are that camera pose and spatial arrangement are the most important DVs for both diversity and alignment, while object texture has minimal impact. They then transfer these insights to a real robot: collector experiments on three tasks reproduce the simulation trends, and retriever experiments on DROID show that retrieving demonstrations with aligned camera pose or object location can improve success rates over target-only training. The headline claim is that these retrieval strategies 'consistently outperform existing training strategies by up to 70%', where the existing strategy is co-training on the entire DROID dataset.

Significance. If the central claims hold, the paper would give dataset collectors actionable guidance (prioritize camera-pose and spatial diversity; do not spend effort on texture variation) and give practitioners a cheap way to improve downstream performance (retrieve aligned subsets from large datasets like DROID rather than co-training on everything). The procedural generator itself, with roughly one million controlled trajectories, is a substantial contribution, and the authors make a genuine effort to validate simulation insights on a real robot whose setup did not participate in DROID collection, which reduces data-pollution concerns. The counterfactual retriever experiments (removing the relevant skill/object) are a well-designed probe of the value of skill alignment. However, the quantitative headline is currently under-supported: the full-DROID baseline collapses to 0% on all six real tasks with no diagnosis, the 'up to 70%' figure does not match the reported numbers, and none of the central comparisons include variance estimates or multiple seeds. These issues bear directly on the paper's main conclusion about retrieval versus full-dataset training.

major comments (4)
  1. [§5.4.2, Fig. 5] The full-DROID co-training baseline achieves 0% success on all six real-world tasks, while the same target-only policies reach 5–60%. The paper attributes this to 'too much diversity' causing unstable training, but no evidence rules out a training artifact. Only one training recipe is reported (300 epochs, ω=0.5, LR 1e-4, batch 128; Appendix E), with no learning curves, no checkpoint analysis, no hyperparameter search, and no matched-size random-subset baseline. Because the claim that 'retrieval beats full-dataset training' rests entirely on this comparison, the authors should add ablations of the training recipe on full DROID and a random subset of DROID of size comparable to the retrieved sets. As is, the 0% baseline is consistent with optimization failure rather than dataset diversity.
  2. [Abstract and §5.4.2] The claim that retrieval strategies 'outperform existing training strategies by up to 70%' is not supported by Fig. 5. The largest gain in Fig. 5 is 80 percentage points over the target-only policy (serve snack: 5%→85%) and 85 points over the full-DROID baseline; no task shows a 70% relative or absolute improvement over target-only. The authors need to specify the exact comparison underlying the 70% figure or correct it.
  3. [§5.4.2 and Appendix E] The central quantitative comparisons are made without variance estimates or multiple seeds. Real-world policies are evaluated on 20 rollouts and simulation policies on 30 rollouts, but no standard errors or confidence intervals are given. For 20 rollouts, a 50% success rate has a standard error of about 11 percentage points, so differences of 5–15 points among retrieval strategies (e.g., wipe board: 55 vs 65; put marker: 20 vs 35) are not clearly significant. The authors should report multiple seeds or confidence intervals before making fine-grained claims such as 'a 25% increase in performance when co-training with aligned camera poses.'
  4. [§5.3 vs §5.4.2] The simulation results do not show the collapse that the real-world full-DROID baseline exhibits. In Table 3, the 'all' strategy (full combinatorial variation) achieves 90%, 73.3%, 43.3%, 33.3%, and 36.7% success on the five benchmark tasks, respectively, rather than 0%. The paper's explanation that 'visuomotor policies are inherently prone to unstable training in the presence of heterogeneous data' is therefore not supported by the authors' own simulation data, and the real-world 0% result likely reflects a DROID-specific artifact. The authors should reconcile this discrepancy or provide a concrete analysis of the full-DROID training runs (e.g., loss curves, action statistics).
minor comments (6)
  1. [Appendix E] The statement '500 epochs (250k gradient steps)' is inconsistent with the reported batch size of 32 and dataset sizes of about 1,000 demos; 500 epochs would correspond to roughly 15k steps. Please verify the number.
  2. [Appendix J] The metadata pipeline (syntactic parsing of language instructions, LLaVA color detection, and gripper-based object localization) is central to the DROID retrieval results, but no validation or error-rate analysis is reported; a small human-annotated evaluation of the object/color/location labels would substantially strengthen confidence in the retrieval.
  3. [Tables 1 and 9] The column layout is garbled for the collector experiments (e.g., the target-only column and the co-training columns are not clearly separated), making it difficult to verify the claims about baseline versus camPose diversity. Please reformat these tables.
  4. [Section 5.4 and Appendix E] The claim that 'empirically, we found no difference in performance between models trained for 300 epochs and those trained for 600' is presented without supporting data; please provide the comparison or qualify the statement.
  5. [Section 5.4.2] The DROID retrieval thresholds (e.g., 20 cm in X/Y and 10 cm in Z for camera pose, and the 60×60×30 cm cuboid for object spatial) are chosen without sensitivity analysis; since these thresholds define the retrieved subsets, a brief sensitivity check would make the results more robust.
  6. [Abstract] The phrase '1000s of hours' is informal; consider 'thousands of hours.'

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the central claims are empirical and tested on held-out real-world data.

full rationale

The paper's central claims are empirical outcomes measured on held-out data, not derivations from assumptions. The simulation study in Sections 4 and 5 generates datasets with controlled dimensions of variation and measures downstream policy success; conclusions such as 'camera poses and spatial arrangements are crucial' and 'object textures have minimal impact' are contingent results of those experiments, not consequences of the definitions of diversity and alignment in Section 3.2. The real-world retriever experiments in Section 5.4.2 apply retrieval strategies selected from simulation insights to DROID, an external dataset collected by other labs, and compare against full-DROID co-training, with success rates measured across 20 rollouts. No equation in the paper defines a predicted success rate in terms of the outcomes it claims to predict, and no parameter is fitted to the target results. Author-overlapping citations such as MimicGen are used as data-generation tools or prior context, not as a uniqueness theorem or as the sole justification for the empirical ranking. The full-DROID baseline scoring 0% on all tasks is a legitimate experimental-robustness concern that could reflect an optimization artifact, but that is a correctness question rather than a circularity, because the comparison is still an external measurement rather than a construction. Overall, the derivation chain is self-contained against external benchmarks and no circular step can be exhibited.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the listed domain assumptions and hand-chosen experimental constants. No invented physical entities are introduced; the MimicLabs dataset is a new resource but not a postulate.

free parameters (5)
  • co-training mixing weight omega = 0.5
    All co-training experiments use omega=0.5 to balance target and co-training batches (Appendix E.1), chosen by hand; ablations at 0.3 and 0.7 appear only in Fig. 13.
  • DROID camera pose retrieval threshold = 20cm in X/Y, 10cm in Z
    Appendix J: demos are kept when camera extrinsic position is within 20cm of the target camera in X and Y and 10cm in Z. The threshold is set by hand with no sensitivity analysis.
  • DROID object spatial cuboid = 60x60x30cm
    Appendix J: retrieved demos keep object positions in a 60x60x30cm cuboid centered on the testing distribution, chosen to cover the test distribution.
  • Number of co-training demonstrations = 1000 (simulation), 100 (real collector)
    Co-training experiments fix N_C: 1000 demos in simulation (Section 5) and 100 demos in real collector experiments (Appendix I). The choice is not swept.
  • Simulation target demos count = 10
    All simulation retriever experiments use 10 target demonstrations (Section 5.2). This is a design constant.
assumptions (5)
  • domain assumption Simulation insights from MimicGen-generated data transfer to real-world manipulation.
    Section 5.4 validates simulation findings on real tasks, implicitly assuming the procedural data generator (built on MimicGen) preserves the task-relevant training signal.
  • domain assumption The five dimensions of variation are independent and sufficient to characterize dataset composition.
    Section 3.2 defines DVs and treats them as independent; the study does not test interactions beyond the single-DV misalignment experiments.
  • domain assumption Balanced co-training with omega=0.5 is a representative way to combine target and co-training data.
    Appendix E.1 introduces the weighted sampler; the central comparisons all use omega=0.5.
  • domain assumption DROID language instructions and LLaVA-based color detection provide reliable metadata for retrieval.
    Appendix J uses parsed instructions, gripper-close positions, and LLaVA v1.5 7b color answers to label DROID demos; errors in this metadata propagate to the retrieval comparison.
  • domain assumption MimicGen decomposition of human demonstrations into object-centric segments preserves the manipulation strategy needed for the target tasks.
    Section 4 uses MimicGen to scale from about 500 human demos to 1M trajectories; the study assumes the synthesized trajectories are equivalent in learning value to real demonstrations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What Matters in Learning from Large-Scale Datasets for Robot Manipulation." pith.science (2026). https://pith.science/paper/IMJPHJWW

@misc{pith2026250613536,
  author       = {Pith},
  title        = {Pith review of: What Matters in Learning from Large-Scale Datasets for Robot Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IMJPHJWW}},
  note         = {Machine review of arXiv:2506.13536}
}
read the original abstract

Imitation learning from large multi-task demonstration datasets has emerged as a promising path for building generally-capable robots. As a result, 1000s of hours have been spent on building such large-scale datasets around the globe. Despite the continuous growth of such efforts, we still lack a systematic understanding of what data should be collected to improve the utility of a robotics dataset and facilitate downstream policy learning. In this work, we conduct a large-scale dataset composition study to answer this question. We develop a data generation framework to procedurally emulate common sources of diversity in existing datasets (such as sensor placements and object types and arrangements), and use it to generate large-scale robot datasets with controlled compositions, enabling a suite of dataset composition studies that would be prohibitively expensive in the real world. We focus on two practical settings: (1) what types of diversity should be emphasized when future researchers collect large-scale datasets for robotics, and (2) how should current practitioners retrieve relevant demonstrations from existing datasets to maximize downstream policy performance on tasks of interest. Our study yields several critical insights -- for example, we find that camera poses and spatial arrangements are crucial dimensions for both diversity in collection and alignment in retrieval. In real-world robot learning settings, we find that not only do our insights from simulation carry over, but our retrieval strategies on existing datasets such as DROID allow us to consistently outperform existing training strategies by up to 70%. More results at https://robo-mimiclabs.github.io/

Figures

Figures reproduced from arXiv: 2506.13536 by the authors.

Figure 1
Figure 1. MimicLabs overview. Our framework encompasses: (1) A procedural dataset generator that creates diverse datasets with controlled composition. (2) A large-scale dataset composition study to analyze the impact of dataset diversity and alignment inspired by practical settings. (3) Extensive real-world experiments using existing large-scale robot datasets, based on study insights. different dataset compositions intractab… view at source ↗
Figure 2
Figure 2. Characterizing Co-Training Settings: (a) We consider four co-training settings character￾ized by the diversity of their co-training datasets and their alignment with the target dataset. This is illustrated with the object spatial DV of the coffee pod. (b) The collectors aim to select and increase the diversity of selected DV(s) to improve the utility of a dataset. (c) The retrievers aim to extract a subset from an e… view at source ↗
Figure 3
Figure 3. MimicLabs Dataset. We use our data generator to create a large-scale, multi-task dataset to emulate a realistic scenario where multiple labs collaborate to create a dataset with large variations in diverse DVs. We leverage this dataset to conduct our dataset composition study. 4 PROCEDURAL TASK AND DEMONSTRATION GENERATION To understand how different data composition choices influence downstream policy learning, we … view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Real World Tasks. We leverage our insights from the simulation study to conduct experiments with 7 manipulation tasks in the real world. Retrieving skills for co-training, even in the presence of overall heterogeneity, significantly boosts downstream performance. For e…
Figure 5
Figure 5. Figure 5: Retriever results on a real robot. We compare the performance of models trained on retrieved datasets with those co-trained on the entirety of DROID. Target task #demos Target only DROID obj/skill +camPose +objTex +objSpat serve snack 20 5 0 65 70 35 85 bin can 20 60 0…
Figure 6
Figure 6. Figure 6: Illustrating camera distributions in MimicLabs discretized into 5 bins. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Procedural texture generation in MimicLabs. Fractal textures were used in the Collector’s [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: The green and red bounding boxes define the reset distributions for the object and the receptacle in each retriever task. For our real-world experiments, we evaluate a variety of challenging and diverse table-top manip￾ulation tasks that comprise multiple motion primit…
Figure 9
Figure 9. Figure 9: Hardware setup. The red circles show the camera set-up in the real experiment. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: Retrieved cotraining demos for the wipe board task with the camera pose aligned. We can see that the spatial locations and textures of the objects in the demos are not necessarily aligned with our target object. However, the camera pose is the same. Target Retrieved …
Figure 11
Figure 11. Figure 11: Retrieved cotraining demos for the serve snack task with the object spatial location aligned. The camera poses and textures seen in the demos are not the same as our target task, but the location of the object (in front of the robot and level with its base) is similar…
Figure 12
Figure 12. Figure 12: Retrieved cotraining demos for the pour task with the object texture aligned. The camera poses and spatial locations of the target object are not the same, but the color is aligned. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Results showing success rates on two tasks using BC-RNN and Diffusion Policy for [PITH_FULL_IMAGE:figures/full_fig_p029_13.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Static In, Dynamic Out: Counterfactual Action Augmentation for Moving Object Manipulation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    SIDO morphs static demonstrations into counterfactual future-pose samples, training a goal-conditioned policy that, paired with a pose predictor, grasps objects whose motion was unseen during training.

  2. RoboLight: A Dataset with Linearly Composable Illumination for Robotic Manipulation

    cs.RO 2026-03 conditional novelty 6.0 of 10

    A dataset that records identical robot manipulation tasks under 14 controlled lighting conditions and uses HDR linearity to synthesize 196,000 additional lighting-varied episodes.

Reference graph

Works this paper leans on

13 extracted references · 12 canonical work pages · cited by 2 Pith papers

  1. [1]

    not-diverse and misaligned:|S(Z (k) C )|≈|S(Z (k) T )|andS(Z (k) T )∩S(Z (k) C ) =ϕ

    work page

  2. [2]

    diverse and misaligned:|S(Z (k) C )|≫|S(Z (k) T )|andS(Z (k) T )∩S(Z (k) C ) =ϕ

    work page

  3. [3]

    diverse and aligned:|S(Z (k) C )|≫|S(Z (k) T )|andS(Z (k) T )⊂S(Z (k) C )

    work page

  4. [4]

    2, 1-4 starting from top-left going in clockwise order

    not-diverse and aligned (perfect alignment): |S(Z (k) C )| ≈ |S(Z(k) T )| andS(Z (k) T )⊂ S(Z (k) C ) We illustrate these cases comparingDT andDC along a DV in Fig. 2, 1-4 starting from top-left going in clockwise order. As shown in our experiments, cases 1 and 2 (diverse or not, with misalignment) are important cases to study from a collector’s perspecti...

    work page 2025

  5. [7]

    camPose: we select one of the 4 external cameras as the target camera during evaluation and co-train models using various combinations of the cameras

    work page

  6. [8]

    In total, there are∼290 unique task instances in each lab, with skill-level overlap designed to test positive retrieval strategies

    make coffee where X can be replaced by one of 7 distinct objects available in each lab for data collection and policy evaluation, Y can be a drawer or a microwave, distinct instances of which are available in each lab. In total, there are∼290 unique task instances in each lab, with skill-level overlap designed to test positive retrieval strategies. Additi...

    work page 2025

  7. [9]

    Note that we do not ablate on material properties of these objects

    objTex: we pick one color of the target object for evaluation, and co-train models with demos of varying target object textures. Note that we do not ablate on material properties of these objects. We also keep our lighting conditions consistent within each dataset to remove any variations in object visuals due to lighting-material interactions

    work page

  8. [10]

    The different co-training spatial distributions are concentric boxes that increase in size until they cover the target spatial distribution completely

    objSpat: we pick a spatial distribution for the target object during evaluation and co-train models with demos where the target object has varying spatial distributions. The different co-training spatial distributions are concentric boxes that increase in size until they cover the target spatial distribution completely. DV data collection:to build the dat...

    work page

Show all 13 references
  1. [11]

    Since all four external cameras were streaming simultaneously, we could build co-training datasets with combinations of the different cameras through post-processing

    camPose: we collected 100 demos of the task, with a small reset distribution and a single texture. Since all four external cameras were streaming simultaneously, we could build co-training datasets with combinations of the different cameras through post-processing

  2. [12]

    for our baseline dataset

    objTex: we used the data collected in 1. for our baseline dataset. We then collected an additional 50 demos with a second color to build our second co-training dataset (50 from each) and so on with a third color. All demos were collected with a small spatial distribution

  3. [13]

    for our baseline dataset

    objSpat: again, we used the data collected in 1. for our baseline dataset. We then collected enough demos to cover a medium reset distribution of 38x38cm and even more demos to cover a large reset distribution of 50x50cm. Each co-training dataset (small, medium, large) consist...

    2025

  4. [2024]

    USER: What’s the color of the <target object> in the image? Answer in just one adjective word. ASSISTANT:

    to detect the color of the target object from this image. The prompt provided to the model was: "USER: What’s the color of the <target object> in the image? Answer in just one adjective word. ASSISTANT:". The different types of retrieval we performed on DROID are:obj/skill,obj...

    2025

  5. [8856]

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox

    URLhttps://openreview.net/forum?id=jcleXdnRA1. Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation, 2022. URLhttps://arxiv.org/abs/2209.05451. Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martín-Mar...

    2022 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.