Pith. sign in

REVIEW 3 major objections 6 minor 62 references

One demonstrated trajectory, expanded through a 3D generative model, can train a robot policy that succeeds from initial poses far outside the demonstration—including the opposite side of the object.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A robot policy trained on one real demonstration plus AI-generated 3D views succeeds from novel initial poses, including opposite-side starts, across six real manipulation tasks.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A credible demo-augmentation paper that deserves review; the geometric fidelity of its generated mesh is the load-bearing assumption and remains unquantified. the 3 major comments →

arxiv 2509.06191 v1 pith:7T7Y3AUE submitted 2025-09-07 cs.RO cs.CVcs.LG

Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)

Yifei Ren , Edward Johns This is my paper
classification cs.RO cs.CVcs.LG
keywords one-shot imitation learning3D generative modelsnovel view synthesisomnidirectional policydata augmentationdiffusion policyrobot manipulation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

One-shot behavioural cloning traditionally needs many demonstrations because the policy must cover every initial state. This paper claims that a single real-world demonstration, recorded through a wrist-mounted RGB camera, can be amplified by a 3D generative model into a training set large enough to learn an omnidirectional policy—one that succeeds even from poses never observed, including the opposite side of the object. The key step is generating the object's unseen geometry from five demonstrated frames, then rendering thousands of novel-view images labelled with collision-free actions that guide the robot back to the demonstrated skill. Across six everyday tasks, the augmented policy reached 85% success from nearby starts and 73.3% from far starts, against 18.3%/5% for a point-cloud baseline and 28.3%/3.3% for a partial-NeRF baseline. The consequence is that viewpoint robustness, normally bought with hundreds of demonstrations, can be obtained from one demonstration plus generative imagination.

Core claim

The paper's central claim is that the failure mode of one-shot behavioural cloning—out-of-distribution initial states—can be removed by replacing the missing observations rather than by collecting more demonstrations. OP-Gen takes the five posed wrist-camera images captured during the single demonstration, asks a multi-view generative model (EscherNet) to produce 100 consistent novel views, builds a NeRF and a mesh from those views, plans collision-free anchored trajectories from randomly sampled start poses to the demonstrated 'bottleneck' pose, renders synthetic observations along those trajectories, and labels each with the relative end-effector action. Training a diffusion policy on thes

What carries the argument

The load-bearing mechanism is the anchored trajectory generation (ATG) pipeline, which turns imagined views into executable training data. Starting from a random end-effector pose, CuRobo plans a collision-free path to the demonstrated bottleneck pose using a mesh extracted from the generated NeRF; the path is then re-focused so the wrist camera always looks at the object, with small orientation perturbations, and spherical linear interpolation (SLERP) smooths rotations between anchor points. Rendering each corrected pose through the NeRF and assigning the corresponding relative actions creates image-action pairs whose distribution covers viewpoints never seen in the demonstration. EscherNet

Load-bearing premise

The entire pipeline relies on the generative model turning five wrist-camera frames into an accurate picture of the object's unseen half; if the imagined back of the object is hallucinated, the planned trajectories and rendered images can be physically wrong, and the policy fails.

What would settle it

Take a test object, scan it fully, then run OP-Gen from only five wrist-camera frames and measure the error between the generated mesh and the scan on the side opposite the demonstration. If real-world Omni success stays high even when that unseen-side error is large (tens of millimetres or a clearly wrong handle), the claim that accurate generated geometry carries the method would be refuted; conversely, if success tracks that error, the generative model's geometry is the decisive factor.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • One-shot imitation learning can produce policies that generalise across the full viewing sphere around an object, not just near the demonstrated pose.
  • 3D generative augmentation dominates partial point-cloud and partial-NeRF augmentation for unseen viewpoints, because only generative models complete the unseen half of the object.
  • Policy quality tracks cross-view consistency of generated renderings more than raw image fidelity, so advances in multi-view-consistent generation should transfer directly into manipulation performance.
  • The anchored-trajectory design matters: naive motion planning, unfocused cameras, or abrupt rotations all collapse success rates, so augmentation quality is a policy-design problem, not just a rendering problem.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the pipeline is instance-specific and object-level, the main bottleneck is the generative model's ability to hallucinate true geometry; a direct test would measure the distance between the generated mesh and a full scan of the unseen side and correlate it with Omni success.
  • Editorial inference: the same 'imagine the unseen, then plan back to the demonstrated phase' recipe could apply to multi-demonstration or multi-object settings, with generative models used to fill gaps rather than to create the entire training set.
  • Editorial inference: if 3D generation advances to articulated or dynamic scenes, the static-object restriction and the need to keep skill phases unchanged could be relaxed, letting the method cover contact-rich and deformable-object tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes OP-Gen, a one-shot imitation learning method that uses a 3D generative model (EscherNet) to synthesize novel views of an object from five wrist-camera images from a single demonstration, builds a NeRF and a mesh from those synthetic views, and then generates a large dataset of 'imagined' image-action pairs by planning collision-free trajectories from random initial poses to a bottleneck point. A diffusion policy is trained on this augmented dataset and deployed on a Franka Panda with a wrist RGB camera. The authors evaluate on six real-world tasks (grasping drill, mug, plane, coffee pot; trash in bin; opening air fryer drawer), comparing against no augmentation, a partial point-cloud baseline (OP-PCD), a partial-NeRF baseline (SPARTN), and a full-scan NeRF upper bound. They report 85% and 73.3% average success rates for OP-Gen in Narrow and Omni settings, respectively, versus 83.3% for the upper bound in Omni, and they include ablations of the anchored trajectory generation pipeline and an SSIM-based analysis of rendering quality. The paper also states several limitations, including reliance on 3D generation quality and lack of dynamic interaction augmentation.

Significance. If the underlying geometric assumption holds, the paper is a valuable step: it demonstrates a practical way to turn a single demonstration into a large, viewpoint-covering training set for visuomotor policies, using strong real-world comparisons against relevant baselines and a full-scan upper bound. The inclusion of six real tasks, 20 rollouts per task, a component ablation, and a clearly stated limitations section are strengths. The main risk is that the method's load-bearing assumption—that EscherNet's generated geometry is accurate enough for collision-free planning and actionable renderings—is not directly validated. The paper's real-world results are encouraging, but the lack of statistical inference and the indirect appearance-only validation of geometry currently leave the central claim only partially supported.

major comments (3)
  1. [Sec. III-C, III-D and Fig. 5] The claim that OP-Gen's imagined trajectories are executable depends on the mesh extracted from the EscherNet-to-Instant-NGP pipeline being geometrically accurate and correctly scaled. The only quantitative validation is SSIM, an appearance metric that cannot detect hallucinated backside geometry, wrong scale, or collision-invalid surfaces. The paper itself attributes failures to 'subtle misalignments between the generated 3D model and the physical object' (Sec. IV-B) but never measures this misalignment. Please add a quantitative geometry check (e.g., chamfer distance / F-score between the generated mesh and the full-scan mesh, depth error on held-out views, or physical collision-checking of planned trajectories) and relate it to per-task success rates. Without this, the 73.3% Omni result is not attributable to the generated geometry rather than to robustness on the sampled initial pose
  2. [Table I, Sec. IV-B] Each condition is based on 10 rollouts per task. Differences such as Drill-Narrow (OP-Gen 90% vs UB 90%) or Mug-Omni (90% vs 100%) are within binomial sampling error, so claims of 'significantly outperforming' and 'approaching the upper bound' are not statistically supported as stated. Please report raw per-rollout initial poses, add confidence intervals or exact binomial tests (per task or pooled with task as a blocking factor), and specify exactly how the Narrow/Omni initial poses were sampled so the comparison is reproducible and not sensitive to a particular draw.
  3. [Sec. IV-A vs Sec. III-D] The simulation success criterion is a to-go distance to the bottleneck of less than 1 cm, which is the same threshold used to label the gripper-close action during augmentation. The simulation metric is therefore measuring agreement with the training label, not task success, and it is computed in the same NeRF used to generate the augmented data. The real-world results are the appropriate external test; please present simulation as a diagnostic and clarify this circularity, especially since the Linear ablation in Table II shows simulation can report high success (95%) when real-world success is much lower (43.33%).
minor comments (6)
  1. [Table I] The 'Total' column appears to be the average of the Narrow and Omni condition averages, not a total success count. Rename it to 'Overall Avg' and clarify what 'Avg Time(s)' includes for each method (e.g., No Aug is just the 10 s demonstration; OP-Gen includes generation time).
  2. [Sec. III-D] The statement 'Since fewer than 1% of augmented poses are near the bottleneck, we oversample poses within 1 cm of it' should specify the oversampling mechanism (duplication, resampling weight, or explicit generation) and report the resulting class balance.
  3. [Sec. III-E] Please clarify how the action horizon of 4 is executed at test time: is the full chunk predicted and executed open-loop, or is a receding-horizon scheme used (predict 4, execute 1, re-predict)? This affects the interpretation of the real-world results.
  4. [Sec. IV-C and Fig. 5] The thresholds 'SSIM above 0.95 is necessary for small, textureless objects' and 'SSIM above 0.8 suffices for larger objects' are drawn from a small number of tasks and two method families. Soften these claims or add per-task data points in Fig. 5 so the reader can assess the strength of the correlation.
  5. [Fig. 4 and Fig. 5] Add explicit legends or marker conventions for each method in Fig. 5, and state in the caption whether the SSIM values are averaged over all tested viewpoints or only over the Omni directions. This would clarify the observed overlap between SPARTN/OP-PCD and OP-Gen.
  6. [Sec. III-C] Specify how the 5 input images are selected from the demonstration (uniformly sampled? from which phase?) and whether the 100 queried views are purely synthetic or include any of the real demo images. This affects reproducibility and the interpretation of the NeRF construction.

Circularity Check

1 steps flagged

Central real-world claim is externally benchmarked, but the simulation success metric is defined by the same bottleneck proximity used to label actions.

specific steps
  1. self definitional [Section III-D (Action Assignment) and Section IV-A (Simulation)]
    "The gripper state is assigned a value of 1 (close) only when the pose is within 1 cm of the bottleneck. ... The task is considered successful if this to-go distance is less than 1 cm."

    The augmented action labels are generated by planning collision-free trajectories to the bottleneck, and gripper close is triggered only within 1 cm of the bottleneck. The simulation success criterion is the same quantity: after the policy commands the skill phase, success is declared if the remaining distance to the bottleneck is less than 1 cm. Thus the simulation metric measures how well the policy reproduces the training objective (reaching the bottleneck), not an independent task outcome. This is a self-referential evaluation, but it does not determine the paper's central real-world claim, which uses external success criteria such as grasping and lifting, placing trash, or opening the drawer.

full rationale

The paper's main claim—73.3% Omni success from a single demonstration—is supported by real-world rollouts with externally defined success criteria (grasping and lifting, placing trash, opening a drawer). These are not fitted from the augmented dataset and therefore break any circularity. The pipeline uses external, independently published components (EscherNet, Instant-NGP, CuRobo, diffusion policy) without relying on self-citations as load-bearing evidence. The only circular element is the simulation evaluation, where success is defined as reaching the same bottleneck point used in generating the action labels. This affects the internal ablations (Table II) but not the headline real-world comparison. Hence the overall circularity is minor, consistent with a score of 2.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

No free parameters are fitted to the reported success rates; the method's central claim is empirical. The listed parameters are engineering choices (thresholds, dataset size) that affect performance but were not tuned per task, to the authors' account. The heavy reliance is on external tools: EscherNet's scale-aware novel view synthesis, Instant-NGP's reconstruction/mesh, CuRobo's planning, XMem's segmentation, and the diffusion-policy backbone.

free parameters (5)
  • Bottleneck detection distance threshold
    Used in Sec. III-B to split the demonstration into motion and skill phases; exact value not given, chosen by hand and indirectly affects all augmented data labels.
  • Gripper close threshold = 1 cm
    In Sec. III-D, the gripper state is set to close only when the pose is within 1 cm of the bottleneck, and oversampling of that region is used to balance data. Same threshold later defines simulation success.
  • Augmented dataset size K = 20000
    Total number of generated data points (Sec. III-D); chosen practical size, not optimized.
  • Number of input images to EscherNet = 5
    Sec. III-C: 5 images sampled from the demonstration as generator input; design choice.
  • Evaluation pose categories = +/-45 deg fan
    Sec. IV-B: initial poses categorized as Narrow or Omni based on a 45-degree fan around the first demo camera view; an evaluation choice, not part of the method.
axioms (6)
  • domain assumption EscherNet produces up-to-scale, 3D-consistent novel views from 5 images of an object.
    Invoked in Sec. III-C; if the generated views are not consistent or not to scale, the downstream NeRF and mesh are wrong. Relies on the cited paper, not revalidated here.
  • domain assumption The Instant-NGP NeRF built from EscherNet renderings is a faithful enough world model for rendering and for extracting a usable mesh.
    Sec. III-C: NeRF is used for rendering and mesh extraction. Errors from the generative model propagate into the NeRF and into the collision-checked trajectories.
  • domain assumption CuRobo's motion planner returns collision-free trajectories with respect to the extracted mesh.
    Sec. III-D: planned trajectories are assumed collision-free and executable. The ablation 'Linear' shows collisions cause failures when planning is ablated.
  • domain assumption XMem segmentation is accurate enough at train and test time.
    Sec. III-B: foreground object segmentation is used during training and testing; the paper lists segmentation accuracy as a limitation.
  • domain assumption The demonstration can be split into motion and skill phases with a single bottleneck point.
    Sec. III-B: this decomposition is inherited from prior work [4][9][11][12][35] and is assumed valid for all six tasks.
  • domain assumption Diffusion policy can learn a successful policy from the augmented dataset.
    Sec. III-E: the chosen backbone is assumed adequate; no comparison to other policy architectures is made.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)." pith.science (2026). https://pith.science/paper/7T7Y3AUE

@misc{pith2026250906191,
  author       = {Pith},
  title        = {Pith review of: Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7T7Y3AUE}},
  note         = {Machine review of arXiv:2509.06191}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent 3D generative models, which are capable of generating full object shapes from just a few images, now open up new opportunities in robotics. In this work, we show that 3D generative models can be used to augment a dataset from a single real-world demonstration, after which an omnidirectional policy can be learned within this imagined dataset. We found that this enables a robot to perform a task when initialised from states very far from those observed during the demonstration, including starting from the opposite side of the object relative to the real-world demonstration, significantly reducing the number of demonstrations required for policy learning. Through several real-world experiments across tasks such as grasping objects, opening a drawer, and placing trash into a bin, we study these omnidirectional policies by investigating the effect of various design choices on policy behaviour, and we show superior performance to recent baselines which use alternative methods for data augmentation.

Figures

Figures reproduced from arXiv: 2509.06191 by Edward Johns, Yifei Ren.

Figure 1
Figure 1. Figure 1: We utilise 3D generative models for data augmentation to enable one-shot imitation learning from a single [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The OP-Gen pipeline begins with a single demonstration [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: An illustration of how anchored trajectory generation [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: As shown, the UB generates renderings that are [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results showing renders from different [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 16 canonical work pages · 1 internal anchor

  1. [1]

    π 0: A vision- language-action flow model for general robot control,

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichteret al., “π 0: A vision- language-action flow model for general robot control,”arXiv preprint arXiv:2410.24164, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Aloha unleashed: A simple recipe for robot dexterity,

    T. Z. Zhao, J. Tompson, D. Driess, P. Florence, K. Ghasemipour, C. Finn, and A. Wahid, “Aloha unleashed: A simple recipe for robot dexterity,”arXiv preprint arXiv:2410.13126, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Eschernet: A generative model for scalable view synthesis,

    X. Kong, S. Liu, X. Lyu, M. Taher, X. Qi, and A. J. Davison, “Eschernet: A generative model for scalable view synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 9503–9513

    work page 2024

  4. [4]

    Demogen: Synthetic demonstration generation for data-efficient visuomotor pol- icy learning,

    Z. Xue, S. Deng, Z. Chen, Y . Wang, Z. Yuan, and H. Xu, “Demogen: Synthetic demonstration generation for data-efficient visuomotor pol- icy learning,”arXiv preprint arXiv:2502.16932, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Diffusion policy: Visuomotor policy learning via ac- tion diffusion,

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song, “Diffusion policy: Visuomotor policy learning via ac- tion diffusion,”The International Journal of Robotics Research, p. 02783649241273668, 2023

    2023

  6. [6]

    Roboagent: Generalization and efficiency in robot ma- nipulation via semantic augmentations and action chunking,

    H. Bharadhwaj, J. Vakil, M. Sharma, A. Gupta, S. Tulsiani, and V . Kumar, “Roboagent: Generalization and efficiency in robot ma- nipulation via semantic augmentations and action chunking,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 4788–4795

    2024

  7. [7]

    Bc-z: Zero-shot task generalization with robotic imitation learning,

    E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, “Bc-z: Zero-shot task generalization with robotic imitation learning,” inConference on Robot Learning. PMLR, 2022, pp. 991–1002

    2022

  8. [8]

    Learning fine-grained bimanual manipulation with low-cost hardware,

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,”arXiv preprint arXiv:2304.13705, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Coarse-to-fine imitation learning: Robot manipulation from a single demonstration,

    E. Johns, “Coarse-to-fine imitation learning: Robot manipulation from a single demonstration,” inIEEE International Conference on Robotics and Automation (ICRA), 2021

    work page 2021

  10. [10]

    Demonstrate once, imitate immediately (dome): Learning visual servoing for one- shot imitation learning,

    E. Valassakis, G. Papagiannis, N. Di Palo, and E. Johns, “Demonstrate once, imitate immediately (dome): Learning visual servoing for one- shot imitation learning,” in2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 8614– 8621

    2022

  11. [11]

    One-Shot Imitation Learning: A Pose Estimation Perspective

    P. Vitiello, K. Dreczkowski, and E. Johns, “One-shot imitation learn- ing: A pose estimation perspective,”arXiv preprint arXiv:2310.12077, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  12. [12]

    You only demonstrate once: Category-level manipulation from single visual demonstration,

    B. Wen, W. Lian, K. Bekris, and S. Schaal, “You only demonstrate once: Category-level manipulation from single visual demonstration,” arXiv preprint arXiv:2201.12716, 2022

    Pith/arXiv arXiv 2022

  13. [13]

    On the effectiveness of retrieval, alignment, and replay in manipulation,

    N. Di Palo and E. Johns, “On the effectiveness of retrieval, alignment, and replay in manipulation,”IEEE Robotics and Automation Letters, vol. 9, no. 3, pp. 2032–2039, 2024

    work page 2032

  14. [14]

    Miles: Making imitation learning easy with self-supervision,

    G. Papagiannis and E. Johns, “Miles: Making imitation learning easy with self-supervision,” inProceedings of the Conference on Robot Learning (CoRL), 2024

    work page 2024

  15. [15]

    Keypoint action tokens enable in-context imitation learning in robotics,

    N. Di Palo and E. Johns, “Keypoint action tokens enable in-context imitation learning in robotics,” inProceedings of Robotics: Science and Systems (RSS), 2024

    work page 2024

  16. [16]

    Instant policy: In-context imitation learning via graph diffusion,

    V . V osylius and E. Johns, “Instant policy: In-context imitation learning via graph diffusion,”arXiv preprint arXiv:2411.12633, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Dall-e-bot: Introducing web- scale diffusion models to robotics,

    I. Kapelyukh, V . V osylius, and E. Johns, “Dall-e-bot: Introducing web- scale diffusion models to robotics,”IEEE Robotics and Automation Letters, vol. 8, no. 7, pp. 3956–3963, 2023

    2023

  18. [18]

    Dream2Real: Zero- shot 3D object rearrangement with vision-language models,

    I. Kapelyukh, Y . Ren, I. Alzugaray, and E. Johns, “Dream2Real: Zero- shot 3D object rearrangement with vision-language models,” inIEEE International Conference on Robotics and Automation (ICRA), 2024

    work page 2024

  19. [19]

    Language models as zero-shot trajectory generators,

    T. Kwon, N. Di Palo, and E. Johns, “Language models as zero-shot trajectory generators,”IEEE Robotics and Automation Letters, 2024

    work page 2024

  20. [20]

    Reinforcement learning with augmented data,

    M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas, “Reinforcement learning with augmented data,”Advances in neural information processing systems, vol. 33, pp. 19 884–19 895, 2020

    2020

  21. [21]

    Domain adversarial reinforcement learning,

    B. Li, V . Franc ¸ois-Lavet, T. Doan, and J. Pineau, “Domain adversarial reinforcement learning,”arXiv preprint arXiv:2102.07097, 2021

    Pith/arXiv arXiv 2021

  22. [22]

    Image augmentation is all you need: Regularizing deep reinforcement learning from pixels,

    D. Yarats, I. Kostrikov, and R. Fergus, “Image augmentation is all you need: Regularizing deep reinforcement learning from pixels,” in International conference on learning representations, 2021

    2021

  23. [23]

    Retinagan: An object-aware approach to sim-to-real transfer,

    D. Ho, K. Rao, Z. Xu, E. Jang, M. Khansari, and Y . Bai, “Retinagan: An object-aware approach to sim-to-real transfer,” in2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021, pp. 10 920–10 926

    2021

  24. [24]

    Rl-cyclegan: Reinforcement learning aware simulation-to-real,

    K. Rao, C. Harris, A. Irpan, S. Levine, J. Ibarz, and M. Khansari, “Rl-cyclegan: Reinforcement learning aware simulation-to-real,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11 157–11 166

    work page 2020

  25. [25]

    Sim2real view invariant visual servoing by recurrent control,

    F. Sadeghi, A. Toshev, E. Jang, and S. Levine, “Sim2real view invariant visual servoing by recurrent control,”arXiv preprint arXiv:1712.07642, 2017

    Pith/arXiv arXiv 2017

  26. [26]

    Scaling robot learning with semantically imagined experience,

    T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, J. Peralta, B. Ichteret al., “Scaling robot learning with semantically imagined experience,”arXiv preprint arXiv:2302.11550, 2023

    Pith/arXiv arXiv 2023

  27. [27]

    Zero-shot text-to-image generation,

    A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. V oss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in International conference on machine learning. Pmlr, 2021, pp. 8821– 8831

    work page 2021

  28. [28]

    Hierarchi- cal text-conditional image generation with clip latents,

    A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchi- cal text-conditional image generation with clip latents,”arXiv preprint arXiv:2204.06125, vol. 1, no. 2, p. 3, 2022

    Pith/arXiv arXiv 2022

  29. [29]

    High-resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 684–10 695

    2022

  30. [30]

    Photorealistic text-to-image diffusion models with deep lan- guage understanding,

    C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans et al., “Photorealistic text-to-image diffusion models with deep lan- guage understanding,”Advances in neural information processing systems, vol. 35, pp. 36 479–36 494, 2022

    2022

  31. [31]

    Nerf in the palm of your hand: Corrective augmentation for robotics via novel-view synthesis,

    A. Zhou, M. J. Kim, L. Wang, P. Florence, and C. Finn, “Nerf in the palm of your hand: Corrective augmentation for robotics via novel-view synthesis,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 17 907–17 917

    work page 2023

  32. [32]

    Nerf: Representing scenes as neural radiance fields for view synthesis,

    B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoor- thi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,”Communications of the ACM, vol. 65, no. 1, pp. 99–106, 2021

    2021

  33. [33]

    Nerf-aug: Data augmentation for robotics with neural radiance fields,

    E. Zhu, M. Levy, M. Gwilliam, and A. Shrivastava, “Nerf-aug: Data augmentation for robotics with neural radiance fields,”arXiv preprint arXiv:2411.02482, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    Diffusion meets dagger: Supercharging eye-in-hand imitation learning,

    X. Zhang, M. Chang, P. Kumar, and S. Gupta, “Diffusion meets dagger: Supercharging eye-in-hand imitation learning,”arXiv preprint arXiv:2402.17768, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Mimicgen: A data generation system for scalable robot learning using human demonstrations,

    A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y . Narang, L. Fan, Y . Zhu, and D. Fox, “Mimicgen: A data generation system for scalable robot learning using human demonstrations,”arXiv preprint arXiv:2310.17596, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    Skillmimicgen: Automated demonstration generation for efficient skill learning and deployment,

    C. Garrett, A. Mandlekar, B. Wen, and D. Fox, “Skillmimicgen: Automated demonstration generation for efficient skill learning and deployment,”arXiv preprint arXiv:2410.18907, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Dexmimicgen: Automated data generation for biman- ual dexterous manipulation via imitation learning,

    Z. Jiang, Y . Xie, K. Lin, Z. Xu, W. Wan, A. Mandlekar, L. Fan, and Y . Zhu, “Dexmimicgen: Automated data generation for biman- ual dexterous manipulation via imitation learning,”arXiv preprint arXiv:2410.24185, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Constraint-preserving data generation for visuomotor policy learning,

    K. Lin, V . Ragunath, A. McAlinden, A. Prasad, J. Wu, Y . Zhu, and J. Bohg, “Constraint-preserving data generation for visuomotor policy learning,”arXiv preprint arXiv:2508.03944, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    Reconciling reality through simulation: A real- to-sim-to-real approach for robust manipulation,

    M. Torne, A. Simeonov, Z. Li, A. Chan, T. Chen, A. Gupta, and P. Agrawal, “Reconciling reality through simulation: A real- to-sim-to-real approach for robust manipulation,”arXiv preprint arXiv:2403.03949, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Robot see robot do: Imitating articulated ob- ject manipulation with monocular 4d reconstruction,

    J. Kerr, C. M. Kim, M. Wu, B. Yi, Q. Wang, K. Goldberg, and A. Kanazawa, “Robot see robot do: Imitating articulated ob- ject manipulation with monocular 4d reconstruction,”arXiv preprint arXiv:2409.18121, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Re 3sim: Generating high-fidelity simulation data via 3d-photorealistic real-to-sim for robotic manipulation,

    X. Han, M. Liu, Y . Chen, J. Yu, X. Lyu, Y . Tian, B. Wang, W. Zhang, and J. Pang, “Re 3sim: Generating high-fidelity simulation data via 3d-photorealistic real-to-sim for robotic manipulation,”arXiv preprint arXiv:2502.08645, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Novel demonstration generation with gaussian splatting enables ro- bust one-shot manipulation,

    S. Yang, W. Yu, J. Zeng, J. Lv, K. Ren, C. Lu, D. Lin, and J. Pang, “Novel demonstration generation with gaussian splatting enables ro- bust one-shot manipulation,”arXiv preprint arXiv:2504.13175, 2025

    Pith/arXiv arXiv 2025

  43. [43]

    Dream to manipulate: Compositional world models empowering robot imitation learning with imagination,

    L. Barcellona, A. Zadaianchuk, D. Allegro, S. Papa, S. Ghidoni, and E. Gavves, “Dream to manipulate: Compositional world models empowering robot imitation learning with imagination,”arXiv preprint arXiv:2412.14957, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Is single-view mesh recon- struction ready for robotics?

    F. Nolte, B. Sch ¨olkopf, and I. Posner, “Is single-view mesh recon- struction ready for robotics?”arXiv preprint arXiv:2505.17966, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    View-invariant policy learning via zero-shot novel view synthesis,

    S. Tian, B. Wulfe, K. Sargent, K. Liu, S. Zakharov, V . Guizilini, and J. Wu, “View-invariant policy learning via zero-shot novel view synthesis,”arXiv preprint arXiv:2409.03685, 2024

    Pith/arXiv arXiv 2024

  46. [46]

    Robotwin: Dual-arm robot benchmark with generative digital twins (early version),

    Y . Mu, T. Chen, S. Peng, Z. Chen, Z. Gao, Y . Zou, L. Lin, Z. Xie, and P. Luo, “Robotwin: Dual-arm robot benchmark with generative digital twins (early version),” inEuropean Conference on Computer Vision. Springer, 2024, pp. 264–273

    work page 2024

  47. [47]

    Gpt-4v(ision) system card,

    “Gpt-4v(ision) system card,” 2023

    work page 2023

  48. [48]

    Instant neural graphics primitives with a multiresolution hash encoding,

    T. M ¨uller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,”ACM transactions on graphics (TOG), vol. 41, no. 4, pp. 1–15, 2022

    2022

  49. [49]

    3d gaussian splatting for real-time radiance field rendering

    B. Kerbl, G. Kopanas, T. Leimk ¨uhler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering.”ACM Trans. Graph., vol. 42, no. 4, pp. 139–1, 2023

    2023

  50. [50]

    Physically embodied gaussian splatting: A realtime correctable world model for robotics,

    J. Abou-Chakra, K. Rana, F. Dayoub, and N. S ¨underhauf, “Physically embodied gaussian splatting: A realtime correctable world model for robotics,”arXiv preprint arXiv:2406.10788, 2024

    Pith/arXiv arXiv 2024

  51. [51]

    Graspsplats: Efficient ma- nipulation with 3d feature splatting,

    M. Ji, R.-Z. Qiu, X. Zou, and X. Wang, “Graspsplats: Efficient ma- nipulation with 3d feature splatting,”arXiv preprint arXiv:2409.02084, 2024

    Pith/arXiv arXiv 2024

  52. [52]

    Zero-1-to-3: Zero-shot one image to 3d object,

    R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. V ondrick, “Zero-1-to-3: Zero-shot one image to 3d object,” in Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 9298–9309

    2023

  53. [53]

    Syncdreamer: Generating multiview-consistent images from a single- view image,

    Y . Liu, C. Lin, Z. Zeng, X. Long, L. Liu, T. Komura, and W. Wang, “Syncdreamer: Generating multiview-consistent images from a single- view image,”arXiv preprint arXiv:2309.03453, 2023

    Pith/arXiv arXiv 2023

  54. [54]

    Wonder3d: Single image to 3d using cross-domain diffusion,

    X. Long, Y .-C. Guo, C. Lin, Y . Liu, Z. Dou, L. Liu, Y . Ma, S.-H. Zhang, M. Habermann, C. Theobaltet al., “Wonder3d: Single image to 3d using cross-domain diffusion,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 9970–9980

    2024

  55. [55]

    Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion,

    V . V oleti, C.-H. Yao, M. Boss, A. Letts, D. Pankratz, D. Tochilkin, C. Laforte, R. Rombach, and V . Jampani, “Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion,” inEuropean Conference on Computer Vision. Springer, 2025, pp. 439–457

    work page 2025

  56. [56]

    Realfusion: 360deg reconstruction of any object from a single image,

    L. Melas-Kyriazi, I. Laina, C. Rupprecht, and A. Vedaldi, “Realfusion: 360deg reconstruction of any object from a single image,” inPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 8446–8455

    work page 2023

  57. [57]

    Zero123++: a single image to consistent multi-view diffusion base model,

    R. Shi, H. Chen, Z. Zhang, M. Liu, C. Xu, X. Wei, L. Chen, C. Zeng, and H. Su, “Zero123++: a single image to consistent multi-view diffusion base model,”arXiv preprint arXiv:2310.15110, 2023

    Pith/arXiv arXiv 2023

  58. [58]

    Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model,

    H. K. Cheng and A. G. Schwing, “Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 640–658

    work page 2022

  59. [59]

    Curobo: Parallelized collision-free minimum-jerk robot motion gen- eration,

    B. Sundaralingam, S. K. S. Hari, A. Fishman, C. Garrett, K. Van Wyk, V . Blukis, A. Millane, H. Oleynikova, A. Handa, F. Ramoset al., “Curobo: Parallelized collision-free minimum-jerk robot motion gen- eration,”arXiv preprint arXiv:2310.17274, 2023

    Pith/arXiv arXiv 2023

  60. [60]

    Denoising diffusion implicit models,

    J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,”arXiv preprint arXiv:2010.02502, 2020

    Pith/arXiv arXiv 2010

  61. [61]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  62. [62]

    Image quality assessment: from error visibility to structural similarity,

    Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,”IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004

    2004

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.