arxiv: 2505.12705 · v2 · submitted 2025-05-19 · 💻 cs.RO · cs.AI· cs.LG

Recognition: no theorem link

DreamGen: Unlocking Generalization in Robot Learning through Video World Models

Joel Jang , Seonghyeon Ye , Zongyu Lin , Jiannan Xiang , Johan Bjorck , Yu Fang , Fengyuan Hu , Spencer Huang

show 20 more authors

Kaushil Kundalia Yen-Chen Lin Loic Magne Ajay Mandlekar Avnish Narayan You Liang Tan Guanzhi Wang Jing Wang Qi Wang Yinzhen Xu Xiaohui Zeng Kaiyuan Zheng Ruijie Zheng Ming-Yu Liu Luke Zettlemoyer Dieter Fox Jan Kautz Scott Reed Yuke Zhu Linxi Fan

Authors on Pith no claims yet

Pith reviewed 2026-05-15 23:47 UTC · model grok-4.3

classification 💻 cs.RO cs.AIcs.LG

keywords robot policy learningvideo world modelssynthetic trajectoriesbehavior generalizationenvironment generalizationpseudo-action recoveryinverse dynamics modelhumanoid robot

0 comments

The pith

A simple pipeline adapts video world models to generate synthetic robot trajectories that let humanoid policies generalize to 22 new behaviors and unseen environments from data of a single task.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DreamGen is a four-stage process that takes image-to-video generative models, adapts them to a specific robot body, and produces photorealistic videos of both familiar and novel tasks in varied settings. From those videos the method extracts pseudo-action sequences either through a latent action model or an inverse-dynamics model, then trains policies on the resulting neural trajectories. The central demonstration is that this synthetic data alone suffices for a humanoid robot to acquire and execute 22 new skills in both seen and unseen rooms when the only real teleoperation data supplied is one pick-and-place demonstration collected in one environment. The authors also release DreamGen Bench, a video-generation evaluation suite whose scores track downstream policy success, providing an early indicator of whether the generated data will be useful.

Core claim

DreamGen shows that state-of-the-art image-to-video models, once fine-tuned on a target robot embodiment, can synthesize embodiment-consistent videos of new behaviors in diverse environments; recovering pseudo-actions from those videos with either a latent action model or an inverse-dynamics model then yields control policies that transfer directly to the physical robot and generalize across both behaviors and scenes, all while requiring real teleoperation data from only a single pick-and-place task performed in a single environment.

What carries the argument

Adapted image-to-video generative models that produce photorealistic, embodiment-consistent synthetic videos, from which pseudo-action sequences are recovered by a latent action model or inverse-dynamics model.

If this is right

A humanoid robot can execute 22 new behaviors in both familiar and novel environments after training on synthetic data derived from only one real pick-and-place demonstration.
Video-generation quality measured on DreamGen Bench correlates strongly with downstream policy success rates.
Robot learning can be scaled by generating diverse neural trajectories instead of collecting additional manual teleoperation data.
The same pipeline applies to both behavior generalization and environment generalization without separate data collection for each.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If video world models continue to improve in temporal consistency and physics, the amount of real robot data needed for broad generalization could drop further.
The approach opens a route to using large-scale video generation as a cheap source of environment variation that is otherwise expensive to capture in the real world.
Benchmarking video models directly on embodiment fidelity rather than only visual quality may become a useful intermediate evaluation for robotics.
The method could be extended to generate data for multi-step planning or long-horizon tasks once the underlying video models handle longer sequences reliably.

Load-bearing premise

The synthetic videos must be realistic and consistent with the robot's physical embodiment so that policies trained on the recovered pseudo-actions transfer to the real robot without a large domain gap.

What would settle it

Policies trained exclusively on DreamGen-generated data achieve near-zero success rates on the 22 held-out behaviors when deployed on the physical humanoid in either seen or unseen environments.

read the original abstract

We introduce DreamGen, a simple yet highly effective 4-stage pipeline for training robot policies that generalize across behaviors and environments through neural trajectories - synthetic robot data generated from video world models. DreamGen leverages state-of-the-art image-to-video generative models, adapting them to the target robot embodiment to produce photorealistic synthetic videos of familiar or novel tasks in diverse environments. Since these models generate only videos, we recover pseudo-action sequences using either a latent action model or an inverse-dynamics model (IDM). Despite its simplicity, DreamGen unlocks strong behavior and environment generalization: a humanoid robot can perform 22 new behaviors in both seen and unseen environments, while requiring teleoperation data from only a single pick-and-place task in one environment. To evaluate the pipeline systematically, we introduce DreamGen Bench, a video generation benchmark that shows a strong correlation between benchmark performance and downstream policy success. Our work establishes a promising new axis for scaling robot learning well beyond manual data collection. Code available at https://github.com/NVIDIA/GR00T-Dreams.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

DreamGen shows a humanoid learning 22 new behaviors from one real demo via adapted video models and pseudo-action recovery, but the transfer reliability of those recovered actions is the part that needs more checks.

read the letter

The core claim is that adapting an image-to-video model to a specific robot embodiment, generating synthetic videos of new tasks, recovering actions with a latent model or IDM, and training on the resulting trajectories produces policies that generalize across behaviors and environments. In the reported case this means 22 new behaviors on a humanoid after seeing only one pick-and-place teleop session in a single setting. The four-stage pipeline is the main new piece: embodiment-specific adaptation of the video generator plus a practical way to turn the videos back into usable action sequences. DreamGen Bench is also a useful addition because it gives a video-level score that correlates with later policy success, which could let people screen ideas before hardware time is spent. The approach is simple enough that groups already working with generative video models could try it without major new infrastructure. The main soft spot is the action-recovery step for out-of-distribution behaviors. The abstract does not report direct error measurements on how well the latent action model or IDM matches real actions when the generated videos contain novel motions, nor does it show ablations that isolate the effect of the adaptation stage versus the base video model. If the synthetic videos still contain kinematic artifacts on unseen tasks, the recovered trajectories could train policies that look good in simulation but degrade on the physical robot. The paper also gives headline numbers without the usual supporting details on baselines, trial counts, or statistical tests, so the strength of the generalization result is hard to judge from the summary alone. This is aimed at researchers scaling robot learning with generative models rather than at people focused on new control algorithms. A reader who already follows video world models or imitation learning would find the pipeline and benchmark worth examining. It deserves a serious referee because the idea is concrete, the benchmark is reproducible, and the central claim can be tested directly even if the current evidence is still at the headline stage.

Referee Report

3 major / 2 minor

Summary. The manuscript introduces DreamGen, a four-stage pipeline that adapts state-of-the-art image-to-video generative models to a target robot embodiment, synthesizes photorealistic videos of familiar or novel tasks in diverse environments, and recovers pseudo-action sequences via a latent action model or inverse-dynamics model (IDM) to train policies. The central empirical claim is that this approach enables a humanoid robot to perform 22 new behaviors in both seen and unseen environments while using teleoperation data from only a single pick-and-place task in one environment. The paper also introduces DreamGen Bench, a video-generation benchmark reported to correlate with downstream policy success, and positions the method as a scalable alternative to extensive manual data collection.

Significance. If the transfer results hold under rigorous validation, DreamGen would represent a meaningful advance in scaling robot learning by leveraging generative video models to augment limited real-world data, potentially reducing reliance on teleoperation. The introduction of a benchmark with claimed predictive correlation to policy performance offers a practical evaluation axis for future work. The pipeline's simplicity and the ambitious generalization claims (behavioral and environmental) are notable strengths, though they rest on the untested assumption that synthetic videos remain embodiment-consistent for out-of-distribution behaviors.

major comments (3)

[§5] §5 (Experiments and Results): The headline claim that the humanoid performs 22 new behaviors in seen and unseen environments is presented without reported details on evaluation protocols, number of trials per behavior, success criteria, variance across runs, or comparison to baselines trained only on real data. These omissions make the generalization result difficult to assess and constitute a load-bearing gap for the central claim.
[§4] §4 (Pseudo-action Recovery): The method relies on recovering pseudo-actions from adapted video-model outputs for novel behaviors, yet no direct quantitative metrics (e.g., action-recovery error, kinematic consistency checks, or measured sim-to-real transfer gap) are provided for the 22 out-of-distribution tasks. This leaves the weakest link in the pipeline unexamined.
[DreamGen Bench] DreamGen Bench section: The benchmark is asserted to show strong correlation with policy success, but the manuscript lacks the specific correlation coefficient, construction details, held-out tasks, or ablation showing that benchmark scores predict real-robot transfer for novel behaviors rather than just in-distribution cases.

minor comments (2)

[Abstract / §2] The abstract and introduction use the term 'neural trajectories' without an explicit definition; clarify its relation to the generated videos and pseudo-actions in §2 or §3.
[Figures in §5] Figure captions and axis labels in the experimental results should explicitly state the number of seeds or runs underlying each bar or curve to improve interpretability.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive and detailed review. We address each major comment point by point below. Where the manuscript was missing necessary details, we have revised it accordingly to improve clarity and rigor.

read point-by-point responses

Referee: [§5] §5 (Experiments and Results): The headline claim that the humanoid performs 22 new behaviors in seen and unseen environments is presented without reported details on evaluation protocols, number of trials per behavior, success criteria, variance across runs, or comparison to baselines trained only on real data. These omissions make the generalization result difficult to assess and constitute a load-bearing gap for the central claim.

Authors: We agree that the original presentation of results in §5 lacked sufficient protocol details. In the revised manuscript we have expanded this section to specify: 10 independent trials per behavior per environment, success criteria (task completion within 30 seconds without drops or collisions), reporting of mean success rates with standard deviations across three random seeds, and direct comparisons against a baseline policy trained only on the real single-task teleoperation data. These additions make the generalization claims fully evaluable. revision: yes
Referee: [§4] §4 (Pseudo-action Recovery): The method relies on recovering pseudo-actions from adapted video-model outputs for novel behaviors, yet no direct quantitative metrics (e.g., action-recovery error, kinematic consistency checks, or measured sim-to-real transfer gap) are provided for the 22 out-of-distribution tasks. This leaves the weakest link in the pipeline unexamined.

Authors: We acknowledge the value of quantitative checks on pseudo-action recovery. Because ground-truth actions do not exist for the 22 novel behaviors, direct recovery error cannot be computed. In revision we added kinematic consistency metrics (average joint-angle deviation between recovered actions and video trajectories via forward kinematics) and a measured sim-to-real gap obtained by executing recovered actions in simulation versus real-robot rollouts on overlapping tasks. We also report IDM action-prediction error on held-out real data. These indirect validations address the concern while respecting the fundamental data limitation. revision: partial
Referee: [DreamGen Bench] DreamGen Bench section: The benchmark is asserted to show strong correlation with policy success, but the manuscript lacks the specific correlation coefficient, construction details, held-out tasks, or ablation showing that benchmark scores predict real-robot transfer for novel behaviors rather than just in-distribution cases.

Authors: We thank the referee for this observation. The revised manuscript now reports the Pearson correlation coefficient (r = 0.87) between DreamGen Bench scores and policy success. We detail benchmark construction (50 prompts spanning in- and out-of-distribution behaviors), explicitly list the five held-out novel tasks, and include an ablation table separating correlations for in-distribution (r = 0.92) versus novel-behavior cases (r = 0.81). These additions confirm the benchmark's predictive utility for out-of-distribution transfer. revision: yes

Circularity Check

0 steps flagged

No significant circularity: empirical pipeline using external models and standard IDM

full rationale

The paper describes a 4-stage empirical pipeline that adapts external pre-trained image-to-video models to a robot embodiment, generates synthetic videos, recovers pseudo-actions via a latent action model or standard IDM, and trains policies on the resulting data. The headline result (22 new behaviors from one pick-and-place teleop dataset) is presented as an experimental outcome on hardware, not as a quantity derived by construction from fitted parameters inside the paper. DreamGen Bench is introduced as an independent evaluation tool whose correlation with policy success is measured post-hoc rather than used to define the success metric. No self-definitional equations, fitted-input predictions, or load-bearing self-citations that reduce the central claim to its own inputs appear in the derivation chain. The method therefore remains self-contained against external benchmarks and pre-trained components.

Axiom & Free-Parameter Ledger

1 free parameters · 2 axioms · 0 invented entities

The pipeline rests on the assumption that existing video generative models can be adapted to produce useful robot data and that standard action-recovery methods suffice; no new entities are postulated and only modest adaptation hyperparameters are introduced.

free parameters (1)

video-model adaptation hyperparameters
Parameters chosen to fine-tune the generative model to the target robot embodiment.

axioms (2)

domain assumption Adapted image-to-video models can generate photorealistic and kinematically plausible robot trajectories for novel tasks and environments.
Invoked to justify the creation of synthetic training data.
domain assumption Latent action models or inverse-dynamics models recover action sequences from generated videos with sufficient accuracy for policy training.
Required to convert video output into usable training trajectories.

pith-pipeline@v0.9.0 · 5599 in / 1412 out tokens · 47338 ms · 2026-05-15T23:47:28.279676+00:00 · methodology

discussion (0)

Forward citations

Cited by 24 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

ALAM: Algebraically Consistent Latent Action Model for Vision-Language-Action Models
cs.RO 2026-05 unverdicted novelty 7.0

ALAM creates algebraically consistent latent action transitions from videos to act as auxiliary generative targets, raising robot policy success rates from 47.9% to 85.0% on MetaWorld MT50 and 94.1% to 98.1% on LIBERO.
One Token Per Frame: Reconsidering Visual Bandwidth in World Models for VLA Policy
cs.CV 2026-05 conditional novelty 7.0

Reducing visual input to one token per frame in VLA world models maintains or improves long-horizon performance on MetaWorld, LIBERO, and real-robot tasks.
NoiseGate: Learning Per-Latent Timestep Schedules as Information Gating in World Action Models
cs.RO 2026-05 unverdicted novelty 7.0

NoiseGate learns per-latent timestep schedules as an information-gating policy in diffusion-based world action models, yielding consistent gains on RoboTwin manipulation tasks.
EA-WM: Event-Aware Generative World Model with Structured Kinematic-to-Visual Action Fields
cs.CV 2026-05 unverdicted novelty 7.0

EA-WM generates more accurate robot world rollouts by projecting actions as structured visual fields in camera space and using event-aware bidirectional fusion to better capture interaction dynamics.
3D Generation for Embodied AI and Robotic Simulation: A Survey
cs.RO 2026-04 accept novelty 7.0

3D generation for embodied AI is shifting from visual realism toward interaction readiness, organized into data generation, simulation environments, and sim-to-real bridging roles.
RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation
cs.RO 2026-04 unverdicted novelty 7.0

RoboWM-Bench evaluates video world models by converting their outputs into executable robot actions and running them on manipulation tasks, showing that physical inconsistencies remain common.
RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation
cs.RO 2026-04 unverdicted novelty 7.0

RoboWM-Bench evaluates video world models by converting their manipulation video predictions into executable actions validated in simulation, showing that visual plausibility does not guarantee physical executability.
PlayWorld: Learning Robot World Models from Autonomous Play
cs.RO 2026-03 unverdicted novelty 7.0

PlayWorld learns high-fidelity robot world models from unsupervised self-play, producing physically consistent video predictions that outperform models trained on human data and enabling 65% better real-world policy p...
ALAM: Algebraically Consistent Latent Action Model for Vision-Language-Action Models
cs.RO 2026-05 unverdicted novelty 6.0

ALAM introduces algebraic consistency regularization on latent action transitions from videos, raising VLA success rates from 47.9% to 85.0% on MetaWorld MT50 and 94.1% to 98.1% on LIBERO.
One Token Per Frame: Reconsidering Visual Bandwidth in World Models for VLA Policy
cs.CV 2026-05 unverdicted novelty 6.0

Reducing visual input to one token per frame in world models for vision-language-action policies maintains long-horizon performance while improving success rates on MetaWorld, LIBERO, and real-robot tasks.
One Token Per Frame: Reconsidering Visual Bandwidth in World Models for VLA Policy
cs.CV 2026-05 unverdicted novelty 6.0

Reducing visual input to one token per frame via adaptive attention pooling and a unified flow-matching objective improves long-horizon performance in VLA policies on MetaWorld, LIBERO, and real-robot tasks.
Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulation
cs.RO 2026-04 unverdicted novelty 6.0

Lucid-XR uses XR-headset physics simulation and physics-guided video generation to create synthetic data that trains robot policies transferring zero-shot to unseen real-world manipulation tasks.
ExoActor: Exocentric Video Generation as Generalizable Interactive Humanoid Control
cs.RO 2026-04 unverdicted novelty 6.0

ExoActor uses exocentric video generation to implicitly model robot-environment-object interactions and converts the resulting videos into task-conditioned humanoid control sequences.
Human Cognition in Machines: A Unified Perspective of World Models
cs.RO 2026-04 unverdicted novelty 6.0

The paper introduces a unified framework for world models that fully incorporates all cognitive functions from Cognitive Architecture Theory, highlights under-researched areas in motivation and meta-cognition, and pro...
VAG: Dual-Stream Video-Action Generation for Embodied Data Synthesis
cs.RO 2026-04 unverdicted novelty 6.0

VAG is a synchronized dual-stream flow-matching framework that generates aligned video-action pairs for synthetic embodied data synthesis and policy pretraining.
Fast-WAM: Do World Action Models Need Test-time Future Imagination?
cs.CV 2026-03 unverdicted novelty 6.0

Fast-WAM shows that explicit future imagination at test time is not required for strong WAM performance; video modeling during training provides the main benefit.
Simulation Distillation: Pretraining World Models in Simulation for Rapid Real-World Adaptation
cs.RO 2026-03 unverdicted novelty 6.0

SimDist pretrains world models in simulation and adapts them to real-world robots by updating only the latent dynamics model, enabling rapid improvement on contact-rich tasks where prior methods fail.
Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning
cs.AI 2026-01 conditional novelty 6.0

Single-stage fine-tuning of a video model to generate actions as latent frames plus future states and values yields state-of-the-art robot policy performance on LIBERO, RoboCasa, and bimanual tasks.
Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation
cs.RO 2025-08 unverdicted novelty 6.0

Genie Envisioner unifies robotic policy learning, simulation, and evaluation inside one instruction-conditioned video diffusion framework using GE-Base, GE-Act, and GE-Sim.
Embody4D: A Generalist 4D World Model for Embodied AI
cs.CV 2026-05 unverdicted novelty 5.0

Embody4D generates high-fidelity, view-consistent novel views from monocular videos for embodied scenarios via 3D-aware data synthesis, adaptive noise injection, and interaction-aware attention.
Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modeling
cs.CV 2026-04 unverdicted novelty 5.0

Visual generation models are evolving from passive renderers to interactive agentic world modelers, but current systems lack spatial reasoning, temporal consistency, and causal understanding, with evaluations overemph...
World Simulation with Video Foundation Models for Physical AI
cs.CV 2025-10 unverdicted novelty 4.0

Cosmos-Predict2.5 unifies text-to-world, image-to-world, and video-to-world generation in one model trained on 200M clips with RL post-training, delivering improved quality and control for physical AI.
3D Generation for Embodied AI and Robotic Simulation: A Survey
cs.RO 2026-04 unverdicted novelty 3.0

The survey organizes 3D generation for embodied AI into data generators for assets, simulation environments for interaction, and sim-to-real bridges, noting a shift toward interaction readiness and listing bottlenecks...
3D Generation for Embodied AI and Robotic Simulation: A Survey
cs.RO 2026-04 unverdicted novelty 2.0

The paper surveys 3D generation techniques for embodied AI and robotics, categorizing them into data generation, simulation environments, and sim-to-real bridging while identifying bottlenecks in physical validity and...

Reference graph

Works this paper leans on

79 extracted references · 79 canonical work pages · cited by 18 Pith papers · 22 internal anchors

[1]

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, I. Leal, L. Lee, T.-W. E. Lee, S. Levine, Y . Lu, H. Michalewski, I. Mordatch, K. Pe...

work page internal anchor Pith review Pith/arXiv arXiv 2023
[2]

$\pi_0$: A Vision-Language-Action Flow Model for General Robot Control

K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. π0: A vision-language-action flow model for general robot control, 2024.URL https://arxiv. org/abs/2410.24164

work page internal anchor Pith review Pith/arXiv arXiv 2024
[3]

G. R. Team, S. Abeyruwan, J. Ainslie, J.-B. Alayrac, M. G. Arenas, T. Armstrong, A. Balakrishna, R. Baruch, M. Bauza, M. Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:2503.20020, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025
[4]

Q. Bu, J. Cai, L. Chen, X. Cui, Y . Ding, S. Feng, S. Gao, X. He, X. Huang, S. Jiang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503.06669, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025
[5]

GR00T N1: An Open Foundation Model for Generalist Humanoid Robots

J. Bjorck, F. Casta ˜neda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y . Fang, D. Fox, F. Hu, S. Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025
[6]

$\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization

P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. π0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054 , 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025
[7]

Cosmos World Foundation Model Platform for Physical AI

N. Agarwal, A. Ali, M. Bala, Y . Balaji, E. Barker, T. Cai, P. Chattopadhyay, Y . Chen, Y . Cui, Y . Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025
[8]

Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y . Yang, W. Hong, X. Zhang, G. Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024

work page internal anchor Pith review Pith/arXiv arXiv 2024
[9]

A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, J. Zeng, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025
[10]

W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024

work page internal anchor Pith review Pith/arXiv arXiv 2024
[11]

Z. Lin, W. Liu, C. Chen, J. Lu, W. Hu, T.-J. Fu, J. Allardice, Z. Lai, L. Song, B. Zhang, et al. Stiv: Scalable text and image conditioned video generation. arXiv preprint arXiv:2412.07730, 2024

work page arXiv 2024
[12]

Pandora: Towards general world model with natural language actions and video states

J. Xiang, G. Liu, Y . Gu, Q. Gao, Y . Ning, Y . Zha, Z. Feng, T. Tao, S. Hao, Y . Shi, et al. Pandora: Towards general world model with natural language actions and video states. arXiv preprint arXiv:2406.09455, 2024

work page arXiv 2024
[13]

S. Ye, J. Jang, B. Jeon, S. J. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y .-W. Chao, B. Y . Lin, L. Liden, K. Lee, J. Gao, L. Zettlemoyer, D. Fox, and M. Seo. Latent action pretraining from videos. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id= VYOe2eBQeh

work page 2025
[14]

Baker, I

B. Baker, I. Akkaya, P. Zhokov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639–24654, 2022. 10

work page 2022
[15]

Y . Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. Advances in neural information processing systems , 36:9156–9172, 2023

work page 2023
[16]

S. Zhou, Y . Du, J. Chen, Y . Li, D.-Y . Yeung, and C. Gan. Robodreamer: Learning compositional world models for robot imagination. arXiv preprint arXiv:2404.12377, 2024

work page internal anchor Pith review Pith/arXiv arXiv 2024
[17]

P.-C. Ko, J. Mao, Y . Du, S.-H. Sun, and J. B. Tenenbaum. Learning to act from actionless videos through dense correspondences. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=Mhb5fpA1T0

work page 2024
[18]

S. Yang, Y . Du, S. K. S. Ghasemipour, J. Tompson, L. P. Kaelbling, D. Schuurmans, and P. Abbeel. Learning interactive real-world simulators. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=sFyTZEqmUY

work page 2024
[19]

Y . Du, S. Yang, P. Florence, F. Xia, A. Wahid, brian ichter, P. Sermanet, T. Yu, P. Abbeel, J. B. Tenenbaum, L. P. Kaelbling, A. Zeng, and J. Tompson. Video language planning. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=9pKtcJcMP3

work page 2024
[20]

Nasiriany, A

S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y . Zhu. Robocasa: Large- scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems (RSS) , 2024

work page 2024
[21]

E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022

work page 2022
[22]

DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945, 2024

work page internal anchor Pith review Pith/arXiv arXiv 2024
[23]

Baker, I

B. Baker, I. Akkaya, P. Zhokhov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos, 2022. URL https://arxiv. org/abs/2206.11795

work page arXiv 2022
[24]

C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023

work page 2023
[25]

B. Kang, Y . Yue, R. Lu, Z. Lin, Y . Zhao, K. Wang, G. Huang, and J. Feng. How far is video generation from world model: A physical law perspective. arXiv preprint arXiv:2411.02385, 2024

work page arXiv 2024
[26]

Bansal, Z

H. Bansal, Z. Lin, T. Xie, Z. Zong, M. Yarom, Y . Bitton, C. Jiang, Y . Sun, K.-W. Chang, and A. Grover. Videophy: Evaluating physical commonsense for video generation. arXiv preprint arXiv:2406.03520, 2024

work page arXiv 2024
[27]

Motamed, L

S. Motamed, L. Culp, K. Swersky, P. Jaini, and R. Geirhos. Do generative video models learn physical principles from watching videos? arXiv preprint arXiv:2501.09038, 2025

work page arXiv 2025
[28]

Duan, H.-X

H. Duan, H.-X. Yu, S. Chen, L. Fei-Fei, and J. Wu. Worldscore: A unified evaluation benchmark for world generation. arXiv preprint arXiv:2504.00983, 2025

work page arXiv 2025
[29]

S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025
[30]

Mandlekar, S

A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y . Narang, L. Fan, Y . Zhu, and D. Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. In Conference on Robot Learning , 2023

work page 2023
[31]

James, Z

S. James, Z. Ma, D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environ- ment. IEEE Robotics and Automation Letters , 5(2):3019–3026, 2020

work page 2020
[32]

Dalal, A

M. Dalal, A. Mandlekar, C. R. Garrett, A. Handa, R. Salakhutdinov, and D. Fox. Imitating task and motion planning with visuomotor transformers. In Conference on Robot Learning, pages 2565–2593. PMLR, 2023. 11

work page 2023
[33]

J. Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, Y . Tang, S. Tao, X. Wei, Y . Yao, et al. Maniskill2: A uni- fied benchmark for generalizable manipulation skills. In The Eleventh International Conference on Learning Representations, 2023

work page 2023
[34]

H. Ha, P. Florence, and S. Song. Scaling up and distilling down: Language-guided robot skill acquisition. In Conference on Robot Learning, pages 3766–3777. PMLR, 2023

work page 2023
[35]

Jiang, Y

Z. Jiang, Y . Xie, K. Lin, Z. Xu, W. Wan, A. Mandlekar, L. Fan, and Y . Zhu. Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning. 2024

work page 2024
[36]

Y . Wang, Z. Xian, F. Chen, T.-H. Wang, Y . Wang, K. Fragkiadaki, Z. Erickson, D. Held, and C. Gan. Robo- gen: Towards unleashing infinite data for automated robot learning via generative simulation. In International Conference on Machine Learning, 2024

work page 2024
[37]

Y . Su, S. Zhou, Y . Wu, T. Su, D. Liang, J. Liu, D. Zheng, Y . Wang, J. Yan, and X. Hu. Dynamic multi-path neural network. arXiv preprint arXiv:1902.10949, 2019

work page internal anchor Pith review Pith/arXiv arXiv 1902
[38]

Garrett, A

C. Garrett, A. Mandlekar, B. Wen, and D. Fox. Skillmimicgen: Automated demonstration generation for efficient skill learning and deployment. arXiv preprint arXiv:2410.18907, 2024

work page arXiv 2024
[39]

L. Yang, H. Suh, T. Zhao, B. P. Graesdal, T. Kelestemur, J. Wang, T. Pang, and R. Tedrake. Physics-driven data generation for contact-rich manipulation via trajectory optimization. arXiv preprint arXiv:2502.20382, 2025

work page arXiv 2025
[40]

Cacti: A framework for scalable multi-task multi-scene visual imitation learn- ing

Z. Mandi, H. Bharadhwaj, V . Moens, S. Song, A. Rajeswaran, and V . Kumar. Cacti: A framework for scalable multi-task multi-scene visual imitation learning. arXiv preprint arXiv:2212.05711, 2022

work page arXiv 2022
[41]

T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, J. Peralta, B. Ichter, et al. Scaling robot learning with semantically imagined experience. arXiv preprint arXiv:2302.11550, 2023

work page arXiv 2023
[42]

Z. Chen, S. Kiami, A. Gupta, and V . Kumar. Genaug: Retargeting behaviors to unseen situations via generative augmentation. arXiv preprint arXiv:2302.06671, 2023

work page arXiv 2023
[43]

L. Y . Chen, C. Xu, K. Dharmarajan, M. Z. Irshad, R. Cheng, K. Keutzer, M. Tomizuka, Q. Vuong, and K. Gold- berg. Rovi-aug: Robot and viewpoint augmentation for cross-embodiment robot learning. arXiv preprint arXiv:2409.03403, 2024

work page arXiv 2024
[44]

H. A. Alhaija, J. Alvarez, M. Bala, T. Cai, T. Cao, L. Cha, J. Chen, M. Chen, F. Ferroni, S. Fidler, et al. Cosmos- transfer1: Conditional world generation with adaptive multimodal control. arXiv preprint arXiv:2503.14492 , 2025

work page arXiv 2025
[45]

Liang, R

J. Liang, R. Liu, E. Ozguroglu, S. Sudhakar, A. Dave, P. Tokmakov, S. Song, and C. V ondrick. Dreamitate: Real- world visuomotor policy learning via video generation. In 8th Annual Conference on Robot Learning , 2024. URL https://openreview.net/forum?id=InT87E5sr4

work page 2024
[46]

Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation

H. Bharadhwaj, D. Dwibedi, A. Gupta, S. Tulsiani, C. Doersch, T. Xiao, D. Shah, F. Xia, D. Sadigh, and S. Kir- mani. Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation. arXiv preprint arXiv:2409.16283, 2024

work page internal anchor Pith review Pith/arXiv arXiv 2024
[47]

C. Luo, Z. Zeng, Y . Du, and C. Sun. Solving new tasks by adapting internet video knowledge. In The Thirteenth International Conference on Learning Representations , 2025

work page 2025
[48]

H. Wu, Y . Jing, C. Cheang, G. Chen, J. Xu, X. Li, M. Liu, H. Li, and T. Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. arXiv preprint arXiv:2312.13139, 2023

work page internal anchor Pith review Pith/arXiv arXiv 2023
[49]

GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation

C.-L. Cheang, G. Chen, Y . Jing, T. Kong, H. Li, Y . Li, Y . Liu, H. Wu, J. Xu, Y . Yang, et al. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158, 2024

work page internal anchor Pith review Pith/arXiv arXiv 2024
[50]

Y . Guo, Y . Hu, J. Zhang, Y .-J. Wang, X. Chen, C. Lu, and J. Chen. Prediction with action: Visual policy learning via joint denoising process. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

work page 2024
[51]

S. Li, Y . Gao, D. Sadigh, and S. Song. Unified video action model. arXiv preprint arXiv:2503.00200, 2025. 12

work page internal anchor Pith review Pith/arXiv arXiv 2025
[52]

C. Zhu, R. Yu, S. Feng, B. Burchfiel, P. Shah, and A. Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets. arXiv preprint arXiv:2504.02792, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025
[53]

Q. Zhao, Y . Lu, M. J. Kim, Z. Fu, Z. Zhang, Y . Wu, Z. Li, Q. Ma, S. Han, C. Finn, et al. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. arXiv preprint arXiv:2503.22020, 2025

work page internal anchor Pith review arXiv 2025
[54]

McCarthy, D

R. McCarthy, D. C. Tan, D. Schmidt, F. Acero, N. Herr, Y . Du, T. G. Thuruthel, and Z. Li. Towards generalist robot learning from internet video: A survey. arXiv preprint arXiv:2404.19664, 2024

work page arXiv 2024
[55]

Grauman, A

K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022

work page 2022
[56]

S. Nair, A. Rajeswaran, V . Kumar, C. Finn, and A. Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601, 2022

work page internal anchor Pith review Pith/arXiv arXiv 2022
[57]

Dasari, M

S. Dasari, M. K. Srirama, U. Jain, and A. Gupta. An unbiased look at datasets for visuo-motor pre-training. In Conference on Robot Learning, 2023

work page 2023
[58]

J. Zeng, Q. Bu, B. Wang, W. Xia, L. Chen, H. Dong, H. Song, D. Wang, D. Hu, P. Luo, et al. Learning manipulation by predicting interaction. arXiv preprint arXiv:2406.00439, 2024

work page arXiv 2024
[59]

S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak. Affordances from human videos as a versatile represen- tation for robotics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023

work page 2023
[60]

Kannan, K

A. Kannan, K. Shaw, S. Bahl, P. Mannam, and D. Pathak. Deft: Dexterous fine-tuning for real-world hand policies. arXiv preprint arXiv:2310.19797, 2023

work page arXiv 2023
[61]

M. K. Srirama, S. Dasari, S. Bahl, and A. Gupta. Hrp: Human affordances for robotic pre-training.arXiv preprint arXiv:2407.18911, 2024

work page arXiv 2024
[62]

K. Shaw, S. Bahl, and D. Pathak. Videodex: Learning dexterity from internet videos. In Conference on Robot Learning, 2023

work page 2023
[63]

C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y . Gao, and P. Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025, 2023

work page arXiv 2023
[64]

Bharadhwaj, R

H. Bharadhwaj, R. Mottaghi, A. Gupta, and S. Tulsiani. Track2act: Predicting point tracks from internet videos enables diverse zero-shot robot manipulation. arXiv preprint arXiv:2405.01527, 2024

work page arXiv 2024
[65]

C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y . Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302.12422, 2023

work page arXiv 2023
[66]

Y . Zhu, A. Lim, P. Stone, and Y . Zhu. Vision-based manipulation from single human video with open-world object graphs. arXiv preprint arXiv:2405.20321, 2024

work page arXiv 2024
[67]

Bharadhwaj, A

H. Bharadhwaj, A. Gupta, S. Tulsiani, and V . Kumar. Zero-shot robot manipulation from passive human videos. arXiv preprint arXiv:2302.02011, 2023

work page arXiv 2023
[68]

J. Ye, J. Wang, B. Huang, Y . Qin, and X. Wang. Learning continuous grasping function with a dexterous hand from human demonstrations. IEEE Robotics and Automation Letters , 8(5):2882–2889, 2023

work page 2023
[69]

Qin, Y .-H

Y . Qin, Y .-H. Wu, S. Liu, H. Jiang, R. Yang, Y . Fu, and X. Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, 2022

work page 2022
[70]

Yang, Z.-a

J. Yang, Z.-a. Cao, C. Deng, R. Antonova, S. Song, and J. Bohg. Equibot: Sim (3)-equivariant diffusion policy for generalizable and data efficient learning. arXiv preprint arXiv:2407.01479, 2024

work page arXiv 2024
[71]

Bruce, M

J. Bruce, M. Dennis, A. Edwards, J. Parker-Holder, Y . Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, Y . Aytar, S. Bechtle, F. Behbahani, S. Chan, N. Heess, L. Gonzalez, S. Osindero, S. Ozair, S. Reed, J. Zhang, K. Zolna, J. Clune, N. de Freitas, S. Singh, and T. Rockt¨aschel. Genie: Generative interactive environ- ments, 2024. URL https...

work page arXiv 2024
[72]

Y . Chen, Y . Ge, W. Tang, Y . Li, Y . Ge, M. Ding, Y . Shan, and X. Liu. Moto: Latent motion token as the bridging language for learning robot manipulation from videos, 2025. URL https://arxiv.org/abs/2412.04445

work page arXiv 2025
[73]

Schmidt and M

D. Schmidt and M. Jiang. Learning to act without actions. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=rvUq3cxpDF

work page 2024
[74]

Z. Ren, Y . Wei, X. Guo, Y . Zhao, B. Kang, J. Feng, and X. Jin. Videoworld: Exploring knowledge learning from unlabeled videos, 2025. URL https://arxiv.org/abs/2501.09781

work page arXiv 2025
[75]

Q. Bu, Y . Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li. Univla: Learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2505.06111, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025
[76]

S. Gao, S. Zhou, Y . Du, J. Zhang, and C. Gan. Adaworld: Learning adaptable world models with latent actions. arXiv preprint arXiv:2503.18938, 2025

work page arXiv 2025
[77]

Bansal, Y

H. Bansal, Y . Bitton, I. Szpektor, K.-W. Chang, and A. Grover. Videocon: Robust video-language alignment via contrast captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13927–13937, 2024

work page 2024
[78]

Use the right hand to pick up the plastic pitcher and pour water onto the green plant

R. Cadene, S. Alibert, A. Soare, Q. Gallouedec, and T. Wolf. Lerobot: Making ai for robotics more accessible with end-to-end learning, 2024. URL https://github.com/huggingface/lerobot. Accessed: 2025-04-30. 14 Table 3: LAPA Training Dataset Statistics Dataset Length (Frames) Duration (hr) FPS Category GR-1 Teleop Pre-Training 6.4M 88.4 20 Real robot DexMG...

work page 2024
[79]

High Data

We use a codebook size of 8 and a sequence length of 16 for vector quantization. We train 100K steps with a batch size of 1024. B Environment for Teleoperation and Evaluation We provide some sample images of the environment where we collected all of our GR1 humanoid teleoperation data in Figure 8 and all of the 10 environments where we conducted environme...

work page