Pith. sign in

REVIEW 5 major objections 6 minor 5 cited by

EmbodieDreamer: Advancing Real2Sim2Real Transfer for Policy Training via Embodied World Modeling

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training robot policies inside photorealistic simulated video raises real-world success by 29 percent.

desk verdict Promising integration of system ID and video diffusion for sim2real, but the 29.17% RL gain lacks a plain-simulator control so it cannot be attributed to photorealism. read the letter →

arxiv 2507.05198 v1 pith:Q3TUDAGV submitted 2025-07-07 cs.RO cs.AIcs.CV

Boyuan Wang , Xinpan Meng , Xiaofeng Wang , Zheng Zhu , Angen Ye , Yang Wang , Zhiqin Yang , Chaojun Ni
show 2 more authors
Guan Huang Xingang Wang
This is my paper · ORCID
classification cs.ROcs.AIcs.CV
keywords Real2Sim2Realrobotpolicylearningdifferentiablephysicssystemidentificationconditionalvideodiffusionworldmodelpreferenceoptimizationsim-to-realtransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the gap between simulation and reality in robot learning can be narrowed from both directions at once: calibrating the simulator's physics to match real robot motion, and upgrading the simulator's visuals to photorealistic video with a diffusion model. The authors build EmbodieDreamer, a framework whose PhysAligner module estimates control gains, stiffness, damping, and friction with a differentiable surrogate, and whose VisAligner module renders simulated states as realistic video. Using reinforcement learning inside this generated environment, an ACT policy improves average real-world task success across four manipulation tasks from 0.49 to 0.77, a 29.17% relative gain over the single-view supervised baseline. They also report that PhysAligner cuts physical parameter estimation error by 3.74% and speeds optimization by roughly an order of magnitude compared with simulated annealing. If these results hold, simulation alone could supply much of the visual and physical experience needed to improve real robot policies, reducing dependence on costly real-world data collection.

What carries the argument

The framework rests on three connected mechanisms. PhysAligner is a differentiable system identification pipeline: generate simulation rollouts under random physics parameters, fit an MLP surrogate to predict next states from parameters, actions, and states, then backpropagate through the surrogate to optimize friction, stiffness, and damping against real trajectories, avoiding the cost of simulated annealing. VisAligner is a latent video diffusion model built on Stable Video Diffusion that concatenates a real background reference image with simulated robot frames and a foreground object mask, denoising them into photorealistic videos that track the simulation state. For reinforcement learning, a trajectory preference loss adapted from GRAPE scores each rollout by the terminal Euclidean distance between the end-effector and the target, selects the top and bottom 25 trajectories as positive and negative pairs, and fine-tunes the ACT policy with a KL-regularized objective against the supervised reference policy.

What would settle it

Run the same preference-based RL pipeline as in Section 4.3 but feed unrendered simulator frames, rather than VisAligner videos, to the policy, and evaluate on the same four real tasks; if average success stays near 0.77, the photorealistic world model is not what drives the improvement.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a combined physical and visual calibration makes a simulator a viable training ground for real-world manipulation policies. PhysAligner trains an MLP surrogate to imitate the simulator's state transitions across randomly sampled friction, stiffness, and damping values, then refines those parameters by gradient descent against real end-effector trajectories, producing a differentiable, fast approximation of the simulator's non-smooth dynamics. VisAligner takes a real reference image for the static background, a simulated rendering of the robot, and a segmentation mask of the foreground object, and generates photorealistic videos aligned with simulation state. A single-view ACT policy fine-tuned by trajectory preference optimization in this environment reaches a 0.77 average real-world success rate across four tasks, up from 0.49 for the supervised baseline, and the same generated episodes improve imitation learning success on altered backgrounds and object colors.

Load-bearing premise

The load-bearing premise is that the preference labels for reinforcement learning are trustworthy: the paper ranks trajectories by the simulated distance between the end-effector and the target at the final step, and if that distance does not track whether the robot actually grasps and places the object in the real world, the policy would be trained on misaligned preferences and the reported 29.17% improvement would not follow.

Editorial extensions

If this is right

  • If the reported 29.17% improvement holds, a single-view policy trained with RL in the generated environment outperforms the three-view supervised baseline, reducing camera requirements.
  • PhysAligner's roughly tenfold speedup makes per-robot calibration with 50 sampled parameter sets practical, replacing slow simulated annealing for system identification.
  • VisAligner's disentangled robot, foreground, and background conditioning supports generating diverse new scenes from a single real episode without physically moving objects.
  • Imitation learning with generated episodes improves success on altered backgrounds and object colors for both ACT and π0 policies, compared with training only on real data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors do not compare the RL result against the same preference-based RL loop run on ordinary, unrendered simulation images; such an ablation would reveal how much of the 0.77 success rate comes from the photorealistic renderer rather than the physics calibration and preference optimization.
  • Because the terminal end-effector distance is used as the reward proxy, a simple check would be to correlate that simulated distance with real grasp and placement outcomes across the four tasks; the paper reports no such validation.
  • A natural stress test is misalignment between the calibrated camera and URDF overlay and the real robot pose, since VisAligner's video quality presumably depends on that registration; the paper does not measure how policy performance degrades as registration error grows.
  • The roughly two minutes per generated sequence makes closed-loop, real-time world-model interaction impractical at policy frequency; faster sampling or distillation would be needed before the video generator itself could act as a reflexive training environment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes EmbodieDreamer, a Real2Sim2Real framework composed of two modules: PhysAligner, a differentiable surrogate-based optimizer for simulator physical parameters (friction, stiffness, damping), and VisAligner, a conditional video diffusion model that renders low-fidelity simulated trajectories into photorealistic videos. The authors claim that PhysAligner reduces physical parameter estimation error by 3.74% over a simulated annealing baseline while achieving a 10x speedup, that VisAligner improves video generation metrics (FVD 176 vs. 422), and that reinforcement learning (RL) training in the photorealistic environment improves average real-world task success from 0.49 to 0.77, a 29.17% gain (Section 4.3, Table 3). The paper also presents imitation-learning (IL) data-augmentation experiments using generated videos.

Significance. If fully supported, the framework would be a valuable contribution to scalable robot policy learning, addressing both the physics and appearance components of the sim-to-real gap. The paper's strengths include a clear problem decomposition, a sensible video-diffusion design with disentangled foreground/background/robot conditioning, and promising IL results in some changed-scene settings. However, the headline RL result is currently not attributable to VisAligner because of confounding variables, and the PhysAligner accuracy metric is a training-fit error. These issues are fixable with additional experiments, so the paper's potential is visible but not yet established.

major comments (5)
  1. [Section 4.3, Table 3] The reported 29.17% average improvement is computed by comparing SFT with one view (0.49) against RL with EmbodieDreamer (0.77). This comparison bundles three simultaneous changes relative to the SFT baseline: (i) additional TPO/RL fine-tuning, (ii) the simulator's dynamics, which may be PhysAligner-aligned, and (iii) VisAligner's photorealistic rendering. Since no run trains the same ACT+TPO loop on unrendered (raw) simulator observations, the gain cannot be attributed to VisAligner or to photorealism, which is the component highlighted in the title and abstract. Please add a control condition that trains the same policy with the same preference-based RL loop using raw simulator renderings, and report all three conditions.
  2. [Section 4.1, Table 1, Eq. (4)] The 'trajectory error' reported for PhysAligner is the loss Lpara of Eq. (4), minimized on the same 20 RT-1 episodes used for parameter fitting. Reporting this training-fit MSE as the accuracy metric is circular: the 3.74% improvement over SimplerEnv reflects how well the optimized parameters fit the training trajectories, not how accurately they predict held-out real-world trajectories. Please evaluate the optimized parameters on held-out episodes (or report cross-validation) and distinguish fit error from generalization error.
  3. [Section 4.3, reward definition] The preference-based RL reward is defined as the Euclidean distance between the end-effector and the target position at the terminal state of each simulated trajectory. No evidence is provided that this proxy correlates with real-world grasp or placement success; if the proxy is misaligned, TPO may optimize a surrogate objective that does not correspond to task completion. Please provide a correlation analysis between the simulated terminal distance and real success labels, or add an ablation using an alternative reward (e.g., a task-specific success detector).
  4. [Table 3] Success rates in Table 3 are reported as point estimates without the number of trials per task, standard errors, or confidence intervals. Without trial counts, the reader cannot assess whether the 0.49 vs. 0.77 difference is statistically meaningful for each task. Please report the number of trials per cell and, ideally, confidence intervals or raw trial counts.
  5. [Appendix 2, Eq. (15)] The trajectory log-likelihood for the discrete-action ACT policy is asserted in Eq. (15) as a squared-prediction-error pseudo-likelihood, without derivation or empirical validation. Since the TPO loss of Eq. (16) is built on this quantity, the preference optimization signal depends on this assumption. Please either justify the Gaussian pseudo-likelihood for ACT actions or provide an ablation comparing this scoring to an alternative (e.g., a proper softmax-likelihood over action chunks).
minor comments (6)
  1. [Figure 1] The figure caption contains a garbled string of characters ('图⼤字⼤3个关系GRAPE论⽂颜⾊') that appears to be corrupted text; please replace it with a proper English caption.
  2. [Table 3] The table note lists settings '(1) trained from scratch with supervised fine-tuning (SFT), (2) without pre-training, and (3) with reinforcement learning (RL) training' but the rows only distinguish SFT with three views, SFT with one view, and RL with EmbodieDreamer; please align the note with the actual rows.
  3. [Section 4.4] The text says 'We conduct 30 test trials for each task setting' but Table 4 reports success rates that appear to be averages across 50 or 20 episodes for different columns; please clarify the exact number of trials per reported cell.
  4. [Section 4.2] The description of the data split (4,762 training and 100 test samples) does not state whether the split is at the episode level or the frame level; please clarify to avoid potential temporal leakage.
  5. [Appendix 1, Eqs. (8)-(10)] The appendix redefines (f,p,d) inconsistently with Section 3.2: in Section 3.2 these symbols denote friction, stiffness, and damping coefficients, while in Eq. (8) f is the input visual observation, p the ground-truth end-effector pose sequence, and d the diffusion model used for trajectory generation. This notation conflict should be resolved, for example by using different symbols for the physical parameters.
  6. [References] Reference [65] duplicates reference [27] (both are the ACT paper); please consolidate or cite the appropriate version once.

Circularity Check

1 steps flagged · score 6.0 of 10

PhysAligner's headline 3.74% error reduction is the in-sample value of the same MSE minimized in Eq. (4); the RL and video results are external and not circular.

  1. fitted input called prediction [Section 3.2, Eq. (4); Section 4.1, Table 1]
    "Lpara = 1/T sum_{t=1}^T ||P*_t - S_t||^2 ... This iterative process continues until the simulated output aligns closely with the observed trajectories ... Performance is evaluated using the mean squared error (MSE) between the predicted and ground-truth end-effector trajectories, and we additionally report the total computation time for data preparation and parameter optimization."

    Eq. (4) is the training objective for the physical parameters (f,p,d), and Sec. 4.1 states that the comparison in Table 1 is the MSE between predicted and ground-truth end-effector trajectories on the same 20 RT-1 episodes used for fitting, with no held-out split or cross-validation described. The reported 3.74% improvement (0.2245 versus 0.2161) is therefore the difference between the in-sample values of the objective that both methods minimize, not an out-of-sample estimation-error reduction. The fitted low-loss parameters are reported as if they had generalized 'error' rather than as fit quality on the training episodes.

full rationale

The central real-robot claim in Table 3 (single-view SFT 0.49 to RL with EmbodieDreamer 0.77, a 29.17% improvement) is not circular: success is measured on the physical robot and is not used to fit PhysAligner or VisAligner. The video-generation metrics (FVD, PSNR, SSIM, LPIPS) are likewise external to the training loss in Eq. (7). The one place where a headline number reduces to its own training objective is PhysAligner: Table 1's trajectory error is, on the paper's own description, the MSE between predicted and ground-truth trajectories, while Eq. (4) is exactly the MSE minimized on the same 20 RT-1 episodes, with no described held-out evaluation. Thus the 3.74% improvement is an in-sample fit comparison (gradient-based optimization reaching a lower value of the minimized loss than simulated annealing), not a validated estimation-error prediction. This is a partial circularity affecting the physics headline. The appendix's redefinition of (f,p,d) in Eqs. (8)-(10) is another correctness/documentation issue but not a circular step. Self-citations appear only in related-work examples and are not load-bearing.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rely on the adequacy of the simulator, the generalization of the video model, and hand-picked reward and hyperparameters; the fitted physics parameters are the primary free parameters, while M and beta are additional hand-set values.

free parameters (5)
  • stiffness = not reported
    Optimized by PhysAligner to match real trajectories; central to the Real2Sim alignment claim.
  • damping = not reported
    Optimized together with stiffness and friction in the PhysAligner gradient-based refinement.
  • friction = not reported
    Optimized together with stiffness and damping; the paper does not report the resulting values.
  • M, number of preferred and rejected trajectories = 25
    Hand-chosen hyperparameter for the preference-based RL loop (Section 4.3).
  • beta, preference regularization coefficient = not reported
    Appears in the TPO loss (Eq. 12 and Eq. 16) but its value is not given.
assumptions (5)
  • domain assumption The simulator with optimized stiffness, damping, and friction is an adequate model of the real robot's dynamics for policy transfer.
    All Real2Sim conclusions depend on this, yet no held-out or cross-configuration validation is provided.
  • domain assumption Replaying URDF joint trajectories with a manually calibrated virtual camera yields frames aligned with the real camera view.
    Section 3.3 uses this as the conditioning signal for VisAligner; small calibration errors would cause visual inconsistency.
  • domain assumption Terminal end-effector distance to the target in simulation is a valid reward proxy for real task success.
    Section 4.3 uses this distance to rank trajectories, but no correlation analysis with real success is given.
  • ad hoc to paper A Gaussian pseudo-likelihood over action prediction errors (Eq. 15) is a valid trajectory log-likelihood for ACT.
    Appendix 2 introduces this to make TPO tractable; it is not derived from ACT's actual action distribution.
  • domain assumption The latent video diffusion model generalizes to novel simulation frames and maintains temporal consistency.
    VisAligner's value depends on this; only 100 test videos are used for quantitative evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EmbodieDreamer: Advancing Real2Sim2Real Transfer for Policy Training via Embodied World Modeling." pith.science (2026). https://pith.science/paper/Q3TUDAGV

@misc{pith2026250705198,
  author       = {Pith},
  title        = {Pith review of: EmbodieDreamer: Advancing Real2Sim2Real Transfer for Policy Training via Embodied World Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q3TUDAGV}},
  note         = {Machine review of arXiv:2507.05198}
}
read the original abstract

The rapid advancement of Embodied AI has led to an increasing demand for large-scale, high-quality real-world data. However, collecting such embodied data remains costly and inefficient. As a result, simulation environments have become a crucial surrogate for training robot policies. Yet, the significant Real2Sim2Real gap remains a critical bottleneck, particularly in terms of physical dynamics and visual appearance. To address this challenge, we propose EmbodieDreamer, a novel framework that reduces the Real2Sim2Real gap from both the physics and appearance perspectives. Specifically, we propose PhysAligner, a differentiable physics module designed to reduce the Real2Sim physical gap. It jointly optimizes robot-specific parameters such as control gains and friction coefficients to better align simulated dynamics with real-world observations. In addition, we introduce VisAligner, which incorporates a conditional video diffusion model to bridge the Sim2Real appearance gap by translating low-fidelity simulated renderings into photorealistic videos conditioned on simulation states, enabling high-fidelity visual transfer. Extensive experiments validate the effectiveness of EmbodieDreamer. The proposed PhysAligner reduces physical parameter estimation error by 3.74% compared to simulated annealing methods while improving optimization speed by 89.91\%. Moreover, training robot policies in the generated photorealistic environment leads to a 29.17% improvement in the average task success rate across real-world tasks after reinforcement learning. Code, model and data will be publicly available.

Figures

Figures reproduced from arXiv: 2507.05198 by the authors.

Figure 1
Figure 1. EmbodieDreamer is a unified Real2Sim2Real framework that incorporates PhysAligner for rapid physical parameter optimization from real observations and VisAligner for generating visually realistic scenes. Furthermore, EmbodieDreamer supports RL training of policy models through preference learning based on trajectory evaluation, and facilitates IL training by generating diverse, unseen observations. Models trained wi… view at source ↗
Figure 2
Figure 2. EmbodieDreamer framework integrates PhysAligner and VisAligner to reduce the Real2Sim2Real gap in physics and appearance. PhysAligner optimizes simulator dynamics, while VisAligner translates simulated renderings into realistic observations for robot policy training. Simulation Dataset Generation Surrogate Modelling Gradient–based Refinement (𝑓, 𝑝, 𝑑) 𝑎!"# 𝑆!"# 🔥(𝑓, 𝑝, 𝑑) 𝑎!"# 𝑆!"# 🔥 ❄ 𝑃! 𝑃! ∗ 𝑆! 𝑃! ∗ 𝑃! (𝑓, 𝑝, 𝑑) 𝑎… view at source ↗
Figure 3
Figure 3. The figure illustrates the workflow of PhysAligner. First, a large amount of data is generated using a simulator. Then, a surrogate model is trained to fit the data. Finally, the physical parameters are optimized through gradient descent. system parameters, such as control gains, which are often determined empirically or through extensive trial and error. SimplerEnv [4] employs simulated annealing to optimize these … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The framework of VisAligner. A reference image containing the initial background and robot appearance information serves as the first frame of the conditioned video. The subsequent frames are generated by performing pixel-wise addition of the robot’s motion observation…
Figure 5
Figure 5. Figure 5: The visualization comparison of whether foreground object segmentation is used as a [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Visualization of two distinct trajectories generated by the policy model from a shared initial [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Definition and visualization of the three tasks. Each task is divided into two stages, with [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HALO-WA: Hybrid-Attention Latent-Guided Online Reinforcement Learning for World-Action Models

    cs.RO 2026-07 conditional novelty 6.0 of 10

    A hybrid-attention latent-guided online RL adapter lifts frozen world-action models from 26.4% to 87.1% average success on four real precision manipulation tasks in 45–75 minutes each.

  2. GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Long-horizon action-faithful consistency, not short-term visual realism, dominates world-model reliability for robot policy evaluation; GigaWorld-1 implements that roadmap and gains 14.9% on evaluator-alignment metrics.

  3. A Comprehensive Survey on World Models for Embodied AI

    cs.CV 2025-10 conditional novelty 6.0 of 10

    A unified three-axis taxonomy — functionality, temporal modeling, spatial representation — organizes the world-model literature for embodied AI.

  4. GigaWorld-Policy-0.5: A Faster and Stronger WAM Empowered by AutoResearch

    cs.RO 2026-07 conditional novelty 4.0 of 10

    GigaWorld-Policy-0.5 uses a Mixture-of-Transformers action-expert split and mixed world-model pretraining to reach 85 ms action-only inference with claimed success-rate gains.

  5. From World Models to World Action Models: A Concise Tutorial for Robotics

    cs.RO 2026-07 unverdicted novelty 4.0 of 10

    World models are action-conditioned predictors of task-relevant futures; world action models couple those futures to robot actions via four paradigms: imagine-then-execute, feature-conditioned, joint, and auxiliary pr...

Reference graph

Works this paper leans on

67 extracted references · 33 canonical work pages · cited by 5 Pith papers

  1. [1]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE, 2012. 1, 3

    work page 2012

  2. [2]

    Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 2020. 1, 3

    work page 2020

  3. [3]

    Chang, Leonidas J

    Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated part-based interactive environment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. 1, 3

    work page 2020

  4. [4]

    Evaluating real-world robot manipulation policies in simulation

    Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405.05941, 2024. 1, 2, 3, 5, 7

    arXiv 2024

  5. [5]

    Genesis: A universal and generative physics engine for robotics and beyond, December 2024

    Genesis Authors. Genesis: A universal and generative physics engine for robotics and beyond, December 2024. 1, 3

    work page 2024

  6. [6]

    Vlabench: A large-scale benchmark for language-conditioned robotics manipulation with long-horizon reasoning tasks, 2024

    Shiduo Zhang, Zhe Xu, Peiju Liu, Xiaopeng Yu, Yuan Li, Qinghui Gao, Zhaoye Fei, Zhangyue Yin, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. Vlabench: A large-scale benchmark for language-conditioned robotics manipulation with long-horizon reasoning tasks, 2024. 1, 3

    work page 2024

  7. [7]

    Roboverse: Towards a unified platform, dataset and benchmark for scalable and generalizable robot learning, April 2025

    Haoran Geng, Feishi Wang, Songlin Wei, Yuyang Li, Bangjun Wang, Boshi An, Charlie Tianyue Cheng, Haozhe Lou, Peihao Li, Yen-Jen Wang, Yutong Liang, Dylan Goetting, Chaoyi Xu, Haozhe Chen, Yuxi Qian, Yiran Geng, Jiageng Mao, Weikang Wan, Mingtong Zhang, Jiangran Lyu, Siheng Zhao, Jiazhao Zhang, Jialiang Zhang, Chengyang Zhao, Haoran Lu, Yufei Ding, Ran Gon...

    work page 2025

  8. [8]

    Robotwin: Dual-arm robot benchmark with generative digital twins

    Yao Mu, Tianxing Chen, Zanxin Chen, Shijia Peng, Zhiqian Lan, Zeyu Gao, Zhixuan Liang, Qiaojun Yu, Yude Zou, Mingkun Xu, et al. Robotwin: Dual-arm robot benchmark with generative digital twins. arXiv preprint arXiv:2504.13059, 2025. 1, 4

    arXiv 2025

Show all 67 references
  1. [9]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 202...

    2023 arXiv

  2. [10]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868,

    arXiv

  3. [11]

    Human- dreamer: Generating controllable human-motion videos via decoupled generation

    Boyuan Wang, Xiaofeng Wang, Chaojun Ni, Guosheng Zhao, Zhiqin Yang, Zheng Zhu, Muyang Zhang, Yukun Zhou, Xinze Chen, Guan Huang, Lihong Liu, and Xingang Wang. Human- dreamer: Generating controllable human-motion videos via decoupled generation. arXiv preprint arXiv:2503.24026, 2025. 2

    2025 arXiv

  4. [12]

    Humandreamer-x: Photorealistic single-image human avatars reconstruction via gaussian restoration

    Boyuan Wang, Runqi Ouyang, Xiaofeng Wang, Zheng Zhu, Guosheng Zhao, Chaojun Ni, Guan Huang, Lihong Liu, and Xingang Wang. Humandreamer-x: Photorealistic single-image human avatars reconstruction via gaussian restoration. arXiv preprint arXiv:2504.03536, 2025. 2, 4

    2025

  5. [13]

    Rebot: Scaling robot learning with real-to-sim-to-real robotic video synthesis, 2025

    Yu Fang, Yue Yang, Xinghao Zhu, Kaiyuan Zheng, Gedas Bertasius, Daniel Szafir, and Mingyu Ding. Rebot: Scaling robot learning with real-to-sim-to-real robotic video synthesis, 2025. 2, 4

    2025

  6. [14]

    Roboengine: Plug-and-play robot data augmentation with semantic robot segmentation and background generation

    Chengbo Yuan, Suraj Joshi, Shaoting Zhu, Hang Su, Hang Zhao, and Yang Gao. Roboengine: Plug-and-play robot data augmentation with semantic robot segmentation and background generation. arXiv preprint arXiv:2503.18738, 2025. 2 14

    2025 arXiv

  7. [15]

    Irasim: Learning interactive real-robot action simulators

    Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. Irasim: Learning interactive real-robot action simulators. arXiv:2406.12802, 2024. 2, 4

    2024 arXiv

  8. [16]

    Dexterous hand manipulation via efficient imitation-bootstrapped online reinforcement learning

    Dongchi Huang, Tianle Zhang, Yihang Li, Ling Zhao, Jiayi Li, Zhirui Fang, Chunhe Xia, Lusong Li, and Xiaodong He. Dexterous hand manipulation via efficient imitation-bootstrapped online reinforcement learning. arXiv preprint arXiv:2503.04014, 2025. 2

    2025

  9. [17]

    A survey of imitation learning: Algorithms, recent developments, and challenges

    Maryam Zare, Parham M Kebria, Abbas Khosravi, and Saeid Nahavandi. A survey of imitation learning: Algorithms, recent developments, and challenges. IEEE Transactions on Cybernetics,

  10. [18]

    Deep reinforcement learning for robotics: A survey of real-world successes

    Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martín-Martín, and Peter Stone. Deep reinforcement learning for robotics: A survey of real-world successes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 28694–28698,

  11. [19]

    Sim-to-real rein- forcement learning for vision-based dexterous manipulation on humanoids

    Toru Lin, Kartik Sachdev, Linxi Fan, Jitendra Malik, and Yuke Zhu. Sim-to-real rein- forcement learning for vision-based dexterous manipulation on humanoids. arXiv preprint arXiv:2502.20396, 2025. 2

    2025 arXiv

  12. [20]

    π0: A vision-language-action flow model for general robot control

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, Jam...

    arXiv

  13. [21]

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y . Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin L...

    2025

  14. [22]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213, 2024. 3

    2024 arXiv

  15. [23]

    Motion planning diffusion: Learning and planning of robot motions with diffusion models

    Joao Carvalho, An T Le, Mark Baierl, Dorothea Koert, and Jan Peters. Motion planning diffusion: Learning and planning of robot motions with diffusion models. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1916–1923. IEEE,

    2023

  16. [24]

    3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations

    Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. arXiv preprint arXiv:2403.03954, 2024. 3

    2024 arXiv

  17. [25]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023. 3

    2023

  18. [26]

    Grape: Generalizing robot policy via preference alignment, 2024

    Zijian Zhang, Kaiyuan Zheng, Zhaorun Chen, Joel Jang, Yi Li, Chaoqi Wang, Mingyu Ding, Dieter Fox, and Huaxiu Yao. Grape: Generalizing robot policy via preference alignment, 2024. 3, 9, 12

    2024

  19. [28]

    Rt-2: Vision-language- action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language- action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818,

    arXiv

  20. [29]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023. 3

    2023

  21. [30]

    Rdt-1b: a diffusion foundation model for bimanual manipulation

    Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864, 2024. 3

    2024 arXiv

  22. [31]

    Reactive diffusion policy: Slow-fast visual-tactile policy learning for contact-rich manipulation

    Han Xue, Jieji Ren, Wendi Chen, Gu Zhang, Yuan Fang, Guoying Gu, Huazhe Xu, and Cewu Lu. Reactive diffusion policy: Slow-fast visual-tactile policy learning for contact-rich manipulation. arXiv preprint arXiv:2503.02881, 2025. 3

    2025 arXiv

  23. [32]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla...

    arXiv

  24. [33]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source general...

  25. [34]

    Dexgraspvla: A vision-language-action framework towards general dexterous grasping

    Yifan Zhong, Xuchuan Huang, Ruochong Li, Ceyao Zhang, Yitao Liang, Yaodong Yang, and Yuanpei Chen. Dexgraspvla: A vision-language-action framework towards general dexterous grasping. arXiv preprint arXiv:2502.20900, 2025. 3

    2025

  26. [35]

    Cot-vla: Visual chain-of-thought reasoning for vision-language-action models

    Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, et al. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. arXiv preprint arXiv:2503.22020, 2025. 3

    2025 arXiv

  27. [36]

    Bridgedata v2: A dataset for robot learning at scale

    Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (...

    2023

  28. [37]

    Rt-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Y...

    2022 arXiv

  29. [38]

    MagicDrive: Street view generation with diverse 3d geometry control

    Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu. MagicDrive: Street view generation with diverse 3d geometry control. In International Conference on Learning Representations, 2024. 4

    2024

  30. [39]

    Drive- dreamer: Towards real-world-driven world models for autonomous driving

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drive- dreamer: Towards real-world-driven world models for autonomous driving. arXiv preprint arXiv:2309.09777, 2023. 4

    2023 arXiv

  31. [40]

    World- dreamer: Towards general world models for video generation via predicting masked tokens

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Boyuan Wang, Xinze Chen, and Jiwen Lu. World- dreamer: Towards general world models for video generation via predicting masked tokens. arXiv preprint arXiv:2401.09985, 2024. 4

    2024 arXiv

  32. [41]

    Gaia-1: A generative world model for autonomous driving

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080, 2023. 4 16

    2023 arXiv

  33. [42]

    Drivedreamer4d: World models are effective data machines for 4d driving scene representation

    Guosheng Zhao, Chaojun Ni, Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Boyuan Wang, Youyi Zhang, Wenjun Mei, and Xingang Wang. Drivedreamer4d: World models are effective data machines for 4d driving scene representation. arXiv preprint arXiv:2410.13571,

    arXiv

  34. [43]

    Street gaussians for modeling dynamic urban scenes

    Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street gaussians for modeling dynamic urban scenes. arXiv preprint arXiv:2401.01339, 2024. 4

    2024 arXiv

  35. [44]

    Recondreamer: Crafting world models for driving scene reconstruction via online restoration

    Chaojun Ni, Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Wenkang Qin, Guan Huang, Chen Liu, Yuyin Chen, Yida Wang, Xueyang Zhang, et al. Recondreamer: Crafting world models for driving scene reconstruction via online restoration. arXiv preprint arXiv:2411.19548, 2024. 4

    2024 arXiv

  36. [45]

    Tenenbaum

    Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B. Tenenbaum. Learning to act from actionless videos through dense correspondences, 2023. 4

    2023

  37. [46]

    Learning universal policies via text-guided video generation

    Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in neural information processing systems, 36:9156–9172, 2023. 4

    2023

  38. [47]

    Robo- Dreamer: Learning compositional world models for robot imagination

    Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. Robo- Dreamer: Learning compositional world models for robot imagination. In Ruslan Salakhutdi- nov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenka...

    2024

  39. [48]

    Manipdreamer: Boosting robotic manipulation world model with action tree and visual guidance, 2025

    Ying Li, Xiaobao Wei, Xiaowei Chi, Yuming Li, Zhongyu Zhao, Hao Wang, Ningning Ma, Ming Lu, and Shanghang Zhang. Manipdreamer: Boosting robotic manipulation world model with action tree and visual guidance, 2025. 4

    2025

  40. [49]

    Robotransfer: Geometry-consistent video diffusion for robotic visual policy transfer, 2025

    Liu Liu, Xiaofeng Wang, Guosheng Zhao, Keyu Li, Wenkang Qin, Jiaxiong Qiu, Zheng Zhu, Guan Huang, and Zhizhong Su. Robotransfer: Geometry-consistent video diffusion for robotic visual policy transfer, 2025. 4

    2025

  41. [50]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM ToG, 2023. 4

    2023

  42. [51]

    Nerf: Representing scenes as neural radiance fields for view synthesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 2021. 4

    2021

  43. [52]

    Recondreamer++: Harmonizing generative and reconstructive models for driving scene representation

    Guosheng Zhao, Xiaofeng Wang, Chaojun Ni, Zheng Zhu, Wenkang Qin, Guan Huang, and Xingang Wang. Recondreamer++: Harmonizing generative and reconstructive models for driving scene representation. arXiv preprint arXiv:2503.18438, 2025. 4

    2025 arXiv

  44. [53]

    Wonderturbo: Generating interactive 3d world in 0.72 seconds

    Chaojun Ni, Xiaofeng Wang, Zheng Zhu, Weijie Wang, Haoyun Li, Guosheng Zhao, Jie Li, Wenkang Qin, Guan Huang, and Wenjun Mei. Wonderturbo: Generating interactive 3d world in 0.72 seconds. arXiv preprint arXiv:2504.02261, 2025. 4

    2025 arXiv

  45. [54]

    Gs-gvins: A tightly-integrated gnss-visual-inertial navigation system augmented by 3d gaussian splatting

    Zelin Zhou, Saurav Uprety, Shichuang Nie, and Hongzhou Yang. Gs-gvins: A tightly-integrated gnss-visual-inertial navigation system augmented by 3d gaussian splatting. arXiv preprint arXiv:2502.10975, 2025. 4

    2025

  46. [55]

    Foundation model-guided gaussian splatting for 4d reconstruction of deformable tissues

    Yifan Liu, Chenxin Li, Hengyu Liu, Chen Yang, and Yixuan Yuan. Foundation model-guided gaussian splatting for 4d reconstruction of deformable tissues. IEEE Transactions on Medical Imaging, 44(6):2672–2682, 2025. 4

    2025

  47. [56]

    Og-mapping: Octree-based structured 3d gaussians for online dense mapping

    Meng Wang, Junyi Wang, Changqun Xia, Chen Wang, and Yue Qi. Og-mapping: Octree-based structured 3d gaussians for online dense mapping. arXiv preprint arXiv:2408.17223, 2024. 4

    2024 arXiv

  48. [57]

    Robo-gs: A physics consistent spatial-temporal model for robotic arm with hybrid representation, 2024

    Haozhe Lou, Yurong Liu, Yike Pan, Yiran Geng, Jianteng Chen, Wenlong Ma, Chenglong Li, Lin Wang, Hengzhen Feng, Lu Shi, Liyi Luo, and Yongliang Shi. Robo-gs: A physics consistent spatial-temporal model for robotic arm with hybrid representation, 2024. 4 17

    2024

  49. [58]

    Splatsim: Zero-shot sim2real transfer of rgb manipulation policies using gaussian splatting, 2024

    Mohammad Nomaan Qureshi, Sparsh Garg, Francisco Yandun, David Held, George Kantor, and Abhishesh Silwal. Splatsim: Zero-shot sim2real transfer of rgb manipulation policies using gaussian splatting, 2024. 4

    2024

  50. [59]

    Auto-encoding variational bayes, 2022

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. 6, 7

    2022

  51. [60]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 6

    2023 arXiv

  52. [61]

    Sam 2: Segment anything in images and videos, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenh...

    2024

  53. [62]

    Zhao, and Chelsea Finn

    Zipeng Fu, Tony Z. Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. In Conference on Robot Learning (CoRL), 2024. 7

    2024

  54. [63]

    Towards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 8

    2018 arXiv

  55. [64]

    The unrea- sonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018. 8

    2018

  56. [65]

    Learning fine-grained bimanual manipulation with low-cost hardware

    Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023. 8

    2023 arXiv

  57. [66]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 10

    2023 arXiv

  58. [67]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 12

    2017 arXiv

  59. [68]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.