Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulation

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FlowDreamer claims that explicitly predicting 3D scene flow before diffusion-based future-frame generation improves RGB-D world models for robot manipulation.

desk verdict Solid, honest two-stage RGB-D world model with explicit 3D scene flow; the single-frame bottleneck is real and acknowledged, and the abstract overclaims a bit, but it deserves a serious referee. read the letter →

arxiv 2505.10075 v1 pith:VIKJ3JYS submitted 2025-05-15 cs.RO cs.CV

Jun Guo , Xiaojian Ma , Yikai Wang , Min Yang , Huaping Liu , Qing Li This is my paper · ORCID
classification cs.ROcs.CV
keywords worldmodelsrobotmanipulationRGB-Dvideoprediction3Dsceneflowlatentdiffusionmodelpredictivecontrolaction-conditioneddynamicsmodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FlowDreamer tries to establish that an RGB-D world model predicts future frames more accurately when it does not leave dynamics implicit: it first predicts 3D scene flow, the displacement of every point in the observed scene from the current frame to the next, from a single RGB-D frame and the robot action, and then conditions a latent diffusion model on that flow to render the next RGB frame. The paper argues that conventional world models merge dynamics and rendering into one network, so frame-level training loss pushes them to polish appearances while neglecting motion, and its experiments support that by comparing against single-stage and two-stage RGB-D baselines. On two action-conditioned video prediction benchmarks and on visual planning tasks, the authors report about 7% higher semantic similarity, 11% better pixel quality, and 6% higher success rate. The method is trained end-to-end despite being modular, with the flow stage and diffusion stage sharing one joint loss. A sympathetic reading is that explicit 3D motion supervision is a cheap way to make learned simulators more physically predictive.

What carries the argument

The load-bearing object is 3D scene flow, the vector field that assigns to each pixel's reconstructed 3D point its displacement to the corresponding point one timestep later, $(x'-x, y'-y, z'-z)$. In simulation it is obtained from object pose matrices, and for real data from depth and optical-flow-style estimators such as RAFT-3D. The dynamics module is a conditional U-Net that takes the RGB-D frame and injects the action through cross-attention, trained with MSE against that flow; the generation module is a latent diffusion denoising U-Net conditioned on the current latent, downsampled depth, predicted flow, and action, with a pretrained VAE encoder and decoder plus a depth estimator for autoregressive future frames. The joint objective ties the two stages together: $L_{\text{diff}} + \alpha L_{\text{flow}}$, with $\alpha$ weighting the flow supervision. This provides a direct, dense supervision signal about motion that the renderer cannot ignore.

What would settle it

Run FlowDreamer on a Robosuite push episode in which the arm contacts a cube and stops instantly, and compare the predicted cube flow with the simulator's ground-truth flow: under the single-frame Markov assumption the network should predict zero cube motion, while ground truth shows the cube sliding with the velocity acquired during contact. Any predicted motion, or any improvement gained by feeding a second history frame, falsifies the assumption that current RGB-D plus action suffices.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that 3D scene flow is an effective explicit motion representation for action-conditioned RGB-D world models, and that separating dynamics prediction from visual rendering improves both prediction fidelity and downstream planning. The architecture has two stages: a conditional U-Net with cross-attention on the action predicts the scene flow $\hat{f}_{t\to t+1}$ from $(I_t, D_t, a_t)$; a fine-tuned latent diffusion denoising U-Net then takes the current RGB latent, the depth, the predicted flow, and the action to generate the next RGB observation. The whole system is trained with a joint loss $L_{\text{diff}} + \alpha L_{\text{flow}}$, where the flow loss is MSE against simulator-derived or estimator-derived ground truth. On video prediction, FlowDreamer outperforms the single-stage diffusion model, the MinkowskiNet two-stage baseline, and a separately trained two-stage model on most semantic, pixel, and media-quality metrics. On the VP2 visual planning benchmark, it improves average success rate over the single-stage model and is competitive with or better than prior video-prediction baselines. The authors also show that reversing the predicted flow degrades generation and that flow prediction error correlates strongly with pixel-level image metrics, evidence that the flow channel is what carries the motion information.

Load-bearing premise

The model assumes that a single RGB-D frame plus the current action determines how the whole scene will move next, with no need for history or velocity; objects that keep sliding after a push therefore cannot be predicted correctly, which the authors observe on Robosuite push tasks.

Editorial extensions

If this is right

  • Using FlowDreamer as the learned simulator in visual MPC raises manipulation success rates over the single-stage RGB-D diffusion world model by about 6% on VP2-style tasks.
  • Flow-supervised world models can be trained end-to-end without giving up modularity: the dynamics module and renderer improve together, so the two stages should not be optimized separately.
  • On real-world data, where flow targets come from estimators rather than simulator poses, FlowDreamer's advantage over the single-stage model is smaller than in simulation, indicating that supervision quality limits the gain.
  • Because flow error tracks pixel-level metrics such as SSIM, PSNR, and LPIPS, flow prediction accuracy can serve as a cheap proxy for frame-prediction quality during development.
  • Reversing the predicted flow degrades generation even when the action condition is unchanged, which means the flow channel carries task-relevant motion information, not just a generic style signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural, untested extension is to condition the dynamics module on one or two history frames or an explicit velocity estimate; given the paper's identified failure mode on post-push sliding, this should specifically recover Robosuite push-task accuracy.
  • Because 3D scene flow is defined in camera-independent displacement space, the flow-conditioned renderer may transfer to new camera poses or embodiments without retraining; the paper reports only fixed-camera benchmarks.
  • Replacing the MSE flow loss with a locally rigid or point-correspondence-aware loss could reduce sensitivity to noisy real-world flow targets and sharpen the dynamics module.
  • The weak correlation between flow error and semantic metrics such as DINOv2 and CLIP suggests that flow supervision buys pixel-level fidelity more than high-level semantic correctness; future work should test whether planning success tracks the former or the latter.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. FlowDreamer proposes a two-stage action-conditioned RGB-D world model for robot manipulation. In stage 1, a conditional U-Net predicts 3D scene flow from the current RGB-D frame and robot action, supervised by simulator-derived or RAFT-3D-estimated flow (Eq. 7). In stage 2, a latent diffusion model conditioned on the current RGB-D frame, the action, and the predicted flow generates the next RGB frame; training is end-to-end with loss L_diff + alpha L_flow (Eq. 11). The paper evaluates video prediction on SimplerEnv RT-1 and Language Table, visual MPC planning on VP2 (RoboDesk and Robosuite), a real-world RT-1 experiment in the appendix, and ablation analyses including reversed flow and flow-error correlation with image metrics. The central claim is that explicit flow-based motion modeling yields better future-frame prediction and planning than single-stage or two-stage RGB-D world models without explicit flow.

Significance. If the central claim holds, explicit 3D scene flow as an intermediate motion representation is a useful and interpretable design choice for robot visual world models, and the two-stage end-to-end training recipe is valuable. The paper has clear strengths: the flow supervision target is well defined for simulation via rigid-body transformations (Appendix B), the reversed-flow experiment (Sec. 4.3, Fig. 6) is a good falsifiable control showing that the flow condition carries information into the diffusion module, and the real-world pipeline (depth estimation plus RAFT-3D) addresses a practical data gap. However, the empirical support is weakened by the absence of significance tests or error bars on the main tables, by an abstract whose headline percentages are not cleanly derivable from the reported numbers, and by a single-frame design that is ill-posed in exactly the non-Markovian manipulation cases the paper targets. The idea is promising, but the evidence as presented supports a narrower, Markovian claim.

major comments (3)
  1. [Sec. 3.2, Eq. (7); Sec. 4.2; Appendix D] The flow-prediction objective is not a well-defined function of the model's inputs in the very manipulation settings the paper targets. Equation (7) supervises a mapping from (I_t, D_t, a_t) to f_hat_{t->t+1} with MSE against simulator-derived scene flow. In Robosuite push tasks, the same (I_t, D_t, a_t) can precede different next-frame motions because an object may be either stationary or still coasting after being released by the arm; the paper concedes this in Sec. 4.2 ('objects would have velocity after being pushed by the robot arm, which cannot be reflected by a single RGB-D image') and Appendix D ('we did not consider any context or history'). The MSE optimum for such ambiguous inputs is a conditional mean that is systematically wrong for both cases. This is not a peripheral caveat: it affects the manipulation tasks where dynamics matter most, and it undermines the claim that explicit flow prediction provides correct dynamics. I recommend either incorporating history/context into the flow predictor, or explicitly restricting the paper's generality claims to Markovian, quasi-static settings and providing evidence that the non-Markovian cases do not dominate the reported gains.
  2. [Abstract; Tables 1-2; Fig. 4] The abstract's headline improvements are not cleanly supported by the reported numbers. In Table 1, FlowDreamer's FVD (268.3853) is worse than SepTrain's (245.9106), so the claim of being best on 'most metrics' is not universal; the abstract's 7% semantic and 11% pixel improvements do not correspond to an identifiable calculation from the tables. Tables 1 and 2 report single point estimates with no standard errors or significance tests, and Fig. 4 reports only min/max across 3-4 seeds. Given that some margins are small (e.g., SSIM 0.8196 vs 0.8135 in Table 1; FVD separated by a much larger relative gap in the opposite direction), the central quantitative claims need error bars, significance tests, or a clear statement of which metrics and baselines define the percentages. The abstract should be revised to match the table facts.
  3. [Sec. 4.2; Sec. 4.1 baselines] The comparison against VP2 baselines is confounded by input context length. All VP2 video-prediction baselines (FitVid, SVG, MCVD, Struct-VRNN, MaskViT, iVideoGPT) use two frames of context, while FlowDreamer uses only a single frame plus an action. The observed success on some planning tasks could therefore reflect the explicit flow representation, the difference in available history, or both. A proper control would be a single-frame variant of a strong baseline or a two-frame variant of FlowDreamer without flow conditioning. Without such a control, the claim that explicit flow is the operative cause of the improvements is not yet established. The paper's own discussion of Robosuite failures in Sec. 4.2 makes this confound particularly salient.
minor comments (5)
  1. [Eq. (7)] The loss is written as MSELoss(f_hat - f), which is not a valid MSE expression; it should be MSELoss(f_hat, f) or ||f_hat - f||^2.
  2. [Eqs. (1), (3), (10)] The diffusion notation is inconsistent: Eq. (1) uses z_t on the right-hand side where z_k is intended, and Eq. (10) writes z^k_{t+1} while Eq. (3) uses z_k. Please unify the notation throughout Sec. 3.
  3. [Fig. 4 and references] The manuscript contains garbled text sequences in several figure captions and at least one reference (e.g., [29] begins 'Nmenicklas A Hansen'); these must be cleaned before a final submission.
  4. [Table 3 and Fig. 7] Please clarify on which data split and which predicted flows the 'flow error' and image metrics are computed; reporting per-task scatter plots or a decomposition by task would make the correlation analysis more interpretable, especially since the DINOv2 and CLIP correlations are weak.
  5. [Sec. 4.1] The sentence stating that SepTrain's similar performance 'indicates that end-to-end training is generally a better approach' goes beyond what a single pair of point estimates can support; softening this claim or adding significance testing would be more appropriate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: stage-1 flow targets are external, stage-2 targets are actual next frames, and no self-citation chain carries the argument.

full rationale

FlowDreamer's derivation chain is self-contained against external targets. Stage 1's loss (Eq. 7) is MSE between the U-Net output and simulator-derived scene flow or RAFT-3D estimates; the simulator target is computed from pose matrices via Eq. 12, and the real-world target comes from an external estimator, not from the model's own predictions. Stage 2's diffusion loss (Eq. 3) targets the actual next-frame latent z_{t+1}, with the denoising U-Net conditioned on the current frame, action, and predicted flow (Eqs. 8-10); the total loss (Eq. 11) is a weighted sum of these two externally supervised objectives. No fitted scalar or learned parameter is renamed as a prediction, and no load-bearing claim reduces to a self-citation: the cited flow/world-model works are by other research groups, and the central comparisons (Tables 1-2, Fig. 4) are against independently trained baseline models evaluated on the same held-out next-frame targets. The paper's own admitted limitation in Sec. 4.2 and Appendix D—that Robosuite push objects retain velocity that a single RGB-D image cannot reflect, so scene flow is not a function of (I_t,D_t,a_t) alone—is a correctness and generality caveat about non-Markovian dynamics, not a circularity: the target flows remain external and the loss still compares predicted flow against those external targets. The correlation analysis in Table 3 is descriptive post-hoc evidence rather than part of the training derivation. Overall, the claimed predictions are not equivalent to their inputs by construction, and the central contribution retains independent empirical content.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The method rests on the availability of scene flow targets (simulator poses or estimated flow) and on a single-frame Markov assumption; no new physical entities are postulated.

free parameters (1)
  • alpha (flow loss weight)
    In Eq. 11, L_total = L_diff + alpha * L_flow. The paper defines alpha as a coefficient but never reports its value or selection procedure, so the balance between the two training objectives is an unreported hand-set parameter.
assumptions (4)
  • domain assumption Ground-truth scene flow in simulation is the per-object rigid motion computed from simulator pose matrices (Eq. 12).
    Assumes every visible object or articulated part moves rigidly between frames, so a single 4x4 transform per part fully determines pixel motion; non-rigid or occluded motion would violate the target definition.
  • domain assumption The next RGB frame is determined by the current RGB-D frame, the action, and the scene flow, with no additional hidden state.
    The architecture conditions on a single RGB-D frame and action only; limitations for objects with post-push velocity are acknowledged in §4.2 and Appendix D.
  • domain assumption Pretrained Stable Diffusion 2.1 and its VAE provide an adequate prior for robot observation rendering.
    The future-generation module is initialized from Stable Diffusion and its autoencoder, inheriting their biases; no analysis of failure modes from this choice is provided.
  • domain assumption For real-world data, RAFT-3D/RAFT and depth estimators provide sufficiently reliable flow targets to train the dynamics module.
    Appendix C shows estimated flows contain noise that reduces performance, and the authors acknowledge this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulation." pith.science (2026). https://pith.science/paper/VIKJ3JYS

@misc{pith2026250510075,
  author       = {Pith},
  title        = {Pith review of: FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VIKJ3JYS}},
  note         = {Machine review of arXiv:2505.10075}
}
read the original abstract

This paper investigates training better visual world models for robot manipulation, i.e., models that can predict future visual observations by conditioning on past frames and robot actions. Specifically, we consider world models that operate on RGB-D frames (RGB-D world models). As opposed to canonical approaches that handle dynamics prediction mostly implicitly and reconcile it with visual rendering in a single model, we introduce FlowDreamer, which adopts 3D scene flow as explicit motion representations. FlowDreamer first predicts 3D scene flow from past frame and action conditions with a U-Net, and then a diffusion model will predict the future frame utilizing the scene flow. FlowDreamer is trained end-to-end despite its modularized nature. We conduct experiments on 4 different benchmarks, covering both video prediction and visual planning tasks. The results demonstrate that FlowDreamer achieves better performance compared to other baseline RGB-D world models by 7% on semantic similarity, 11% on pixel quality, and 6% on success rate in various robot manipulation domains.

Figures

Figures reproduced from arXiv: 2505.10075 by the authors.

Figure 1
Figure 1. Proposed RGB-D world model with flow-based mo￾tion representations. FlowDreamer adopts a two-stage prediction framework, which explicitly predict scene flow as motion repre￾sentations. FlowDreamer achieves better results on future frame prediction and visual planning tasks in various robot manipulation domains. planning algorithms [63, 87, 94, 98]. The use of visual world models alleviates the need for precise scene… view at source ↗
Figure 2
Figure 2. Overview of FlowDreamer. At stage 1, FlowDreamer receives the RGB-D frame and the robot action as input to explicitly predict the scene flow as motion representations. At stage 2, FlowDreamer leverages a denoising U-Net to generate high-resolution next￾step future observation via diffusion. tion. This approach is limited to rigid bodies and requires the world model to possess object-centric awareness, which requires… view at source ↗
Figure 3
Figure 3. Qualitative results on the SimplerEnv RT-1 and Language Table benchmark. We show the predicted frames and the scene flows except for Vanilla, where only RGB frames are being predicted. The R, G, and B channel values in the flow visualization represent the components of the 3D scene flow along the x, y, and z directions, respectively, normalized by the maximum value of the scene flow. RT-1 and Language Table benchmar… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Visual planning results on the VP2 benchmark. We report the mean and the min/max performance of different methods over multiple runs with different random seeds. On the right, “Average” means the average success rate over all reported tasks. GT Vanilla Ours GT Vanilla …
Figure 5
Figure 5. Figure 5: Qualitative results on the Robodesk and Robosuite dataset. The trajectory comes from the validation set, which is split from the original training trajectories and is not used for training. For our method, we show the predicted RGB images and scene flows. boDesk [41] t…
Figure 6
Figure 6. Figure 6: The qualitative results when flows are reversed. With reversed (therefore incorrect) scene flow, the diffusion model in FlowDreamer can only utilize action condition, leading to worse performances on future frame prediction. 10 20 Flow Error 0.85 0.90 0.95 SSIM Correla…
Figure 7
Figure 7. Figure 7: The correlation between the flow prediction error and image assessment metrics. We show the scatter plots of SSIM (higher is better), LPIPS (lower is better), DINOv2 L2 (lower is better), and CLIP score (higher is better) vs. flow er￾ror and report the correlation coef…
Figure 8
Figure 8. Figure 8: Scene flow obtainment pipeline on the real-world dataset. The consistent metric depth is estimated by Video Depth Anything and Depth Anything v2, and the 3D scene flow is esti￾mated by RAFT-3D and refined by RAFT. For RT-1 real world datasets, as there is no ground tru…
Figure 9
Figure 9. Figure 9: Qualitative results on the RT-1 real world dataset. The depth is estimated by Depth Anything v2, and the flow in “GT” are estimated by RAFT-3D, making it a harder scenario. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Comprehensive Survey on World Models for Embodied AI

    cs.CV 2025-10 conditional novelty 6.0 of 10

    A unified three-axis taxonomy — functionality, temporal modeling, spatial representation — organizes the world-model literature for embodied AI.

  2. From World Models to World Action Models: A Concise Tutorial for Robotics

    cs.RO 2026-07 unverdicted novelty 4.0 of 10

    World models are action-conditioned predictors of task-relevant futures; world action models couple those futures to robot actions via four paradigms: imagine-then-execute, feature-conditioned, joint, and auxiliary pr...

Reference graph

Works this paper leans on

99 extracted references · 61 canonical work pages · cited by 2 Pith papers

  1. [1]

    Cosmos world foun- dation model platform for physical ai

    Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foun- dation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025. 1, 2

    arXiv 2025

  2. [2]

    Diffusion for world modeling: Visual details matter in atari

    Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kan- ervisto, Amos Storkey, Tim Pearce, and Franc ¸ois Fleuret. Diffusion for world modeling: Visual details matter in atari. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 1, 2

    2024

  3. [3]

    Fitvid: Overfitting in pixel-level video prediction

    Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. Fitvid: Overfitting in pixel-level video prediction. arXiv preprint arXiv:2106.13195, 2021. 7

    arXiv 2021

  4. [4]

    Zero-shot robotic manipulation with pre-trained image-editing diffusion models

    Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Rich Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pre-trained image-editing diffusion models. In International Conference on Learning Representations (ICLR), 2024. 2

    2024

  5. [5]

    A framework for the robust estimation of optical flow

    Michael J Black and Padmanabhan Anandan. A framework for the robust estimation of optical flow. In International Conference on Computer Vision (ICCV), 1993. 3

    1993

  6. [6]

    Rt-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakr- ishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022. 2, 5, 13

    arXiv 2022

  7. [7]

    Video generation models as world simulators,

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators,

  8. [8]

    Ge- nie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Ge- nie: Generative interactive environments. In International Conference on Machine Learning (ICML), 2024. 2

    2024

Show all 99 references
  1. [9]

    Lucas/kanade meets horn/schunck: Combining local and global optic flow methods

    Andr ´es Bruhn, Joachim Weickert, and Christoph Schn ¨orr. Lucas/kanade meets horn/schunck: Combining local and global optic flow methods. International Journal of Com- puter Vision (IJCV), 2005. 3

    2005

  2. [10]

    Se3-nets: Learning rigid body motion using deep neural networks

    Arunkumar Byravan and Dieter Fox. Se3-nets: Learning rigid body motion using deep neural networks. In Inter- national Conference on Robotics and Automation (ICRA) ,

  3. [11]

    Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation

    Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, et al. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158, 2024. 2

    arXiv

  4. [12]

    Video depth anything: Consistent depth estimation for super-long videos

    Sili Chen, Hengkai Guo, Shengnan Zhu, Feihu Zhang, Zi- long Huang, Jiashi Feng, and Bingyi Kang. Video depth anything: Consistent depth estimation for super-long videos. arXiv preprint arXiv:2501.12375, 2025. 4, 14

    2025 arXiv

  5. [13]

    4d spatio-temporal convnets: Minkowski convolutional neural networks

    Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 5, 13

    2019

  6. [14]

    Auto- mated creation of digital cousins for robust policy learning

    Tianyuan Dai, Josiah Wong, Yunfan Jiang, Chen Wang, Cem Gokmen, Ruohan Zhang, Jiajun Wu, and Li Fei-Fei. Auto- mated creation of digital cousins for robust policy learning. arXiv preprint arXiv:2410.07408, 2024. 1

    2024 arXiv

  7. [15]

    Flownet: Learning optical flow with convolutional networks

    Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Inter- national Conference on Computer Vision (ICCV), 2015. 3

    2015

  8. [16]

    Learning universal policies via text-guided video genera- tion

    Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video genera- tion. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 2

    2024

  9. [17]

    Video language planning

    Yilun Du, Sherry Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B Tenenbaum, Leslie Pack Kaelbling, et al. Video language planning. In International Conference on Learning Repre- sentations (ICLR), 2024. 2

    2024

  10. [18]

    Visual foresight: Model-based deep reinforcement learning for vision-based robotic control

    Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00568, 2018. 1, 2, 6

    2018 arXiv

  11. [19]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In International Conference on Machine Learning...

  12. [20]

    Deep visual foresight for planning robot motion

    Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In International Conference on Robotics and Automation (ICRA), 2017. 2, 6

    2017

  13. [21]

    Unsuper- vised learning for physical interaction through video predic- tion

    Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsuper- vised learning for physical interaction through video predic- tion. In Advances in Neural Information Processing Systems (NeurIPS), 2016. 2

    2016

  14. [22]

    Mo- tion prompting: Controlling video generation with motion trajectories

    Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez-Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, et al. Mo- tion prompting: Controlling video generation with motion trajectories. arXiv preprint arXiv:2412.02700, 2024. 2

    2024 arXiv

  15. [23]

    Maskvit: Masked vi- sual pre-training for video prediction

    Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Mart´ın-Mart´ın, and Li Fei-Fei. Maskvit: Masked vi- sual pre-training for video prediction. In International Con- ference on Learning Representations (ICLR), 2023. 7

    2023

  16. [24]

    Recurrent world models facilitate policy evolution

    David Ha and J ¨urgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Informa- tion Processing Systems (NeurIPS), 2018. 1, 2

    2018

  17. [25]

    Dream to control: Learning behaviors by la- tent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Moham- mad Norouzi. Dream to control: Learning behaviors by la- tent imagination. In International Conference on Learning Representations (ICLR), 2020. 1, 2

    2020

  18. [26]

    Mastering atari with discrete world mod- els

    Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world mod- els. In International Conference on Learning Representa- tions (ICLR), 2021. 9

    2021

  19. [27]

    Mastering diverse domains through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023. 1, 2

    2023 arXiv

  20. [28]

    Td-mpc2: Scalable, robust world models for continuous control

    Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16828, 2023. 2

    2023 arXiv

  21. [29]

    Tem- poral difference learning for model predictive control

    Nmenicklas A Hansen, Hao Su, and Xiaolong Wang. Tem- poral difference learning for model predictive control. In In- ternational Conference on Machine Learning (ICML), 2022. 1, 2

    2022

  22. [30]

    Controllable video generation with sparse trajectories

    Zekun Hao, Xun Huang, and Serge Belongie. Controllable video generation with sparse trajectories. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 2

    2018

  23. [31]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. In Advances in Neural Information Processing Systems (NeurIPS), 2017. 5

    2017

  24. [32]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. In Advances in Neural Informa- tion Processing Systems (NeurIPS), 2020. 1, 2, 3

    2020

  25. [33]

    Determining opti- cal flow

    Berthold KP Horn and Brian G Schunck. Determining opti- cal flow. Artificial intelligence, 1981. 3

    1981

  26. [34]

    Enerverse: Envision- ing embodied future space for robotics manipulation

    Siyuan Huang, Liliang Chen, Pengfei Zhou, Shengcong Chen, Zhengkai Jiang, Yue Hu, Peng Gao, Hongsheng Li, Maoqing Yao, and Guanghui Ren. Enerverse: Envision- ing embodied future space for robotics manipulation. arXiv preprint arXiv:2501.01895, 2025. 2

    2025

  27. [35]

    Flowformer: A transformer architecture for optical flow

    Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: A transformer architecture for optical flow. In European Conference on Computer Vision (ECCV), 2022. 3

    2022

  28. [36]

    Scope of va- lidity of psnr in image/video quality assessment

    Quan Huynh-Thu and Mohammed Ghanbari. Scope of va- lidity of psnr in image/video quality assessment. Electronics letters, 2008. 5

    2008

  29. [37]

    Ditto: Building digital twins of articulated objects from interaction

    Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction. In The IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 5616–5626, 2022. 1

    2022

  30. [38]

    Scalable scene flow from point clouds in the real world

    Philipp Jund, Chris Sweeney, Nichola Abdo, Zhifeng Chen, and Jonathon Shlens. Scalable scene flow from point clouds in the real world. IEEE Robotics and Automation Letters (RA-L), 2021. 3

    2021

  31. [39]

    Model based reinforcement learning for atari

    Łukasz Kaiser, Mohammad Babaeizadeh, Piotr Miłos, Bła˙zej Osi ´nski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model based reinforcement learning for atari. In International Conference on Learning Representations (I...

    2020

  32. [40]

    White, Sam Devlin, Cecily Morrison, and Katja Hofmann

    Anssi Kanervisto, Dave Bignell, Linda Yilin Wen, Mar- tin Grayson, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Tabish Rashid, Tim Pearce, Yuhan Cao, Abdelhak Lemkhenter, Chentian Jiang, Gavin Costello, Gunshi Gupta, Marko Tot, Shu Ishida, Tarun Gupta, Udit Arora,...

    2025

  33. [41]

    Robodesk: A multi-task reinforcement learning benchmark, 2021

    Harini Kannan, Danijar Hafner, Chelsea Finn, and Dumitru Erhan. Robodesk: A multi-task reinforcement learning benchmark, 2021. 2, 7

    2021

  34. [42]

    I can’t believe it’s not scene flow! In European Conference on Computer Vision (ECCV), 2024

    Ishan Khatri, Kyle Vedder, Neehar Peri, Deva Ramanan, and James Hays. I can’t believe it’s not scene flow! In European Conference on Computer Vision (ECCV), 2024. 3

    2024

  35. [43]

    Learning to act from actionless videos through dense correspondences

    Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B Tenenbaum. Learning to act from actionless videos through dense correspondences. In International Conference on Learning Representations (ICLR), 2023. 2, 3, 4

    2023

  36. [44]

    Dense optical tracking: connecting the dots

    Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Dense optical tracking: connecting the dots. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3

    2024

  37. [45]

    Cogact: A foundational vision- language-action model for synergizing cognition and action in robotic manipulation

    Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al. Cogact: A foundational vision- language-action model for synergizing cognition and action in robotic manipulation. arXiv preprint arXiv:2411.19650 ,

    arXiv

  38. [46]

    Evaluating real-world robot manipulation policies in simulation

    Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, et al. Evaluating real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405.05941, 2024. 5, 13

    2024 arXiv

  39. [47]

    Learning to model the world with language

    Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan. Learning to model the world with language. In International Conference on Ma- chine Learning (ICML), 2024. 2

    2024

  40. [48]

    Icp-flow: Lidar scene flow estimation with icp

    Yancong Lin and Holger Caesar. Icp-flow: Lidar scene flow estimation with icp. In The IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), 2024. 3

    2024

  41. [49]

    Flownet3d: Learning scene flow in 3d point clouds

    Xingyu Liu, Charles R Qi, and Leonidas J Guibas. Flownet3d: Learning scene flow in 3d point clouds. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 3

    2019

  42. [50]

    An iterative image reg- istration technique with an application to stereo vision

    Bruce D Lucas and Takeo Kanade. An iterative image reg- istration technique with an application to stereo vision. In International Joint Conference on Artificial Intelligence (IJ- CAI), 1981. 3

    1981

  43. [51]

    Interactive language: Talking to robots in real time

    Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters (RA-L) , 2023. 2, 5

    2023

  44. [52]

    Controllable animation of fluid elements in still images

    Aniruddha Mahapatra and Kuldeep Kulkarni. Controllable animation of fluid elements in still images. InThe IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2

    2022

  45. [53]

    Trans- formers are sample-efficient world models

    Vincent Micheli, Eloi Alonso, and Franc ¸ois Fleuret. Trans- formers are sample-efficient world models. In International Conference on Learning Representations (ICLR), 2023. 2

    2023

  46. [54]

    Efficient world models with context-aware tokenization

    Vincent Micheli, Eloi Alonso, and Franc ¸ois Fleuret. Efficient world models with context-aware tokenization. In Interna- tional Conference on Machine Learning (ICML), 2024. 2 10

    2024

  47. [55]

    Unsupervised learning of object structure and dynamics from videos

    Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin P Murphy, and Honglak Lee. Unsupervised learning of object structure and dynamics from videos. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 7

    2019

  48. [56]

    T3vip: Transformation-based 3d video prediction

    Iman Nematollahi, Erick Rosete-Beas, Seyed Mahdi B Azad, Raghu Rajan, Frank Hutter, and Wolfram Burgard. T3vip: Transformation-based 3d video prediction. In International Conference on Intelligent Robots and Systems (IROS), 2022. 2

    2022

  49. [57]

    Conditional image-to-video genera- tion with latent flow diffusion models

    Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image-to-video genera- tion with latent flow diffusion models. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2

    2023

  50. [58]

    Action-conditional video prediction us- ing deep networks in atari games

    Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction us- ing deep networks in atari games. In Advances in Neural Information Processing Systems (NeurIPS), 2015. 2

    2015

  51. [59]

    Genie 2: A large-scale foundation world model, 2024

    Jack Parker-Holder, Philip Ball, Jake Bruce, Vibhavari Dasagi, Kristian Holsheimer, Christos Kaplanis, Alexandre Moufarek, Guy Scully, Jeremy Shar, Jimmy Shi, Stephen Spencer, Jessica Yung, Michael Dennis, Sultan Kenjeyev, Shangbang Long, Vlad Mnih, Harris Chan, Maxime Gazeau,...

    2024

  52. [60]

    Film: Visual reasoning with a general conditioning layer

    Ethan Perez, Florian Strub, Harm De Vries, Vincent Du- moulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI Conference on Artificial Intelligence (AAAI), 2018. 13

    2018

  53. [61]

    Vi- sion transformers for dense prediction

    Ren ´e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vi- sion transformers for dense prediction. InInternational Con- ference on Computer Vision (ICCV), 2021. 4

    2021

  54. [62]

    Optical flow estimation using a spatial pyramid network

    Anurag Ranjan and Michael J Black. Optical flow estimation using a spatial pyramid network. In The IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) ,

  55. [63]

    Avid: Adapting video diffusion models to world models

    Marc Rigter, Tarun Gupta, Agrin Hilmkil, and Chao Ma. Avid: Adapting video diffusion models to world models. arXiv preprint arXiv:2410.12822, 2024. 1, 2, 4

    2024 arXiv

  56. [64]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1, 3, 4, 13

    2022

  57. [65]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention (MICCAI), 2015. 4

    2015

  58. [66]

    Mastering atari, go, chess and shogi by planning with a learned model

    Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 2020. 2

    2020

  59. [67]

    Videoflow: Exploiting temporal cues for multi-frame optical flow estimation

    Xiaoyu Shi, Zhaoyang Huang, Weikang Bian, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Videoflow: Exploiting temporal cues for multi-frame optical flow estimation. In International Conference on Computer Vision (ICCV), 2023. 3

    2023

  60. [68]

    Flowformer++: Masked cost volume autoencoding for pretraining optical flow estimation

    Xiaoyu Shi, Zhaoyang Huang, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer++: Masked cost volume autoencoding for pretraining optical flow estimation. In The IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2023

  61. [69]

    Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling

    Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. In ACM SIGGRAPH Conference Proceedings ,

  62. [70]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Confer- ence on Machine Learning (ICML), 2015. 1, 3

    2015

  63. [71]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 1, 2, 3, 4

    2010 arXiv

  64. [72]

    Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume

    Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8934–8943,

  65. [73]

    Learning latent dynamic robust representations for world models

    Ruixiang Sun, Hongyu Zang, Xin Li, and Riashat Islam. Learning latent dynamic robust representations for world models. In International Conference on Machine Learning (ICML), 2024. 2

    2024

  66. [74]

    Dyna, an integrated architecture for learn- ing, planning, and reacting

    Richard S Sutton. Dyna, an integrated architecture for learn- ing, planning, and reacting. ACM Sigart Bulletin, 1991. 2

    1991

  67. [75]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision (ECCV), 2020. 3, 14

    2020

  68. [76]

    Raft-3d: Scene flow using rigid- motion embeddings

    Zachary Teed and Jia Deng. Raft-3d: Scene flow using rigid- motion embeddings. In The IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), 2021. 4, 14

    2021

  69. [77]

    A control- centric benchmark for video prediction

    Stephen Tian, Chelsea Finn, and Jiajun Wu. A control- centric benchmark for video prediction. In International Conference on Learning Representations (ICLR) , 2023. 2, 6, 7, 13

    2023

  70. [78]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 5

    2018 arXiv

  71. [79]

    Diffusion models are real-time game engines

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837, 2024. 2

    2024 arXiv

  72. [80]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017. 3

    2017

  73. [81]

    Zeroflow: Scalable scene flow via distillation

    Kyle Vedder, Neehar Peri, Nathaniel Eliot Chodosh, Ishan Khatri, ERIC EATON, Dinesh Jayaraman, Yang Liu, Deva 11 Ramanan, and James Hays. Zeroflow: Scalable scene flow via distillation. In International Conference on Learning Representations (ICLR), 2024. 3

    2024

  74. [82]

    Three-dimensional scene flow

    Sundar Vedula, Simon Baker, Peter Rander, Robert Collins, and Takeo Kanade. Three-dimensional scene flow. In Inter- national Conference on Computer Vision (ICCV) , 1999. 2, 3

    1999

  75. [83]

    High fidelity video prediction with large stochastic recurrent neural networks

    Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc V Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 7

    2019

  76. [84]

    Mcvd-masked conditional video diffusion for prediction, generation, and interpolation

    Vikram V oleti, Alexia Jolicoeur-Martineau, and Chris Pal. Mcvd-masked conditional video diffusion for prediction, generation, and interpolation. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2022. 7

    2022

  77. [85]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. Transactions on Image Processing (TIP), 2004. 5

    2004

  78. [86]

    Attention is all you need

    A Waswani, N Shazeer, N Parmar, J Uszkoreit, L Jones, A Gomez, L Kaiser, and I Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017. 4

    2017

  79. [87]

    ivideogpt: Interactive videogpts are scalable world models

    Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. ivideogpt: Interactive videogpts are scalable world models. In Advances in Neu- ral Information Processing Systems (NeurIPS) , 2024. 1, 2, 7, 8

    2024

  80. [88]

    Daydreamer: World models for physical robot learning

    Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. In Conference on Robot Learning (CoRL), 2022. 2

    2022

  81. [89]

    Spatialtracker: Tracking any 2d pixels in 3d space

    Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3

    2024

  82. [90]

    Flow as the cross-domain manipulation interface

    Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gor- don Wetzstein, Manuela Veloso, and Shuran Song. Flow as the cross-domain manipulation interface. In Conference on Robot Learning (CoRL), 2024. 3, 4

    2024

  83. [91]

    Learning 3d dynamic scene representations for robot manip- ulation

    Zhenjia Xu, Zhanpeng He, Jiajun Wu, and Shuran Song. Learning 3d dynamic scene representations for robot manip- ulation. In Conference on Robot Learning (CoRL), 2020. 2, 4

    2020

  84. [92]

    V olumetric correspon- dence networks for optical flow

    Gengshan Yang and Deva Ramanan. V olumetric correspon- dence networks for optical flow. Advances in Neural Infor- mation Processing Systems (NeurIPS), 32, 2019. 3

    2019

  85. [93]

    Depth any- thing v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2. In Advances in Neural Information Processing Sys- tems (NeurIPS), 2025. 4, 14

    2025

  86. [94]

    Learning interactive real-world simulators

    Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kaelbling, Dale Schuur- mans, and Pieter Abbeel. Learning interactive real-world simulators. In International Conference on Learning Rep- resentations (ICLR), 2024. 1, 2

    2024

  87. [95]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In The IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) ,

  88. [96]

    3d- vla: A 3d vision-language-action generative world model

    Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d- vla: A 3d vision-language-action generative world model. In International Conference on Machine Learning (ICML) ,

  89. [97]

    Robodreamer: Learning com- positional world models for robot imagination

    Siyuan Zhou, Yilun Du, Jiaben Chen, Y ANDONG LI, Dit- Yan Yeung, and Chuang Gan. Robodreamer: Learning com- positional world models for robot imagination. In Interna- tional Conference on Machine Learning (ICML), 2024. 2

    2024

  90. [98]

    Irasim: Learning interactive real- robot action simulators

    Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. Irasim: Learning interactive real- robot action simulators. arXiv preprint arXiv:2406.14540 ,

    arXiv

  91. [99]

    episode” refers to a complete trajectory where the robot completes a task. A “sample

    Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Mart ´ın- Mart´ın, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293, 2020. 2, 6 12 A. Implementation Details Details of ...

    2009 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.