Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

Learning Real-World Action-Video Dynamics with Heterogeneous Masked Autoregression

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single masked-autoregressive model pre-trained on millions of trajectories across dozens of robot embodiments generates action-controllable video at 15x the speed of full-sequence diffusion simulators, enabling policy evaluation and…

desk verdict A useful integration of masked autoregression with heterogeneous action tokens for action-video dynamics, but the real-time controllability claim is not supported by the paper's own DeltaPSNR numbers. read the letter →

arxiv 2502.04296 v1 pith:DXGUZQA3 submitted 2025-02-06 cs.RO cs.CVcs.LG

Lirui Wang , Kevin Zhao , Chaoqi Liu , Xinlei Chen This is my paper · ORCID
classification cs.ROcs.CVcs.LG
keywords maskedautoregressionaction-videodynamicsworldmodelsrobotlearningvideogenerationpolicyevaluationsyntheticdataheterogeneouspre-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that a single masked-autoregressive model, pre-trained on millions of robot interaction trajectories from dozens of different robot bodies, can act as a real-time, action-conditioned video simulator for robotics. It argues that the usual full-sequence diffusion approach to video generation is too slow for interactive use, and that predicting masked chunks of video and action tokens in an autoregressive order keeps visual fidelity and action controllability while running much faster. The payoff, if the claim holds, is a practical way to generate synthetic robot training data and to evaluate policies without deploying them on physical hardware.

What carries the argument

The engine of the method is masked autoregression over a joint sequence of video tokens and action tokens, paired with a heterogeneous action interface. The joint distribution over past and future observations and actions is factored as a product of conditional distributions over masked token sets, so that policy learning, forward dynamics, and full-dynamics simulation all become instances of the same next-token prediction problem. A shared spatial-temporal transformer handles all embodiments, while each domain gets its own action encoder and decoder plus per-block modulation layers to inject action information; action streams are resampled to a common 2Hz rate so that different action frequencies are comparable. Training uses either cross-entropy loss on vector-quantized video tokens with mean-squared-error loss on actions, or a denoising diffusion objective on continuous soft tokens. At inference the model unmasks in few iterations, two in practice, per frame rather than running diffusion over the whole sequence, which is what gives the speed advantage.

What would settle it

Measure whether small perturbations of a single 2Hz action chunk change the generated video: if the PSNR difference stays near zero, the model is not actually following actions, and the central controllability claim collapses.

Watch

Extended reading notes

Core claim

HMA is a framework that learns action-video dynamics jointly across heterogeneous robot data by combining per-embodiment action encoders and decoders with a shared spatial-temporal transformer trained under a masked autoregressive objective. On a pre-training mixture of more than 3 million trajectories spanning roughly 40 embodiments with action dimensions from 2 to 28, HMA generates future frames that are both visually faithful and sensitive to the input action sequence. Compared with a prior full-sequence diffusion interactive simulator, HMA reports higher PSNR and stronger action controllability, measured by DeltaPSNR, while generating frames more than 15 times faster; the discrete-token variant reaches real-time rates around 22 frames per second. After fine-tuning, the model can be used as a video simulator for policy evaluation, with evaluation scores across four policies correlating with ground-truth simulation, and as a synthetic-data generator that lets policies trained from sparse real data match their full-data performance.

Load-bearing premise

The load-bearing assumption is that resampling all action streams to a shared 2Hz rate preserves the causal action-to-video relationship, so that 2Hz action chunks are enough to control generated video accurately across every embodiment.

Editorial extensions

If this is right

  • A single pre-trained HMA checkpoint generates plausible video across 40 embodiment datasets spanning 2-DoF to 28-DoF action spaces.
  • The discrete-token variant reaches about 22 frames per second, and the diffusion-head variant is still 15 times faster than the compared full-sequence diffusion simulator at a similar parameter count.
  • Used as a simulator, HMA ranks four Robomimic policies in the same order as ground-truth physics simulation, with a reported Pearson correlation of 0.95.
  • Adding 90 percent synthetic HMA-generated trajectories to 10 real trajectories brings Robomimic policy success to 100 percent, the same as training on all 100 original trajectories.
  • Autoregressive rollout can be extrapolated to more than 100 frames, roughly ten times the training horizon, with only small degradation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If controllability improves monotonically with more data, the 2Hz action chunking sets an effective bandwidth limit; testing on high-frequency manipulation tasks would reveal whether the shared trunk has to encode action timing rather than just action values.
  • The policy-evaluation result relies on human annotation of success from generated video; an automatic reward or success classifier could turn HMA into a closed-loop evaluation loop for many policies at once, a direction the paper names as future work.
  • The near-zero DeltaPSNR reported after real-world fine-tuning is a warning that controllability metrics may be dominated by static scene content; a direct test would perturb only the action chunks and measure per-frame sensitivity early in the rollout.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Heterogeneous Masked Autoregression (HMA), a generative model for action-video dynamics trained on a heterogeneous mixture of robotic datasets (over 3 million trajectories across 40 embodiments). The architecture uses per-embodiment action encoders/decoders with a shared spatial-temporal transformer trunk, and two generation objectives: discrete VQ tokens with cross-entropy loss plus action MSE, and continuous soft-token diffusion (MAR). At inference it uses masked autoregressive unmasking with M=2 iterations. The authors claim better visual fidelity and controllability than IRASim, 15x faster inference, and demonstrate post-training uses as a video simulator for policy evaluation, synthetic data generation, and imitation policy.

Significance. If the quantitative claims held, HMA would be a useful contribution: it is one of the first attempts to scale action-conditioned video world models across many heterogeneous embodiments and to use masked autoregression for real-time interactive simulation. The paper's strengths include a large pretraining mixture, a scaling study across datasets/trajectories/model sizes, evaluation against an external baseline (IRASim) on real-world data, and multiple downstream applications with a limitations section. The unifying framework covering forward dynamics, passive dynamics, full dynamics, and policy prediction is attractive. However, the current evidence does not consistently support the headline claims: the FID in Table 2 contradicts the 'better visual fidelity' claim, the controllability metric in Table 3 is near zero for the fast discrete variant, and the speed comparison in Table 1 mixes model sizes and baseline settings. These issues are fixable with targeted experiments and careful rewriting, but they are load-bearing.

major comments (4)
  1. [§5.1, Table 2] The text states that HMA achieves better visual fidelity than IRASim, but Table 2 reports FID of 33.56 for HMA versus 23.22 for IRASim. FID is a standard distributional fidelity metric, and a higher value is worse; the table therefore does not support the fidelity claim as written. The authors should either explain why FID is worse despite better PSNR/SSIM/LPIPS/FVD, or restrict the fidelity claim to the metrics that favor HMA. Since the abstract and introduction assert better visual fidelity, this discrepancy must be resolved before the headline claim can stand.
  2. [§4, §5.1, Table 3] The paper defines DeltaPSNR as a controllability metric and states that a small value means the model is less affected by actions and therefore not controllable. In Table 3, the real-world finetuned discrete HMA models yield DeltaPSNR of 0.01 and 0.06, which are effectively zero by the paper's own criterion. Nevertheless, §5.1 says Table 3 shows better visual fidelity and controllability. This is internally inconsistent. The only non-negligible DeltaPSNR (6.06) is reported in Table 2 for the diffusion-based MAR variant, not for the 22.72 FPS discrete variant that underlies the real-time speed claim. The authors must provide direct controllability evidence for the fast discrete variant, or revise the claim that it is a controllable real-time simulator.
  3. [§5.1, Table 1] The '15x faster' claim is not supported by the presented comparison. The text cites the pair 4.44 vs 0.28 FPS, but 0.28 FPS is the non-amortized IRASim baseline; the amortized IRASim runs at 0.58 FPS, giving only 7.7x speedup for HMA-Base MAR. Moreover, HMA-Base MAR has 96M parameters against IRASim-XL's 679M, so the comparison is not matched in capacity. At similar parameter counts, HMA-XL MAR (741M) runs at 2.01 FPS, which is 3.5x faster than amortized IRASim, still fast but not 15x. The speed claim should specify the exact baseline (amortized or not) and report speed at matched or similar parameter counts.
  4. [§3.2] The fixed 2Hz resampling of all action streams is a foundational assumption for cross-embodiment training, but the paper provides no evidence that 2Hz preserves the causal action-to-video relationship for embodiments with faster dynamics. Given the near-zero DeltaPSNR in Table 3, this assumption is load-bearing for the controllability claim. A concrete validation would be to train the same model on a subset of datasets at a higher frequency (or with action interpolation) and measure DeltaPSNR or ground-truth perturbation sensitivity; without it, the generality of the controllable-simulator claim is not established.
minor comments (6)
  1. [Table 5] There is a typo in the caption: 'bewteen' should be 'between'.
  2. [§5.1] The parameter comparison is imprecise: HMA-Base MAR has 96M parameters, which is more than one eighth of IRASim-XL's 679M; either state the exact ratio or remove the 'less than 1/8th' phrasing.
  3. [Figure 6] The axes of the scaling plots are not labeled; please label the x-axes (number of datasets, number of trajectories, hidden dimension) and the y-axes (perplexity and DeltaPSNR).
  4. [§4] The claim that the continuous objective 'can directly operate on pixel space' is confusing because Eq. (3) uses a latent z; clarify whether z is a latent from a VAE or a pixel-space loss.
  5. [Tables 3-4] The notation 'HMA' versus 'HMA+' is used without a definition in the table captions; define 'HMA+' as 'pretrained then finetuned' in the text and captions.
  6. [§4, Metrics] The perplexity metric used in Figure 5 and Tables 3-4 is not precisely defined; state the exact computation (e.g., average per-token validation cross-entropy expressed as perplexity).

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; central claims are empirical and externally benchmarked, with only a minor non-load-bearing self-citation to HPT.

full rationale

No circular step is present in the paper's derivation chain. The load-bearing claim is that masked autoregression over heterogeneous action-video data produces a fast, controllable video simulator. That claim is empirical rather than definitional: Eqs. (2) and (3) are standard cross-entropy/MSE and diffusion losses, and the results are assessed against an external baseline (IRASim, Ref. [60]) and against the MuJoCo ground-truth simulator (Tab. 5). Table 2 reports PSNR/SSIM/LPIPS/FID/FVD and DeltaPSNR for HMA versus IRASim, so the fidelity and speed comparisons do not reduce to the paper's own fitted parameters. The architecture does borrow the heterogeneous action-stem/trunk design from HPT [49], a prior paper with overlapping authors (“the network architecture of HMA follows the heterogeneous pre-training in HPT [49]”), but this self-citation is a design antecedent rather than evidence for the speed/fidelity/controllability conclusions; nothing in the paper invokes a uniqueness theorem or a prior self-cited result to forbid alternatives. The paper itself acknowledges “imperfect controllability,” and the near-zero DeltaPSNR values in Tab. 3 are a legitimate empirical concern about the real-time discrete model's action sensitivity. That concern is an internal-consistency and validity issue, not circularity: the metric is not re-used as the conclusion by construction, and no fitted input is renamed as a prediction. The 2Hz action resampling is an explicit hyperparameter choice, not a hidden circularity. Overall, the derivation is self-contained once the external baselines and ground-truth simulator checks are accepted, so the circularity score is low.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper builds on existing approaches (HPT, MAR, MaskGIT, IRASim) and does not introduce new physical entities. The main assumptions are about tokenization quality, action resampling, and metric validity.

free parameters (5)
  • Action frequency standardization = 2Hz
    All action streams are resampled to 2Hz (Section 3.2) to unify datasets; this discards original action frequencies and could affect controllability.
  • Context length = 12 frames (4 prompt + 8 prediction)
    Chosen to balance generation length and compute (Section 3.2, 3.5); longer horizons are extrapolated at inference.
  • Unmasking iterations M = 2
    Number of unmasking steps at inference for MAR (Section 3.5); authors find M=2 sufficient.
  • Patch size = 2
    Spatial patch size for context length reduction (Section 3.5).
  • Model size variants = 44M, 96M, 679M, 741M
    Models sizes in tables, central to the speed/fidelity comparison.
assumptions (5)
  • domain assumption Pretrained VQ tokenizers (1xGPT/Open-MAGVIT2) and Stable Video Diffusion VAE preserve sufficient visual information for robot dynamics.
    The model operates on compressed tokens; if these drop motion-critical details, the fidelity claim fails. Section 3.3-3.5.
  • standard math Masked autoregressive training with random masking order learns a correct joint distribution over observations and actions.
    Assumed from prior MAR literature [8,27]; the paper relies on this without proof. Section 3.3.
  • ad hoc to paper Action heterogeneity can be reduced to per-embodiment stems/heads plus 2Hz resampling.
    The main modeling choice from HPT [49]; it assumes no loss of controllability from frequency downsampling. Section 3.2.
  • domain assumption Open X-Embodiment and human video datasets are representative enough for generalizable dynamics.
    The 3M trajectories are treated as sufficient for learning general world models. Section 4.
  • domain assumption DeltaPSNR measures controllability.
    The paper defines DeltaPSNR as sensitivity of outputs to actions; sensitivity is not necessarily correctness. Section 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Real-World Action-Video Dynamics with Heterogeneous Masked Autoregression." pith.science (2026). https://pith.science/paper/DXGUZQA3

@misc{pith2026250204296,
  author       = {Pith},
  title        = {Pith review of: Learning Real-World Action-Video Dynamics with Heterogeneous Masked Autoregression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DXGUZQA3}},
  note         = {Machine review of arXiv:2502.04296}
}
read the original abstract

We propose Heterogeneous Masked Autoregression (HMA) for modeling action-video dynamics to generate high-quality data and evaluation in scaling robot learning. Building interactive video world models and policies for robotics is difficult due to the challenge of handling diverse settings while maintaining computational efficiency to run in real time. HMA uses heterogeneous pre-training from observations and action sequences across different robotic embodiments, domains, and tasks. HMA uses masked autoregression to generate quantized or soft tokens for video predictions. \ourshort achieves better visual fidelity and controllability than the previous robotic video generation models with 15 times faster speed in the real world. After post-training, this model can be used as a video simulator from low-level action inputs for evaluating policies and generating synthetic data. See this link https://liruiw.github.io/hma for more information.

Figures

Figures reproduced from arXiv: 2502.04296 by the authors.

Figure 1
Figure 1. Action-Video Dynamics Model from Heterogeneous Robot Interactions. HMA utilizes heterogeneous datasets compris￾ing over 3 million trajectories (videos) from 40 distinct embodi￾ments to pre-train a full dynamics model with next-set-of-token predictions using masked autoregression. After pre-training, the resulting action-video dynamics model is versatile, supporting ap￾plications such as video simulation, policy eval… view at source ↗
Figure 2
Figure 2. Dynamics Model. Masked autoregression in the dynam￾ics model generalizes multiple problem settings including policy learning, forward and passive dynamics, and full dynamics. ffull-dynamics(Ohistory, Ahistory) 7→ (Ofuture, Afuture), passive video predictions fpassive(Ohistory) 7→ Ofuture, forward dy￾namics: fforward-dynamics(Ohistory, Ahistory) 7→ Ofuture, and pol￾icy models: fpolicy(Ohistory, Ahistory) 7→ Afuture a… view at source ↗
Figure 3
Figure 3. Network Architecture. The HMA model architecture maps low-level video and action sequences across different em￾bodiments into a shared latent space. For actions, embodiment projectors are activated based on the training sample. The spatial￾temporal Transformer produces the output video and action tokens for future frames. and the temporal attention is causal in predicting tokens in future steps. This model architect… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Pre-trained Video Model Generation. We show that a single unified HMA model can generate realistic (left 3 columns) and diverse (right 3 columns) videos across multiple embodiment datasets with heterogeneous action spaces. Each group shows three generated frames from a…
Figure 5
Figure 5. Figure 5: (b), we show that the modulation method can out￾perform other methods. In particular, token concatenation along the sequence dimension for each frame does not have enough expressiveness and compute on action-conditioning, compared to per-layer modulation. 4.2. Scaling …
Figure 6
Figure 6. Figure 6: Experiments on Scaling Behaviors of HMA. We observe positive trends in the scaling performance of heterogeneous video models across axes including the number of datasets, number of trajectories, and model sizes. The evaluation metrics on fidelity (perplexity) and contr…
Figure 7
Figure 7. Figure 7: Qualitative Comparisons Between Tokenizers and Models. Despite longer convergence time, diffusion-based methods (Eq. (3)) on soft tokens generate better visual quality than on VQ tokens (Eq. (2)), qualitatively and measured by PSNR [PITH_FULL_IMAGE:figures/full_fig_p0…
Figure 8
Figure 8. Figure 8: Video Controllability. HMA can follow user action inputs to generate physically plausible object permanence (top row) and block pushing interactions (bottom row). These video predictions are both at out-of-distribution settings and at a much longer horizon than trainin…
Figure 9
Figure 9. Figure 9: Policy Evaluation with HMA. By learning the action￾video dynamics over both successful and failed examples, HMA can be used to evaluate policies, similar to a traditional simulator [46]. The autoregressive horizon at inference time is 10 times more than the training ti…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation

    cs.RO 2026-07 unverdicted novelty 6.5 of 10

    Step Forcing trains a few-step autoregressive video world model so RoboWorld closed-loop rollouts plus a task-progress VLM judge recover real-world policy rankings at r=0.989 and ρ=0.970.

  2. BWM: A Low-Cost High-Fidelity World Simulator for Robot Learning

    cs.RO 2026-07 conditional novelty 5.0 of 10

    BWM is an action-conditioned video world model that, on the WorldArena benchmark and a six-task physical-robot test, outperforms compared simulators both as a data generator and as a policy evaluator.

  3. RoboScape: Physics-informed Embodied World Model

    cs.CV 2025-06 conditional novelty 5.0 of 10

    RoboScape jointly learns RGB video, depth, and keypoint-token consistency in one autoregressive world model, improving video quality, geometry, action control, synthetic-data policy training, and policy evaluation for...

Reference graph

Works this paper leans on

60 extracted references · 23 canonical work pages · cited by 3 Pith papers

  1. [1]

    1X World Model Challenge, June 2024

    1X Technologies. 1X World Model Challenge, June 2024. 1, 2, 3, 4

    work page 2024

  2. [2]

    Diffusion for world modeling: Visual details matter in atari

    Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and Franc ¸ois Fleuret. Diffusion for world modeling: Visual details matter in atari. arXiv preprint arXiv:2405.12399, 2024. 1, 2, 3

    arXiv 2024

  3. [3]

    Neuro-dynamic programming: an overview

    Dimitri P Bertsekas and John N Tsitsiklis. Neuro-dynamic programming: an overview. In Proceedings of 1995 34th IEEE conference on decision and control, volume 1, pages 560–564. IEEE, 1995. 2

    work page 1995

  4. [4]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 4

    arXiv 2023

  5. [5]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. 1, 2

    work page 2024

  6. [6]

    Genie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker- Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first Interna- tional Conference on Machine Learning , 2024. 1, 2, 3, 5, 6

    work page 2024

  7. [7]

    Se3-nets: Learning rigid body motion using deep neural networks

    Arunkumar Byravan and Dieter Fox. Se3-nets: Learning rigid body motion using deep neural networks. In 2017 IEEE In- ternational Conference on Robotics and Automation (ICRA), pages 173–180. IEEE, 2017. 2

    work page 2017

  8. [8]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315–11325, 2022. 2, 3, 4, 7

    work page 2022

Show all 60 references
  1. [9]

    Decision transformer: Reinforcement learn- ing via sequence modeling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learn- ing via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021. 2

    2021

  2. [10]

    Genaug: Retargeting behaviors to unseen situations via gener- ative augmentation

    Zoey Chen, Sho Kiami, Abhishek Gupta, and Vikash Kumar. Genaug: Retargeting behaviors to unseen situations via gener- ative augmentation. arXiv preprint arXiv:2302.06671, 2023. 2

    2023 arXiv

  3. [11]

    Diffusion policy: Visuomotor policy learning via action diffu- sion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffu- sion. The International Journal of Robotics Research, page 02783649241273668, 2023. 8, 9

    2023

  4. [12]

    The epic-kitchens dataset: Collection, challenges and base- lines

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide 9 Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. The epic-kitchens dataset: Collection, challenges and base- lines. IEEE Transactions on Pattern Analysi...

    2020

  5. [13]

    Rescal- ing egocentric vision: Collection, pipeline and challenges for epic-kitchens-100

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, An- tonino Furnari, Evangelos Kazakos, Jian Ma, Davide Molti- santi, Jonathan Munro, Toby Perrett, Will Price, et al. Rescal- ing egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Jo...

    2022

  6. [14]

    Scaling cross-embodied learning: One policy for manipulation, navigation, locomotion and aviation

    Ria Doshi, Homer Walke, Oier Mees, Sudeep Dasari, and Sergey Levine. Scaling cross-embodied learning: One policy for manipulation, navigation, locomotion and aviation. arXiv preprint arXiv:2408.11812, 2024. 1

    2024 arXiv

  7. [15]

    Video language planning

    Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B Tenenbaum, et al. Video language planning. arXiv preprint arXiv:2310.10625, 2023. 2

    2023 arXiv

  8. [16]

    Deep visual foresight for planning robot motion

    Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Confer- ence on Robotics and Automation (ICRA), pages 2786–2793. IEEE, 2017. 2

    2017

  9. [17]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings...

    2024

  10. [18]

    World models

    David Ha and J ¨urgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018. 2

    2018 arXiv

  11. [19]

    Td-mpc2: Scalable, robust world models for continuous control

    Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16828, 2023. 2

    2023 arXiv

  12. [20]

    Latent video diffusion models for high-fidelity long video generation

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221 ,

    arXiv

  13. [21]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bern- hard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 6

    2017

  14. [22]

    Video diffusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022. 2, 6

    2022

  15. [23]

    Image quality metrics: Psnr vs

    Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pages 2366–2369. IEEE, 2010. 6

    2010

  16. [24]

    Planning with diffusion for flexible behavior synthe- sis

    Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthe- sis. arXiv preprint arXiv:2205.09991, 2022. 2

    2022 arXiv

  17. [25]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. arXiv preprint arXiv:2304.02643, 2023. 1

    2023 arXiv

  18. [26]

    Videopoet: A large language model for zero-shot video generation.arXiv preprint arXiv:2312.14125, 2023

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos´e Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video generation.arXiv preprint arXiv:2312.14125, 2023. 2

    2023 arXiv

  19. [27]

    Autoregressive image generation without vector quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.11838, 2024. 2, 3, 4, 7

    2024 arXiv

  20. [28]

    Learning particle dynamics for manip- ulating rigid bodies, deformable objects, and fluids

    Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manip- ulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv:1810.01566, 2018. 2

    2018 arXiv

  21. [29]

    Mardini: Masked autoregres- sive diffusion for video generation at scale

    Haozhe Liu, Shikun Liu, Zijian Zhou, Mengmeng Xu, Yan- ping Xie, Xiao Han, Juan C P ´erez, Ding Liu, Kumara Ka- hatapitiya, Menglin Jia, et al. Mardini: Masked autoregres- sive diffusion for video generation at scale. arXiv preprint arXiv:2410.20280, 2024. 2, 3

    arXiv

  22. [30]

    Open-magvit2: An open-source project toward democratizing auto-regressive visual generation.arXiv preprint arXiv:2409.04410, 2024

    Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang, and Ying Shan. Open-magvit2: An open-source project toward democratizing auto-regressive visual generation.arXiv preprint arXiv:2409.04410, 2024. 4

    2024 arXiv

  23. [31]

    Interactive language: Talking to robots in real time

    Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023. 2, 6, 7, 8, 9

    2023

  24. [32]

    What matters in learn- ing from offline human demonstrations for robot manipula- tion

    Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Mart´ın-Mart´ın. What matters in learn- ing from offline human demonstrations for robot manipula- tion. In arXiv preprint arXiv:2108.03298, ...

    2021 arXiv

  25. [33]

    A mathematical introduction to robotic manipulation

    Richard M Murray, Zexiang Li, and S Shankar Sastry. A mathematical introduction to robotic manipulation . CRC press, 2017. 2

    2017

  26. [34]

    Visual reinforcement learn- ing with imagined goals

    Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learn- ing with imagined goals. Advances in neural information processing systems, 31, 2018. 2

    2018

  27. [35]

    Open x-embodiment: Robotic learning datasets and rt-x mod- els

    Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, et al. Open x-embodiment: Robotic learning datasets and rt-x mod- els. arXiv preprint arXiv:2310.08864, 2023. 4

    arXiv

  28. [36]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 1

    2023

  29. [37]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  30. [38]

    Language models are unsuper- vised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsuper- vised multitask learners. OpenAI blog, 1(8):9, 2019. 1

    2019

  31. [39]

    Humanoid locomotion as next token prediction

    Ilija Radosavovic, Bike Zhang, Baifeng Shi, Jathushan Ra- jasegaran, Sarthak Kamat, Trevor Darrell, Koushil Sreenath, and Jitendra Malik. Humanoid locomotion as next token prediction. arXiv preprint arXiv:2402.19469, 2024. 2

    2024 arXiv

  32. [40]

    Avid: Adapting video diffusion models to world models

    Marc Rigter, Tarun Gupta, Agrin Hilmkil, and Chao Ma. Avid: Adapting video diffusion models to world models. arXiv preprint arXiv:2410.12822, 2024. 1, 2

    2024 arXiv

  33. [41]

    Masked world 10 models for visual control

    Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world 10 models for visual control. In Conference on Robot Learning, pages 1332–1344. PMLR, 2023. 2

    2023

  34. [42]

    Outra- geously large neural networks: The sparsely-gated mixture- of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outra- geously large neural networks: The sparsely-gated mixture- of-experts layer. arXiv preprint arXiv:1701.06538, 2017. 4

    2017 arXiv

  35. [43]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502,

    2010 arXiv

  36. [44]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213, 2024. 1

    2024 arXiv

  37. [45]

    Visual autoregressive modeling: Scalable im- age generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Li- wei Wang. Visual autoregressive modeling: Scalable im- age generation via next-scale prediction. arXiv preprint arXiv:2404.02905, 2024. 2

    2024 arXiv

  38. [46]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026–5033. IEEE, 2012. 8

    2012

  39. [47]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 6

    2018 arXiv

  40. [48]

    Diffusion models are real-time game engines, 2024

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines, 2024. 1, 2

    2024

  41. [49]

    Scaling proprioceptive-visual learning with heterogeneous pre-trained transformers

    Lirui Wang, Xinlei Chen, Jialiang Zhao, and Kaiming He. Scaling proprioceptive-visual learning with heterogeneous pre-trained transformers. arXiv preprint arXiv:2409.20537,

    arXiv

  42. [50]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 6

    2004

  43. [51]

    Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer

    Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022. 4

    2022 arXiv

  44. [52]

    Learn- ing interactive real-world simulators

    Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learn- ing interactive real-world simulators. arXiv preprint arXiv:2310.06114, 2023. 1, 2

    2023 arXiv

  45. [53]

    Latent action pretraining from videos

    Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, et al. Latent action pretraining from videos. arXiv preprint arXiv:2410.11758, 2024. 2

    2024 arXiv

  46. [54]

    Meta-world: A benchmark and evaluation for multi-task and meta rein- forcement learning

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta rein- forcement learning. In Conference on robot learning, pages 1094–1100. PMLR, 2020. 4

    2020

  47. [55]

    Scaling robot learn- ing with semantically imagined experience

    Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Jodilyn Peralta, Brian Ichter, et al. Scaling robot learn- ing with semantically imagined experience. arXiv preprint arXiv:2302.11550, 2023. 2

    2023 arXiv

  48. [56]

    Autoregressive dynamics models for offline policy evaluation and optimization

    Michael R Zhang, Tom Le Paine, Ofir Nachum, Cosmin Padu- raru, George Tucker, Ziyu Wang, and Mohammad Norouzi. Autoregressive dynamics models for offline policy evaluation and optimization. arXiv preprint arXiv:2104.13877, 2021. 2

    2021 arXiv

  49. [57]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 6

    2018

  50. [58]

    Learning fine-grained bimanual manipulation with low- cost hardware

    Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low- cost hardware. arXiv preprint arXiv:2304.13705, 2023. 9

    2023 arXiv

  51. [59]

    Nerf in the palm of your hand: Corrective augmentation for robotics via novel-view synthesis

    Allan Zhou, Moo Jin Kim, Lirui Wang, Pete Florence, and Chelsea Finn. Nerf in the palm of your hand: Corrective augmentation for robotics via novel-view synthesis. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17907–17917, 2023. 2

    2023

  52. [60]

    Irasim: Learning interactive real- robot action simulators

    Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. Irasim: Learning interactive real- robot action simulators. arXiv preprint arXiv:2406.14540,

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.