Pith. sign in

REVIEW 3 major objections 5 minor 7 cited by

Pre-Trained Video Generative Models as World Simulators

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A lightweight fine-tuning recipe turns pre-trained video generators into action-following world simulators.

desk verdict Solid engineering contribution with a missing fine-tune-only baseline; the gains are consistent but the specific components' contribution isn't isolated. read the letter →

arxiv 2502.07825 v1 pith:ZZRFFGWJ submitted 2025-02-10 cs.CV cs.AI

Haoran He , Yang Zhang , Liang Lin , Zhongwen Xu , Ling Pan This is my paper · ORCID
classification cs.CVcs.AI
keywords action-conditionedvideogenerationworldmodelssimulatorsdiffusionautoregressivetransformersmotion-reinforcedlossmodel-basedreinforcementlearningprioritizedimagination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video generators trained on internet-scale data produce realistic clips from static prompts, but they do not respond to an agent's actions. DWS claims that fine-tuning such a model with a two-layer action-conditioning module and a motion-reinforced loss is enough to turn it into a world simulator that follows a specified action trajectory frame by frame. The paper tests the recipe on diffusion-based and autoregressive transformer video models across robot-pushing and game domains, reporting better action-conditioned prediction and more sample-efficient model-based reinforcement learning. If the claim holds, world simulation no longer requires training an environment-specific model from scratch: the visual priors already in pre-trained video generators can be redirected to interactive use.

What carries the argument

The load-bearing objects are two training-time additions inside an otherwise unchanged pre-trained video model. The action-conditioned module regresses per-frame scale $\alpha_i$ and shift $\beta_i$ from the action embedding $c_i$ with two linear layers and applies them as $x_i \leftarrow x_i + \mathrm{FFN}(\mathrm{LayerNorm}(x_i)(1+\alpha_i)+\beta_i)$ before each transformer block, so the action at time $i$ directly modulates the frame at time $i$. The motion-reinforced loss reweights the original MSE or cross-entropy loss by $\omega_{i+1}=c\,\mathrm{softmax}(|x_{i+1}-x_i|)$ in pixel space, and by $c\,\mathbf{1}[x_{i+1}\neq x_i]$ in token space, shifting learning toward inter-frame changes. Together they implement the paper's principle that dynamic transitions, not static detail, are what a world simulator must model. In the MBRL setting, prioritized imagination selects initial observations by the magnitude of their TD error, so the world model spends its rollouts on transitions that most improve the policy.

What would settle it

Fine-tune each base video model on the same action-conditioned data using only the original MSE or cross-entropy loss, with no action-conditioned module and no motion-reinforced loss, and evaluate on the same BAIR, Procgen, and Atari splits; if this fine-tuned base reaches the DWS numbers, the central claim collapses. The paper does not report this comparison, and its ablation figure shows training loss curves rather than final evaluation metrics.

Watch

Extended reading notes

Core claim

Given an initial frame and a sequence of actions, DWS fine-tuning makes a pre-trained video generative model predict future frames that track those actions. The paper's central claim is that consistent dynamic transition modeling is what turns a video generator into a world simulator, and that two small changes are enough to induce it: a per-frame action-conditioned module that injects each action's embedding as scale and shift parameters into every transformer block, and a motion-reinforced loss that upweights pixels whose value changes between consecutive frames. With these changes, the paper reports improved action-conditioned video prediction on the BAIR, Procgen, and Atari benchmarks for both Open-Sora, a diffusion model, and iVideoGPT, an autoregressive transformer. It further reports that the resulting simulator, combined with a reward head and prioritized imagination, supports model-based RL that beats strong baselines on tested games and improves offline RL when its generated transitions augment the dataset.

Load-bearing premise

The load-bearing premise is that the action-conditioning module and motion-reinforced loss, not generic fine-tuning on the action-conditioned data, account for the reported improvements, since the paper's video-prediction baselines were not action-conditioned fine-tuned.

Editorial extensions

If this is right

  • If DWS is correct, any released video generative model can be converted into an interactive simulator by fine-tuning a small add-on module, avoiding training a world model from scratch.
  • Action-conditioned video prediction on robotics and game benchmarks should improve on the tested base models by the reported margins under standard quality metrics.
  • Model-based RL agents can use such a simulator for imagined rollouts and, with prioritized imagination, learn from fewer environment interactions than uniform imagination would allow.
  • The same world simulator can also generate synthetic transitions that improve offline RL datasets, effectively doubling the available data in the paper's Coinrun and Ninja experiments.
  • Because the conditioning module attaches to each transformer block, the recipe should transfer to newer video architectures without redesigning the world-model training loop.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the video-prediction comparisons set DWS against base models that were not fine-tuned with the action-conditioned dataset; fine-tuning the base models with only the original supervised loss on that same data would isolate what the new module and loss actually contribute.
  • Beyond the paper: the motion-reinforced loss is a generic reweighting of supervision toward inter-frame change, so it could transfer to other video-prediction tasks where dynamics matter more than static detail, such as traffic or physical-scene forecasting; the paper tests only action-conditioned generation.
  • Beyond the paper: prioritized imagination by TD-error magnitude is a general sampling principle and could be attached to other world models that generate rollouts from replay states.
  • The paper's concluding limitation states that DWS is aimed at bounded temporal horizons and spatial resolutions; that scope boundary should be carried forward when extrapolating the claimed gains to longer or higher-resolution simulation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes DWS, a fine-tuning method that adapts pre-trained video generative models into action-conditioned world simulators. It introduces a lightweight two-linear-layer scale/shift action-conditioning module inserted into transformer blocks, and a motion-reinforced loss that reweights the original video-generation objective toward inter-frame changes. The method is evaluated with Open-Sora (a diffusion model) and iVideoGPT (an autoregressive transformer) on BAIR, Procgen, and Atari action-conditioned video prediction, and the resulting world models are used for model-based RL with a proposed prioritized imagination scheme. The authors report consistent improvements in FVD, PSNR, SSIM, and LPIPS, as well as improved MBRL returns and offline-RL augmentation results.

Significance. If the proposed components are shown to be responsible for the reported gains, DWS would be a practical, low-cost recipe for turning large pre-trained video models into interactive simulators, and the two-architecture evaluation plus downstream RL experiments are valuable. The paper is also transparent about its main limitation (short horizons, low resolutions). However, the experiments currently do not isolate the proposed method from plain fine-tuning, and one of the loss formulas appears to invert the intended motion weighting. Both issues must be resolved before the central claims can be accepted.

major comments (3)
  1. [§5.1, Tables 1–2; Fig. 3(b)] The reported gains are not isolated from plain fine-tuning: every DWS row in Tables 1–2 is a model fine-tuned on the action-conditioned dataset, while the base-model rows are pre-trained checkpoints that were not fine-tuned on those data. For iVideoGPT, which already supports action conditioning, the missing control is iVideoGPT fine-tuned on the same BAIR/Atari/Procgen data without the action-conditioned module or motion-reinforced loss; for Open-Sora, the control is the compressed 12-layer model fine-tuned on the same data with action text conditioning but without the module. The ablation in Fig. 3(b) reports only SFT training-loss curves, not held-out FVD/PSNR/SSIM/LPIPS, so it cannot show that the proposed components are responsible for final-generation quality. Without these controls, the central claim that consistent dynamic transition modeling is the key is not experimentally supported.
  2. [§4.2 / Appendix B.2, Eq. (7)] Equation (7) in Appendix B.2 defines ω_{i+1} = c I(x_{i+1} = x_i) with c = e, which upweights tokens that are unchanged between steps; this is the opposite of the motion-reinforcement intent stated in Eq. (2), where ω is proportional to Softmax(|x_{i+1} − x_i|). Since this weight enters L_motion for all iVideoGPT experiments, the actual objective used in the reported results is ambiguous. Please correct the formula (likely I(x_{i+1} ≠ x_i)) and state explicitly which form was used in Tables 1–2 and Fig. 6.
  3. [§5.3, Table 3] The offline-RL gains are small and the standard deviations overlap across the three seeds: for example, Coinrun CQL goes from 8.58 ± 0.29 to 8.81 ± 0.21, and Ninja IQL from 5.70 ± 0.35 to 6.33 ± 0.17. The text nevertheless says augmentation significantly enhances performance. Report more seeds and/or significance testing, or soften the claim to a modest improvement.
minor comments (5)
  1. [§5.1 / Appendix B.1] The table captions should state explicitly that the 'Open-Sora' rows are the authors' 12-layer compressed variant with a T5-small text encoder, not the released 1.1B Open-Sora model; otherwise comparisons against published Open-Sora results are ambiguous.
  2. [Fig. 5 caption] The caption reads 'Quantitative comparison' but the figure shows qualitative examples; it should be 'Qualitative comparison.'
  3. [§4.1] The term 'architecture-agnostic' is stronger than what is tested: both backbones are transformer-based. Please either temper the claim to 'transformer-based diffusion and autoregressive architectures' or evaluate the module on a non-transformer backbone.
  4. [Figs. 6–7] The MBRL curves are described as averages over five seeds but no variance is shown; add error bars or shaded regions, especially because the text claims a 7× improvement in Breakout.
  5. [§4.2, Eq. (2)] The weight construction ω_{i+1} = c Softmax(|x_{i+1} − x_i|) is underspecified: it should state whether Softmax is taken over spatial/token dimensions and how the values are mapped to the stated range [1, c].

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DWS is an empirical fine-tuning method evaluated on held-out data, though a missing plain-fine-tuning control and an internal inconsistency in Eq. (7) are correctness concerns, not circularity.

full rationale

None of the load-bearing claims reduces to its input by construction. DWS is an empirical fine-tuning method: the action-conditioned module (Eq. 1) is a trainable scale/shift on frame embeddings, and the motion-reinforced loss (Eqs. 2-3, 5, 8) reweights the existing supervised objective using pixel differences computed from ground-truth frames. The reported target quantities—FVD, PSNR, SSIM, and LPIPS on BAIR/Atari/Procgen test videos, and returns from environment rollouts in Fig. 6 and Table 3—are measured on held-out data, not recovered from training weights. The base models (Open-Sora, iVideoGPT) are external checkpoints, and the MBRL comparisons use external algorithms (PPO, PPG, DreamerV3, CQL, IQL); no load-bearing result is justified solely by a self-citation. Two reviewer concerns are real but are correctness/control issues rather than circularity: (1) Tables 1-2 compare DWS-fine-tuned models to base checkpoints that were not fine-tuned on the action-conditioned dataset, so the gains are not isolated from plain fine-tuning, and Fig. 3(b) shows only training loss, not held-out metrics; (2) Appendix Eq. (7) defines omega_{i+1}=c I(x_{i+1}=x_i), which upweights unchanged tokens, contradicting the stated intent of Eq. (2) and weakening the motion-reinforced-loss rationale. Neither issue makes the derivation self-referential, so the circularity score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The framework introduces no new physical or mathematical objects; it introduces the action-conditioned module and motion-weighted loss as engineering components. The main free parameter is the motion-reinforcement strength c. Axioms are domain assumptions about transferable video priors and the adequacy of pixel-motion weighting.

free parameters (1)
  • motion-reinforced strength c = c = e for transformer-based models; unspecified for diffusion models
    Chosen by hand in Eq. (2)/(7); modulates how strongly moving pixels/tokens are weighted in the loss. The paper does not justify the choice or ablate c.
assumptions (3)
  • domain assumption Pre-trained video models contain transferable dynamic priors for action-conditioned simulation
    The entire fine-tuning approach assumes that priors from internet video data help action-conditioned world models (Sec. 3.2, Sec. 4).
  • ad hoc to paper Pixel-level inter-frame difference (or token-change) indicates action-relevant dynamics
    Motion-reinforced loss (Sec. 4.2, Eq. 2/7) weights training by |x_{i+1}-x_i| or I(x_{i+1} != x_i); no evidence that these correlate with action controllability.
  • ad hoc to paper Two linear layers with scale/shift conditioning provide sufficient frame-level action control
    Sec. 4.1 Eq. (1): the action-conditioned module assumes FiLM-style modulation is enough for precise action-frame alignment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pre-Trained Video Generative Models as World Simulators." pith.science (2026). https://pith.science/paper/ZZRFFGWJ

@misc{pith2026250207825,
  author       = {Pith},
  title        = {Pith review of: Pre-Trained Video Generative Models as World Simulators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZZRFFGWJ}},
  note         = {Machine review of arXiv:2502.07825}
}
read the original abstract

Video generative models pre-trained on large-scale internet datasets have achieved remarkable success, excelling at producing realistic synthetic videos. However, they often generate clips based on static prompts (e.g., text or images), limiting their ability to model interactive and dynamic scenarios. In this paper, we propose Dynamic World Simulation (DWS), a novel approach to transform pre-trained video generative models into controllable world simulators capable of executing specified action trajectories. To achieve precise alignment between conditioned actions and generated visual changes, we introduce a lightweight, universal action-conditioned module that seamlessly integrates into any existing model. Instead of focusing on complex visual details, we demonstrate that consistent dynamic transition modeling is the key to building powerful world simulators. Building upon this insight, we further introduce a motion-reinforced loss that enhances action controllability by compelling the model to capture dynamic changes more effectively. Experiments demonstrate that DWS can be versatilely applied to both diffusion and autoregressive transformer models, achieving significant improvements in generating action-controllable, dynamically consistent videos across games and robotics domains. Moreover, to facilitate the applications of the learned world simulator in downstream tasks such as model-based reinforcement learning, we propose prioritized imagination to improve sample efficiency, demonstrating competitive performance compared with state-of-the-art methods.

Figures

Figures reproduced from arXiv: 2502.07825 by the authors.

Figure 1
Figure 1. Our fine-tuned video generative model can serve as an effective world simulator for in￾teracting with different policies and generating diverse trajectories. Different policies can lead to different terminal states, which are highlighted with yellow circles. To address the aforementioned challenges, we propose a novel framework, Dynamic World Simulation (DWS), which is a unified, architecture-agnostic approach for e… view at source ↗
Figure 2
Figure 2. Illustration of the action-conditioned module, [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Motion-reinforced loss enhances the action-controllability of video generative models, [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The qualitative results of generated videos for different domains, including games and [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Quantitative comparison between pre-trained [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Averaged Return across five random seeds on Atari and Procgen environments. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Prioritized imagination improves the perfor [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL

    cs.AI 2026-05 conditional novelty 7.0 of 10

    Masked diffusion language models, not larger autoregressive LLMs, are the better building block for text-based world models in agentic RL, improving rollout fidelity, diversity, and downstream task success.

  2. End-to-End Training for Autoregressive Video Diffusion via Self-Resampling

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Resampling Forcing trains autoregressive video diffusion models on self-resampled degraded histories with a causal mask, achieving stable long-horizon generation without a teacher or discriminator.

  3. Can Your Model Separate Yolks with a Water Bottle? Benchmarking Physical Commonsense Understanding in Video Generation Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    PhysVidBench evaluates text-to-video models with 383 PIQA-derived prompts and a caption-based QA pipeline, finding all tested models score below 40% on physical commonsense, with spatial and temporal reasoning the weakest.

  4. Long-Context State-Space Video World Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A hybrid state-space and local-attention architecture gives autoregressive video diffusion models long-term spatial memory with constant per-frame inference cost, demonstrated on Maze and Minecraft.

  5. UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Driving

    cs.CV 2026-01 conditional novelty 5.0 of 10

    A unified VLM for autonomous driving that couples trajectory planning with future-frame image generation improves open- and closed-loop planning metrics on Bench2Drive and nuScenes.

  6. RoboScape: Physics-informed Embodied World Model

    cs.CV 2025-06 conditional novelty 5.0 of 10

    RoboScape jointly learns RGB video, depth, and keypoint-token consistency in one autoregressive world model, improving video quality, geometry, action control, synthetic-data policy training, and policy evaluation for...

  7. Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Sparse-vDiT replaces dense attention with fixed per-head sparse patterns chosen offline, achieving 1.58-1.85x end-to-end speedups on CogVideoX1.5, HunyuanVideo, and Wan2.1 with minimal quality loss.

Reference graph

Works this paper leans on

32 extracted references · 8 canonical work pages · cited by 7 Pith papers

  1. [3]

    Leveraging procedural generation to benchmark reinforcement learning

    Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pages 2048–2056. PMLR,

    work page

  2. [4]

    Following Dreamerv3 (Hafner et al., 2023), we use a symlog transformation for reward prediction. 19 Algorithm 1 Model-Based Reinforcement Learning (Janner et al., 2019a) with Prioritized Imagination 1: Initialize real replay buffer Breal with random policy 2: Initialize actor-critic πϕ, vϕ, world model pθ 3: Initially train model pθ on Breal 4: Initialize...

    work page 2023

  3. [5]

    Zihan Ding, Amy Zhang, Yuandong Tian, and Qinqing Zheng

    URL https://oasis-model.github.io/. Zihan Ding, Amy Zhang, Yuandong Tian, and Qinqing Zheng. Diffusion world model.arXiv preprint arXiv:2402.03570,

    arXiv

  4. [7]

    Maskvit: Masked visual pre-training for video prediction

    Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martín-Martín, and Li Fei-Fei. Maskvit: Masked visual pre-training for video prediction. arXiv preprint arXiv:2206.11894,

    arXiv

  5. [10]

    Open-sora plan: Open-source large video generation model

    Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model. arXiv preprint arXiv:2412.00131,

    arXiv

  6. [11]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003,

    arXiv

  7. [12]

    Latte: Latent diffusion transformer for video generation

    Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048,

    arXiv

  8. [13]

    Vincent Micheli, Eloi Alonso, and François Fleuret

    URL https://openreview.net/forum?id=3w6xuXDOdY. Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world mod- els. In The Eleventh International Conference on Learning Representations , 2023a. URL https: //openreview.net/forum?id=vhFu1Acb0xb. Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-eff...

    arXiv

Show all 32 references
  1. [15]

    Avid: Adapting video diffusion models to world models

    Marc Rigter, Tarun Gupta, Agrin Hilmkil, and Chao Ma. Avid: Adapting video diffusion models to world models. arXiv preprint arXiv:2410.12822,

    arXiv

  2. [16]

    15 John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

    URL https://openreview.net/forum?id=TdBaDGCpjly. 15 John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  3. [17]

    Karen Simonyan and Andrew Zisserman

    URL https://deepmind.google/technologies/veo/. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556,

    arXiv

  4. [19]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al

    URL https://openreview.net/forum?id= gojL67CfS8. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv p...

    arXiv

  5. [20]

    Towards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717,

    arXiv

  6. [22]

    Pandora: Towards general world model with natural language actions and video states

    Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, et al. Pandora: Towards general world model with natural language actions and video states. arXiv preprint arXiv:2406.09455,

    arXiv

  7. [23]

    Videogpt: Video generation using vq-vae and transformers

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157,

    arXiv

  8. [25]

    Playable game generation

    Mingyu Yang, Junyou Li, Zhongbin Fang, Sheng Chen, Yangbin Yu, Qiang Fu, Wei Yang, and Deheng Ye. Playable game generation. arXiv preprint arXiv:2412.00887, 2024a. Sherry Yang, Jacob C Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, Andre Barreto, Pieter Abbeel, and Dale Sch...

    arXiv

  9. [26]

    Yang Zhang, Chenjia Bai, Bin Zhao, Junchi Yan, Xiu Li, and Xuelong Li

    URL https://openreview.net/forum?id=WxnrX42rnS. Yang Zhang, Chenjia Bai, Bin Zhao, Junchi Yan, Xiu Li, and Xuelong Li. Decentralized transform- ers with centralized aggregation are sample-efficient multi-agent world models. arXiv preprint arXiv:2406.15836,

    arXiv

  10. [27]

    Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong

    URL https://github.com/hpcaitech/Open-Sora. Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. Irasim: Learning interactive real-robot action simulators. arXiv:2406.12802,

    arXiv

  11. [28]

    17 A Details of Dataset Collection A.1 Atari In our experiments on two Atari games - Breakout and Battle Zone, we utilized a well-established baseline for Atari, Deep Q-Learning (DQN) agent (Mnih et al., 2013), to collect the offline dataset for tuning. Specifically, we adopte...

    2013

  12. [30]

    and spatial-temporal trans- former (Chen et al., 2023, Ma et al.,

    2023

  13. [31]

    For our implementation of DWS, we utilized Open-Sora version 1.2 as our base model

    architecture , is a text-and-frame-conditioned video generation model. For our implementation of DWS, we utilized Open-Sora version 1.2 as our base model. To maintain consistency with the original architecture, we employed the same Variational Autoencoder (VAE) provided in the...

    2023

  14. [84]

    In terms of the hyperparameter of DQN, we followed the default setting provided at https://github.com/ vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py

    for DQN agent training. In terms of the hyperparameter of DQN, we followed the default setting provided at https://github.com/ vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py. A.2 Procgen Following a similar approach to our Atari collection experiments, we collected replay da...

    2022

  15. [1998]

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al

    URL https://api.semanticscholar.org/CorpusID: 264703640. Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690,

    arXiv

  16. [2013]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127,

    arXiv

  17. [2017]

    The matrix: Infinite-horizon world generation with real-time moving control

    Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, and Hongyang Zhang. The matrix: Infinite-horizon world generation with real-time moving control. arXiv preprint arXiv:2412.03568,

    arXiv

  18. [2018]

    Diffusion models are real-time game engines

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837,

    arXiv

  19. [2019]

    Dream to control: Learning behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020a. URL https://openreview.net/forum?id=S1lOTC4tDS. Danijar Hafner, Timothy Lillicrap, Moh...

    2010 arXiv

  20. [2020]

    Phasic policy gradient

    Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. InInternational Conference on Machine Learning, pages 2020–2027. PMLR,

    2020

  21. [2021]

    Learning interactive real-world simulators

    Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114,

    arXiv

  22. [2022]

    https://iclr-blog-track.github.io/2022/03/25/ppo- implementation-details/

    URL https://iclr-blog-track.github.io/2022/03/ 25/ppo-implementation-details/. https://iclr-blog-track.github.io/2022/03/25/ppo- implementation-details/. Quan Huynh-Thu and Mohammed Ghanbari. Scope of validity of psnr in image/video quality assessment. Electronics Letters, 44:800–801,

    2022

  23. [2023]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  24. [2024]

    Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan

    URL https://arxiv.org/abs/2405.12399. Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. Fitvid: Overfitting in pixel-level video prediction. arXiv preprint arXiv:2106.13195,

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.