Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Playing with Transformer at 30+ FPS via Next-Frame Diffusion

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Next-Frame Diffusion claims the first autoregressive video generation above 30 FPS on a single A100 GPU, using block-wise causal attention, video-domain consistency distillation, and speculative sampling that exploits repeated action…

desk verdict Solid systems paper that hits 30+ FPS on Minecraft by combining block-wise causal attention, video consistency distillation, and action-repetition speculative sampling, but the headline number depends on an unquantified action prior and is really an offline throughput, not a demonstrated interactive rate. read the letter →

arxiv 2506.01380 v2 pith:767WIGSQ submitted 2025-06-02 cs.CV cs.AI

Xinle Cheng , Tianyu He , Jiayi Xu , Junliang Guo , Di He , Jiang Bian This is my paper
classification cs.CVcs.AI
keywords autoregressivevideogenerationnext-framediffusionblock-wisecausalattentionconsistencydistillationspeculativesamplingflowmatchingaction-conditionedworldmodelsMinecraftVPT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that an autoregressive diffusion transformer can generate video faster than real time by generating each frame's tokens in parallel, distilling diffusion sampling down to four steps, and speculatively predicting several future frames at once when the action input repeats. If true, interactive and streaming video generation, where a user sends actions and the model renders the next frames, could run at over 30 frames per second on a single A100 GPU with a modest 310M-parameter model, without giving up visual quality relative to much larger autoregressive baselines. On the VPT Minecraft benchmark, the accelerated NFD+ variant reaches 31.14 FPS with an FVD of 227, while the non-accelerated 310M model reaches an FVD of 212, beating discrete-token autoregressive baselines on both speed and quality at comparable size.

What carries the argument

The load-bearing object is the block-wise causal attention DiT, which lets the model generate one full frame's tokens in parallel while conditioning only on past frames. The speed story rides on two mechanisms: sCM-style consistency distillation, with a TrigFlow formulation, per-frame independent timesteps, 3D tangent normalization, and adversarial supervision, which reduces sampling to four steps; and speculative sampling, which replicates the current action N times so the model can propose several future frames in a single forward pass and then keeps only the prefix verified against the true action sequence.

What would settle it

A reader could compute the distribution of consecutive identical actions in the VPT test set, or shuffle the action sequence to break all consecutive repeats and rerun NFD+; if the FPS advantage over non-speculative generation disappears, the speculative-sampling speedup claim is falsified.

Watch

Extended reading notes

Core claim

Next-Frame Diffusion (NFD) is an autoregressive video model that operates in continuous latent space: within a frame, all tokens are generated in parallel through bidirectional self-attention, while across frames attention is strictly causal. The paper's central claim is that this architecture, combined with two accelerations, achieves the first autoregressive video generation above 30 FPS on an A100 GPU. Consistency distillation adapted to video cuts sampling to four function evaluations, and speculative sampling generates N future frames in one forward pass using repeated action inputs, discarding any frames whose hypothesized action does not match the true action. On the VPT dataset, NFD+ (310M) runs at 31.14 FPS with FVD 227, PSNR 16.83, LPIPS 0.35, and SSIM 0.43, which the authors say beats autoregressive baselines in both visual quality and sampling efficiency.

Load-bearing premise

Speculative sampling is profitable only if action sequences contain long runs of identical actions; the paper motivates this as an empirical observation but reports no action-change statistics, so if actions in the target environment change every frame the speculated frames are always discarded and the 30+ FPS result collapses to one-frame generation.

Editorial extensions

If this is right

  • Autoregressive video models can be interactive at real-time rates: NFD+ at 310M runs at 31.14 FPS on one A100 GPU, and the 130M variant reaches 42.46 FPS.
  • Four-step distilled sampling plus speculative sampling is sufficient to beat the discrete-token MineWorld baseline on quality as well as speed: NFD+ 310M reports FVD 227 and PSNR 16.83 versus MineWorld 1.2B's FVD 227 and PSNR 15.69, at roughly ten times the frame rate.
  • Speculative sampling alone gives 1.14 to 1.26 times speedup across model sizes, with N=2 optimal for the 130M model, so the technique helps most when decoding is memory-bound.
  • KV caching of noisy context frames yields a further 1.33 times speedup at the largest model size, indicating that the four-step sampling gains compose with standard inference optimizations.
  • Scaling the same training recipe from 130M to 774M parameters improves FVD from 220 to 184, so the approach benefits from scale rather than depending on a single lucky configuration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the speculative-sampling speedup is a property of the action statistics of the environment, not of the model alone; reporting the run-length distribution of repeated actions in VPT would let readers predict how the 1.19 to 1.26 times gains transfer to other interactive domains.
  • My inference: the same block-wise causal attention plus repeated-condition speculation could apply to any sequential generative task with a low-entropy control signal, such as audio effect control, robotic teleoperation, or driving command streams, where consecutive commands often repeat.
  • My inference: because error accumulation is fought by noise injection during sampling, long-horizon consistency may be improvable by tuning that noise schedule per task, a cheap experiment the paper does not run.
  • My inference: the reported 30+ FPS is a lower bound only under the benchmark's action repetition; since the paper deliberately excludes quantization and sparse inference, the same recipe could plausibly push the 774M model past 30 FPS as well.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Next-Frame Diffusion (NFD), an autoregressive diffusion transformer for action-conditioned video generation, with three design components: block-wise causal attention for parallel next-frame token generation, consistency distillation (sCM) adapted to video for few-step sampling, and a speculative sampling scheme that generates multiple future frames under a repeated-action assumption and discards them when the action actually changes. The method is trained and evaluated on the VPT Minecraft dataset. The paper reports that the distilled NFD+ variant reaches 31.14 FPS on an A100 GPU with a 310M-parameter model, and claims this is the first autoregressive video model to exceed 30 FPS, while matching or improving visual quality relative to the MineWorld and Oasis baselines.

Significance. If the reported results hold, the paper makes a useful practical contribution to real-time interactive video generation: it demonstrates that a continuous-latent autoregressive diffusion model with block-wise causal attention can be accelerated with consistency distillation and speculative sampling, and it provides consistent quality/throughput comparisons against two recent baselines. The ablations are fairly extensive and include model-size scaling, conditioning mechanisms, the sCM noise distribution, and the speculative sampling parallelism. The pseudo-code in Appendix A is a helpful reproducibility aid. The main burden falls on the headline throughput claim, which is carried almost entirely by the speculative sampling component: the reader cannot assess whether the 30+ FPS number generalizes without knowing the action-repetition statistics of the evaluation distribution and without an interactive (non-oracle) measurement. Additionally, the distilled NFD+ model is worse than its own teacher on FVD, and all quantitative claims are single-run measurements. The contributions are therefore promising but require stronger empirical support for the central claim.

major comments (3)
  1. [Section 4.2 and Table 2] The 31.14 FPS headline for the 310M model is load-bearing on speculative sampling: without it, the same model runs at 26.15 FPS (Table 2), so the accelerator is only a 1.19x speedup. The paper justifies this speedup as based on an 'empirical observation' that adjacent frames often share the same action, but it reports no distribution of action-change frequencies, no acceptance/rejection rates, and no mean accepted speculative length on the VPT test set. Since the expected speedup is a direct function of that distribution (if actions changed every frame, all speculative frames would be discarded and throughput would fall below the one-frame baseline), the central quantitative claim is not verifiable from the reported data. Please add the action-run length statistics and acceptance rates, and report throughput with a worst-case or shuffled action stream as a lower bound.
  2. [Section 4.2 and Appendix A] The speculative sampling procedure as described and implemented uses ground-truth future actions to decide which speculative frames to keep: the algorithm checks `same_action(act[:, i:i+nframe])` and discards frames after the first action change. This is an offline oracle evaluation. In interactive or streaming deployment, future user or agent actions are not known at the moment of speculative generation, so the reported 31.14 FPS is an offline throughput number measured on recorded VPT actions, not a demonstrated interactive generation rate. The paper should either measure throughput in a live loop where actions arrive one at a time, or explicitly characterize the reported FPS as an upper bound under the VPT action autocorrelation assumption.
  3. [Section 5.1 and Table 1] The claim that NFD 'beats autoregressive baselines in terms of both visual quality and sampling efficiency' is only partially supported for the accelerated NFD+ model. In Table 1, NFD+ 310M has FVD 227, which is worse than its own teacher NFD 310M (FVD 212) and equal to MineWorld 1.2B (FVD 227), even though PSNR and LPIPS improve. The paper should discuss this FVD regression explicitly and justify the overall 'competitive visual quality' conclusion with a metric-level or user-study assessment, rather than relying on the aggregate claim.
minor comments (6)
  1. [Section 5.1 / Implementation Details] There is a typo 'recstruction' in the Implementation Details paragraph; it should be 'reconstruction'.
  2. [Abstract] The sentence 'We, for the first time, achieves autoregressive video generation...' has a subject-verb agreement error; consider 'We, for the first time, achieve...'.
  3. [Section 4.2 / Table 2] The text says 'we adopt N = 2 as the default configuration for all models,' but Table 2 shows that for the 310M model N = 4 yields the highest FPS (31.66 vs. 31.14 for N = 2). Please clarify the selection criterion or justify the default for each model size.
  4. [Section 4.2] The phrase 'compare the predicted actions with the actual subsequent action inputs' is slightly misleading because the model does not predict actions; it simply receives repeated action inputs. Consider saying 'the assumed repeated actions' rather than 'predicted actions'.
  5. [Section 4.3] The Gaussian noise injection magnitude is described only as 'a small amount'; the actual value does not appear in the hyperparameter tables (Tables 5 and 6). Please report the precise noise scale for reproducibility.
  6. [Section 5] The FPS measurement protocol is underspecified: the paper reports batch size 1 on an A100, but not the number of generated frames, warmup runs, or whether the measurement includes tokenization and the scheduler. This information would help reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: NFD's headline FPS and quality numbers are measured against external baselines and do not reduce to fitted inputs or self-citations.

full rationale

Walking the paper's derivation chain, every load-bearing result is either a measured quantity or an externally benchmarked comparison, not a quantity defined in terms of the target conclusion. The 31.14 FPS figure in Table 1 is a measured throughput on an A100, and Table 2 shows the underlying 26.15 FPS without speculative sampling plus the 1.19x speedup from the proposed technique; neither number is a fitted parameter renamed as a prediction. The visual quality metrics (FVD, PSNR, LPIPS, SSIM) are computed against the VPT benchmark and compared with external or published baselines (Oasis, MineWorld). The self-citations that appear—MineWorld [22] for the action-conditioned benchmark and evaluation protocol, VidTok [51] as the tokenizer—are infrastructure or baselines, not the source of the claimed superiority; NFD is compared against MineWorld and beats it, so the comparison is not closed on itself. The speculative-sampling speedup does rest on the empirical observation of action repetition (Sec. 4.2), and the paper does not report action-change statistics or acceptance rates, which is a legitimate support gap for robustness and external validity. However, this is an empirical assumption about the data, not a construction identity: the FPS result is not derived from the assumption in the sense of being equivalent to it by definition or by fitted parameters. The Limitations section (Sec. 6) candidly notes the narrow domain, fixed resolution, and limited context window, further confirming that the paper does not present the target claim as forced by prior author results. No step in the derivation chain qualifies as self-definitional, fitted-input-called-prediction, load-bearing self-citation, imported uniqueness, ansatz-by-citation, or renaming of a known result. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard diffusion and flow-matching math, the sCM consistency distillation recipe, the choice of timestep noise distribution, the speculative sampling action-repetition assumption, and unspecified adversarial loss weighting. No new physical entities are introduced. The main non-standard premise is the action-repetition prior, which is not quantified.

free parameters (5)
  • Pmean = 0.0
    Mean of the log-normal proposal for tan(t_i) in the sCM timestep distribution; chosen empirically after the ablation in Table 4.
  • Pstd = 1.6
    Standard deviation of the log-normal proposal for tan(t_i); chosen empirically after the ablation in Table 4.
  • Speculative sampling parallelism N = 2
    Default number of speculative frames per forward pass; selected from Table 2 even though N=3 and N=4 are marginally faster for the 310M model.
  • Context noise injection magnitude = not reported
    Section 4.3 says a small amount of Gaussian noise is added to context frames, but no value is given; this affects the error-accumulation tradeoff.
  • Adversarial loss weight lambda = not reported
    Equation 8 combines LsCM with lambda times Ladv, but lambda is never specified in the paper.
assumptions (5)
  • standard math Flow matching interpolation x_t = (1-t)x0 + t epsilon with velocity v = epsilon - x0 yields a valid generative ODE for video frames.
    Used in Eqs. 1-2 and Sec. 3.2; standard result from Lipman et al. 2023.
  • domain assumption The sCM consistency distillation objective (Eqs. 4-6) remains valid when each frame receives an independently sampled timestep and the teacher tangent is estimated from a frozen pretrained model.
    Sec. 4.1 adapts sCM to video; no proof is given that per-frame independent noise schedules preserve the consistency property for video data.
  • domain assumption Action sequences in VPT contain sufficiently long runs of repeated actions to make speculative sampling profitable.
    Sec. 4.2 relies on this empirical observation but provides no measured statistics; it is the load-bearing premise for the 30+ FPS claim.
  • domain assumption The adversarial loss with a frozen pretrained discriminator stabilizes distillation and improves fidelity.
    Sec. 4.1 and Eqs. 7-8 use this supervision; the discriminator architecture and loss weight are not specified.
  • domain assumption The pretrained image tokenizer provides latents of sufficient fidelity for frame-level autoregressive generation.
    Secs. 3.1 and 5 rely on a 2D VAE tokenizer with a fine-tuned decoder; all quality comparisons inherit tokenizer quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Playing with Transformer at 30+ FPS via Next-Frame Diffusion." pith.science (2026). https://pith.science/paper/767WIGSQ

@misc{pith2026250601380,
  author       = {Pith},
  title        = {Pith review of: Playing with Transformer at 30+ FPS via Next-Frame Diffusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/767WIGSQ}},
  note         = {Machine review of arXiv:2506.01380}
}
read the original abstract

Autoregressive video models offer distinct advantages over bidirectional diffusion models in creating interactive video content and supporting streaming applications with arbitrary duration. In this work, we present Next-Frame Diffusion (NFD), an autoregressive diffusion transformer that incorporates block-wise causal attention, enabling iterative sampling and efficient inference via parallel token generation within each frame. Nonetheless, achieving real-time video generation remains a significant challenge for such models, primarily due to the high computational cost associated with diffusion sampling and the hardware inefficiencies inherent to autoregressive generation. To address this, we introduce two innovations: (1) We extend consistency distillation to the video domain and adapt it specifically for video models, enabling efficient inference with few sampling steps; (2) To fully leverage parallel computation, motivated by the observation that adjacent frames often share the identical action input, we propose speculative sampling. In this approach, the model generates next few frames using current action input, and discard speculatively generated frames if the input action differs. Experiments on a large-scale action-conditioned video generation benchmark demonstrate that NFD beats autoregressive baselines in terms of both visual quality and sampling efficiency. We, for the first time, achieves autoregressive video generation at over 30 Frames Per Second (FPS) on an A100 GPU using a 310M model.

Figures

Figures reproduced from arXiv: 2506.01380 by the authors.

Figure 1
Figure 1. We present Next-Frame Diffusion (NFD), an autoregressive diffusion transformer that [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Qualitative results of the generated videos. Each row depicts a sequence of frames generated [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Frames generated by NFD+ and MineWorld respectively, which illustrates the superior [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Frames generated by NFD+ and MineWorld respectively, which illustrates a door-opening [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: In this case, both models have previously encountered the brown block. NFD+ successfully [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RoDyn: Taming Interactive Robot-Dynamic 2.5D World Model for Robotic Manipulation

    cs.RO 2025-10 unverdicted novelty 5.0 of 10

    Abstract describes RoDyn but full text describes iMoWM; the record is internally inconsistent and the headline claims are absent from the body.

Reference graph

Works this paper leans on

68 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Cosmos world foundation model platform for physical ai

    Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025

    arXiv 2025

  2. [2]

    Diffusion for world modeling: Visual details matter in atari

    Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos J Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37:58757–58791, 2024

    2024

  3. [3]

    Block diffusion: Interpolating between autoregressive and diffusion language models

    Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and V olodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. In International Conference on Learning Repre- sentations, 2025

    work page 2025

  4. [4]

    Video pretraining (vpt): Learning to act by watching unlabeled online videos

    Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639–24654, 2022

    work page 2022

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 33:1877– 1901, 2020

    work page 1901

  6. [6]

    Genie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024

    2024

  7. [7]

    Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems

    Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang, Shu Jiang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503.06669, 2025

    arXiv 2025

  8. [8]

    Diffusion forcing: Next-token prediction meets full-sequence diffusion

    Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 37:24081–24125, 2024

    2024

Show all 68 references
  1. [9]

    Sana-sprint: One-step diffusion with continuous-time consistency distillation

    Junsong Chen, Shuchen Xue, Yuyang Zhao, Jincheng Yu, Sayak Paul, Junyu Chen, Han Cai, Enze Xie, and Song Han. Sana-sprint: One-step diffusion with continuous-time consistency distillation. arXiv preprint arXiv:2503.09641, 2025

    2025

  2. [10]

    Igor: Image-goal representations are the atomic control units for foundation models in embodied ai

    Xiaoyu Chen, Junliang Guo, Tianyu He, Chuheng Zhang, Pushi Zhang, Derek Cathera Yang, Li Zhao, and Jiang Bian. Igor: Image-goal representations are the atomic control units for foundation models in embodied ai. arXiv preprint arXiv:2411.00785, 2024

    2024 arXiv

  3. [11]

    Cat pruning: Cluster-aware token pruning for text-to-image diffusion models

    Xinle Cheng, Zhuoming Chen, and Zhihao Jia. Cat pruning: Cluster-aware token pruning for text-to-image diffusion models. arXiv preprint arXiv:2502.00433, 2025

    2025 arXiv

  4. [12]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023

    2023

  5. [13]

    Speculative diffusion decoding: Accelerating language generation through diffusion

    Jacob K Christopher, Brian R Bartoldson, Tal Ben-Nun, Michael Cardei, Bhavya Kailkhura, and Ferdinando Fioretto. Speculative diffusion decoding: Accelerating language generation through diffusion. arXiv preprint arXiv:2408.05636, 2024. 10

    2024 arXiv

  6. [14]

    Accelerated diffusion models via speculative sampling

    Valentin De Bortoli, Alexandre Galashov, Arthur Gretton, and Arnaud Doucet. Accelerated diffusion models via speculative sampling. arXiv preprint arXiv:2501.05370, 2025

    2025 arXiv

  7. [15]

    Oasis: A universe in a transformer

    Decart, Quevedo Julian, McIntyre Quinn, Campbell Spruce, Chen Xinlei, and Wachen Robert. Oasis: A universe in a transformer. 2024

    2024

  8. [16]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems (NeurIPS), 34:8780–8794, 2021

    2021

  9. [17]

    Learning universal policies via text-guided video generation

    Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in neural information processing systems, 36:9156–9172, 2023

    2023

  10. [18]

    Scaling rectified flow trans- formers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first international conference on machin...

    2024

  11. [19]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021

    2021

  12. [20]

    Vista: A generalizable driving world model with high fidelity and versatile controllability

    Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. In The Thirty-eighth Annual Conference on Neural Information Processing Syst...

    2024

  13. [21]

    Long-context autoregressive video modeling with next-frame prediction

    Yuchao Gu, Weijia Mao, and Mike Zheng Shou. Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503.19325, 2025

    2025 arXiv

  14. [22]

    Mineworld: a real-time and open-source interactive world model on minecraft

    Junliang Guo, Yang Ye, Tianyu He, Haoyu Wu, Yushu Jiang, Tim Pearce, and Jiang Bian. Mineworld: a real-time and open-source interactive world model on minecraft. arXiv preprint arXiv:2504.08388, 2025

    2025 arXiv

  15. [23]

    World models

    David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018

    2018 arXiv

  16. [24]

    Mastering diverse domains through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023

    2023 arXiv

  17. [25]

    Imagen video: High definition video generation with diffusion models

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022

    arXiv

  18. [26]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 33:6840–6851, 2020

    2020

  19. [27]

    Gaia-1: A generative world model for autonomous driving

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080, 2023

    2023 arXiv

  20. [28]

    Elucidating the design space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems , 35:26565–26577, 2022

    2022

  21. [29]

    Learning to simulate dynamic environments with gamegan

    Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, and Sanja Fidler. Learning to simulate dynamic environments with gamegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1231–1240, 2020

    2020

  22. [30]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015

    2015

  23. [31]

    Videopoet: A large language model for zero-shot video generation

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jose Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video generation. In International Conference on Machine Learning, pages 2510...

    2024

  24. [32]

    Hunyuanvideo: A systematic framework for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  25. [33]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR, 2023

    2023

  26. [34]

    Autoregressive image generation without vector quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37:56424–56445, 2024

    2024

  27. [35]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023

    2023

  28. [36]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023

    2023

  29. [37]

    Simplifying, stabilizing and scaling continuous-time consistency models

    Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-time consistency models. arXiv preprint arXiv:2410.11081, 2024

    2024 arXiv

  30. [38]

    Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022

    2022 arXiv

  31. [39]

    OpenAI. Sora. https://openai.com/index/sora/, 2024

    2024

  32. [40]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019

    2019

  33. [41]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  34. [42]

    Movie gen: A cast of media foundation models

    A Polyak, A Zohar, A Brown, A Tjandra, A Sinha, A Lee, A Vyas, B Shi, CY Ma, CY Chuang, et al. Movie gen: A cast of media foundation models. 2024a. arXiv preprint arXiv:2410.13720, 2024

    2024 arXiv

  35. [43]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part ...

    2015

  36. [44]

    Gaia-2: A controllable multi-view generative world model for autonomous driving

    Lloyd Russell, Anthony Hu, Lorenzo Bertoni, George Fedoseev, Jamie Shotton, Elahe Arani, and Gianluca Corrado. Gaia-2: A controllable multi-view generative world model for autonomous driving. arXiv preprint arXiv:2503.20523, 2025

    2025 arXiv

  37. [45]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022

    2022 arXiv

  38. [46]

    Sand.ai. Magi-1. https://sand.ai/magi, 2025

    2025

  39. [47]

    Mastering atari, go, chess and shogi by planning with a learned model

    Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020

    2020

  40. [48]

    Deep unsuper- vised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256–2265. pmlr, 2015

    2015

  41. [49]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 12

    2010 arXiv

  42. [50]

    Consistency models

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, pages 32211–32252. PMLR, 2023

    2023

  43. [51]

    Vidtok: A versatile and open-source video tokenizer

    Anni Tang, Tianyu He, Junliang Guo, Xinle Cheng, Li Song, and Jiang Bian. Vidtok: A versatile and open-source video tokenizer. arXiv preprint arXiv:2412.13061, 2024

    2024 arXiv

  44. [52]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  45. [53]

    Towards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018

    2018 arXiv

  46. [54]

    Diffusion models are real-time game engines

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837, 2024

    2024 arXiv

  47. [55]

    Sparsedm: Toward sparse efficient diffusion models

    Kafeng Wang, Jianfei Chen, He Li, Zhenpeng Mi, and Jun Zhu. Sparsedm: Toward sparse efficient diffusion models. arXiv preprint arXiv:2404.10445, 2024

    2024 arXiv

  48. [56]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600– 612, 2004

    2004

  49. [57]

    ivideogpt: Interactive videogpts are scalable world models

    Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. ivideogpt: Interactive videogpts are scalable world models. Advances in Neural Information Processing Systems, 37:68082–68119, 2024

    2024

  50. [58]

    Structured 3d latents for scalable and versatile 3d generation

    Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  51. [59]

    Pandora: Towards general world model with natural language actions and video states

    Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, et al. Pandora: Towards general world model with natural language actions and video states. arXiv preprint arXiv:2406.09455, 2024

    2024 arXiv

  52. [60]

    Learning interactive real-world simulators

    Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kaelbling, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In The Twelfth International Conference on Learning Representations, 2024

    2024

  53. [61]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024

    2024 arXiv

  54. [62]

    One-step diffusion with distribution matching distillation

    Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6613–6623, 2024

    2024

  55. [63]

    From slow bidirectional to fast autoregressive video diffusion models

    Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models. arXiv preprint arXiv:2412.07772, 2, 2024

    2024

  56. [64]

    Evaluation agent: Effi- cient and promptable evaluation framework for visual generative models

    Fan Zhang, Shulin Tian, Ziqi Huang, Yu Qiao, and Ziwei Liu. Evaluation agent: Effi- cient and promptable evaluation framework for visual generative models. arXiv preprint arXiv:2412.09645, 2024

    2024 arXiv

  57. [65]

    The unrea- sonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018. 13

    2018

  58. [66]

    Drivedreamer-2: Llm-enhanced world models for diverse driving video gen- eration

    Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang. Drivedreamer-2: Llm-enhanced world models for diverse driving video gen- eration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 10412–10420, 2025

    2025

  59. [67]

    Genad: Gen- erative end-to-end autonomous driving

    Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chenming Zhang, and Long Chen. Genad: Gen- erative end-to-end autonomous driving. In European Conference on Computer Vision, pages 87–104. Springer, 2024

    2024

  60. [68]

    "" model : Distilled NFD + model vid : Input video tensor act : Action sequence tensor

    Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. Ro- bodreamer: Learning compositional world models for robot imagination. In International Conference on Machine Learning, pages 61885–61896. PMLR, 2024. 14 A Implementation Details Hyperparameters....

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.