Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Taming Teacher Forcing for Masked Autoregressive Video Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A masked autoregressive video generation framework, MAGI, claims that conditioning next-frame prediction on complete observation frames instead of masked ones improves first-frame video prediction FVD by 23%.

desk verdict A credible controlled ablation shows CTF beats MTF for short-horizon frame-level video prediction, but the long-horizon and 100-frame claims are under-supported. read the letter →

arxiv 2501.12389 v1 pith:QZYEY5TJ submitted 2025-01-21 cs.CV

Deyu Zhou , Quan Sun , Yuang Peng , Kun Yan , Runpei Dong , Duomin Wang , Zheng Ge , Nan Duan
show 3 more authors
Xiangyu Zhang Lionel M. Ni Heung-Yeung Shum
This is my paper · ORCID
classification cs.CV
keywords videogenerationautoregressivemaskedmodelingteacherforcingcompleteexposurebiaspredictionframe-levelautoregression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that the teacher-forcing scheme used in masked autoregressive video generation is a primary determinant of temporal quality. Its proposed Complete Teacher Forcing (CTF) feeds each masked frame only complete, unmasked history frames during training, unlike Masked Teacher Forcing (MTF), which trains on masked observations. The paper reports that CTF outperforms MTF by about 23% in FVD on first-frame conditioned video prediction, with better motion coherence. It also introduces dynamic interval training and dynamic noise injection to reduce exposure bias and error accumulation, and shows the combined system generates coherent sequences beyond 100 frames after training on 16-frame clips. If correct, this makes frame-level autoregressive video generation a stronger, more scalable alternative to patch-level and non-autoregressive approaches.

What carries the argument

The load-bearing mechanism is the Complete Teacher Forcing paradigm and its custom temporal attention mask. During training, the model input interleaves complete observation frames with masked frames; the attention mask lets each masked frame see only itself and the complete preceding history, so the training-time conditioning matches inference, where a standard causal mask lets each frame attend to previously generated frames. The supporting machinery consists of a diffusion-based masked-token prediction head, learnable interval embeddings that encode the desired frame spacing, and learnable noise-level embeddings used with injected Gaussian noise on observation frames; the latter two are the paper's targeted responses to exposure bias and error accumulation.

What would settle it

Compare CTF and MTF in a fully autoregressive long-horizon setting (100+ frames) with identical dynamic interval training and noise injection, and plot FVD per step. If CTF's advantage over MTF shrinks or reverses after several self-conditioned steps, the claimed benefit is specific to ground-truth conditioning rather than a general property of complete teacher forcing.

Watch

Extended reading notes

Core claim

The central claim is that the training-inference mismatch in masked teacher forcing is a major cause of poor temporal coherence in autoregressive video models. In MTF, a model learns to predict each masked frame from previously masked frames, but at inference it must condition on complete frames it has generated, so the two regimes diverge. CTF removes this mismatch by always conditioning the prediction of a masked frame on complete observation frames from the past, using a deliberately designed temporal attention mask where each masked frame attends to itself and preceding unmasked frames, and observation frames attend only to earlier frames. The paper reports that this yields a roughly 23% FVD improvement over MTF in first-frame conditioned prediction, and that CTF's advantage is specifically in motion and temporal coherence rather than single-frame visual quality.

Load-bearing premise

The reported 23% advantage is measured with ground-truth conditioning frames, and the paper assumes the same advantage survives fully autoregressive generation, where the model conditions on its own past outputs and only unquantified noise injection bridges the gap.

Editorial extensions

If this is right

  • First-frame conditioned video prediction improves by roughly 23% in FVD when the conditioning history is complete rather than masked, with the gain concentrated in motion coherence.
  • Frame-level autoregressive generation with CTF can match or beat prior autoregressive video baselines on standard benchmarks while keeping causal attention, so inference can use KV cache and scale near-linearly with frame count.
  • Training on clips as short as 16 frames can yield coherent prediction beyond 100 frames for scenes with static backgrounds and simple motion, indicating the method's length scalability.
  • Dynamic interval training and dynamic noise injection each improve FVD and FID on their own, and their combination is best, providing a recipe for reducing exposure bias in autoregressive video models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The +23% gain is measured with ground-truth conditioning frames; the paper's fully autoregressive regime is only defended by noise injection, whose match to the real error distribution is not quantified. I infer that the long-sequence benefit would need an explicit self-conditioning comparison to hold in the regime the paper emphasizes.
  • Because CTF doubles the training sequence length only, and inference keeps the standard causal form, I infer the same complete-conditioning trick could transfer to interactive or world-model settings that currently use masked teacher forcing, at the cost of 2x training sequence length.
  • The reported divergence between per-frame FID (MTF better) and FVD (CTF better) suggests evaluations that optimize single-frame quality can miss temporal coherence; I infer that video generation benchmarks should report both, and that frame-wise FID alone is insufficient.
  • The learnable interval embedding gives the model an explicit control signal for motion speed; I infer this could be exposed as a user-specified generation interval or frame-rate control at inference, which the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces MAGI, a hybrid video generation framework that performs frame-level autoregressive generation with a masked diffusion head, and proposes Complete Teacher Forcing (CTF) as an alternative to Masked Teacher Forcing (MTF). In CTF, each masked frame is conditioned on the complete, unmasked observation frames from the history, whereas MTF conditions on masked (partially observable) frames during training. The central empirical claim is that CTF improves FVD by roughly 23% over MTF on first-frame conditioned video prediction. The paper also introduces dynamic interval training and dynamic noise injection to mitigate exposure bias, reports state-of-the-art FVD among autoregressive methods on Kinetics-600 and UCF-101, and claims that MAGI can generate coherent sequences exceeding 100 frames even when trained on 16-frame clips.

Significance. If the central claim is robust, the paper makes a useful and conceptually clean contribution: it identifies a concrete training-inference mismatch in masked autoregressive video generation and demonstrates that conditioning on complete observation frames during training improves temporal coherence. The architecture is practical, supporting KV-cache acceleration and variable context lengths, and the paper includes controlled ablations of the teacher-forcing variants and of the two training strategies. The reported comparisons to prior autoregressive methods are favorable. However, the strength of the quantitative evidence is currently uneven: the headline +23% result is only presented as a figure without exact values or error bars, and the long-horizon (>100 frames) claim rests on a single qualitative example. The paper also does not quantify the match between the injected noise and the actual distribution of rollout errors, which is crucial for the claimed transfer of CTF's advantage to long-sequence generation. These gaps prevent the paper from being accepted in its current form.

major comments (4)
  1. [§5.2, Fig. 5b] The central quantitative claim of a +23% FVD improvement of CTF over MTF is not backed by exact numbers in the text or in a table; it is only visible as curves in Fig. 5b. No standard deviations, confidence intervals, or number of evaluation runs/seeds are reported anywhere in the paper. Given that FVD is known to be sensitive to evaluation protocol and sampling, the reader cannot assess whether the reported difference is significant or stable. Please provide the exact FVD values for MTF and CTF at the reported frames, together with variance estimates based on multiple seeds or bootstrapping.
  2. [§5.5, Fig. 6] The abstract and Sec. 5.5 claim that MAGI 'can generate long, coherent video sequences exceeding 100 frames, even when trained on as few as 16 frames.' The evidence is a single qualitative example in Fig. 6, and the text immediately concedes that performance degrades for non-periodic motions such as diving. There is no quantitative evaluation (e.g., FVD, LPIPS, or motion metrics) at horizons beyond the training length. This is a load-bearing claim for the paper's stated potential for scalable generation. Please add quantitative long-horizon results, or explicitly moderate the claim to what the evidence supports.
  3. [§4 and §5.3] The dynamic noise injection strategy is motivated as simulating inference-time errors from autoregressive rollouts, but the paper provides no measurement of how well Gaussian noise with levels [1,5] matches the actual distribution of errors in model-generated context frames. The CTF advantage is demonstrated in a first-frame-conditioned setting where the conditioning history is mostly ground-truth or early model outputs; the paper's long-horizon transfer claim depends on the noise injection being well-calibrated. Please provide a quantitative comparison between injected noise and observed rollout error statistics, or evaluate CTF by feeding its own predictions back as conditioning frames during evaluation.
  4. [§5.2, Fig. 5b] The MTF versus CTF comparison is described as a controlled ablation, but the paper does not report the exact hyperparameters used for each variant. In particular, Sec. 5.2 states that MTF uses 'an optimal mask ratio of 70%–100%' but does not specify the mask ratio actually used in the MTF training runs, nor whether the two models had identical training epochs, noise levels, interval ranges, and seed initialization. If these settings are not matched, the +23% difference could be attributable to configuration mismatch rather than to the teacher-forcing mechanism. Please report the full hyperparameter sets for both MTF and CTF.
minor comments (6)
  1. [Abstract and §5.5] The phrase 'long, coherent video sequences exceeding 100 frames' appears in the abstract before the quantitative support is presented; please ensure the abstract reflects the qualified nature of this claim (e.g., 'in specific scenarios with relatively static backgrounds and simple motions').
  2. [§1] There are several typos in the introduction: 'reply on' should be 'rely on', and 'casual temporal attention' in §2.1 should be 'causal temporal attention'.
  3. [§5.4] The sentence 'our MAGI achieves -21.4 significantly lower FVD than Omni whose FVD is 32.9' is confusingly phrased; it should state that MAGI's FVD is 21.4 points lower than Omni's.
  4. [Fig. 5 caption] The caption contains a typo, 'predcition' for 'prediction', and the figure legends are small; the 'star' markers indicating FVD scores are difficult to distinguish in grayscale.
  5. [§A.2.2] The authors note that MAGVIT-1 uses 300,000 real videos for FVD evaluation while this paper uses 50,000; although the paper states this avoids benefiting from larger sample size, the comparison in Table 2 is not strictly like-for-like and should be flagged in the table caption.
  6. [§4] The interval embedding vocabulary is stated to cover intervals from 1 to 25, but Sec. A.2.1 says the interval is uniformly sampled from 1 to 25; please clarify whether the vocabulary size of 25 includes the endpoint 25 and whether interval 0 is ever used.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CTF-vs-MTF advantage is an empirical benchmark result, not a derivation from fitted inputs or self-citations.

full rationale

The paper's central claim—CTF outperforms MTF by +23% in FVD—is an experimental measurement on UCF-101 and Kinetics-600, not a quantity derived from the method's definitions. CTF and MTF are defined as different training-conditioning schemes (Eqs. 3 and 4), and the reported FVD/FID figures in Figs. 4-5 and Tabs. 2-3 are obtained by training and evaluating both variants. No fitted constant is later renamed as a prediction; the +23% figure is a direct comparison of measured metrics. The only definitional component is that CTF, by construction, makes the training conditioning distribution (complete frames) equal to the inference conditioning distribution, but the paper does not rest its empirical superiority claim on that equivalence alone—it measures FVD. The long-sequence (100-frame) claim is supported by a qualitative example plus an explicit concession of degradation for non-periodic motions, which is a limitation statement rather than circular reasoning. Citations to MAR and GameNGen are for architecture components and noise-injection practice; they are not used to forbid alternatives or to import a uniqueness theorem, and no author-overlapping citation carries a load-bearing argument. Thus no circular step can be quoted.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central comparison (CTF vs MTF) is an empirical ablation, so it does not depend on fitted constants. The framework does depend on a set of design choices inherited from MAR and GameNGen: diffusion head parameters, mask/denoising steps, VAE tokenizers, and the ranges of the two new learnable embeddings. These are not derived from data and could in principle affect the magnitude of the reported gains.

free parameters (4)
  • Interval embedding vocabulary size = 25 (intervals 1-25)
    Chosen by hand for dynamic interval training; bounds the temporal stride diversity the model can represent.
  • Noise level embedding range = [1, 5]
    Chosen by hand for dynamic noise injection; encodes the amount of Gaussian noise added to observation frames.
  • Masked prediction steps = 64
    Inherited from MAR; number of iterative decoding steps per frame at inference.
  • Diffusion denoising steps = 100
    Number of denoising steps in the diffusion head, following MAR; not justified by an ablation.
assumptions (4)
  • domain assumption Pretrained video VAEs (OmniTokenizer, SD1.4, Cosmos) provide a faithful tokenization for video generation
    All experiments use these frozen tokenizers; if the VAE loses temporal information, the downstream FVD gains may be an artifact of the tokenizer, not the generation model.
  • domain assumption Frame-level autoregressive factorization p(V)=prod_t p(f_t | f_<t) is a valid model of video
    The paper assumes temporal causality at frame granularity; intra-frame dependencies are handled by masked prediction.
  • ad hoc to paper The MAR-style diffusion head with 100 denoising steps and 64 masked iterations approximates the true conditional distribution
    These constants are inherited from MAR; no ablation shows they are optimal for video.
  • domain assumption FVD on 50K (K600) or 2,048 (UCF-101) samples reliably ranks video generation methods
    FVD is the only reported metric for temporal quality; no human evaluation or other metric corroborates the claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Taming Teacher Forcing for Masked Autoregressive Video Generation." pith.science (2026). https://pith.science/paper/QZYEY5TJ

@misc{pith2026250112389,
  author       = {Pith},
  title        = {Pith review of: Taming Teacher Forcing for Masked Autoregressive Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QZYEY5TJ}},
  note         = {Machine review of arXiv:2501.12389}
}
read the original abstract

We introduce MAGI, a hybrid video generation framework that combines masked modeling for intra-frame generation with causal modeling for next-frame generation. Our key innovation, Complete Teacher Forcing (CTF), conditions masked frames on complete observation frames rather than masked ones (namely Masked Teacher Forcing, MTF), enabling a smooth transition from token-level (patch-level) to frame-level autoregressive generation. CTF significantly outperforms MTF, achieving a +23% improvement in FVD scores on first-frame conditioned video prediction. To address issues like exposure bias, we employ targeted training strategies, setting a new benchmark in autoregressive video generation. Experiments show that MAGI can generate long, coherent video sequences exceeding 100 frames, even when trained on as few as 16 frames, highlighting its potential for scalable, high-quality video generation.

Figures

Figures reproduced from arXiv: 2501.12389 by the authors.

Figure 1
Figure 1. Conceptual comparison of masked teacher forcing and our proposed complete teacher forcing mechanisms in auto￾regressive video generation. The blocks here illustrate individual frames. These two mechanisms differ in how history observation frames are used to condition next frame generation during training. (a): The observation frames are masked frames [2, 12] that are par￾tially observable. (b): Every frame is genera… view at source ↗
Figure 2
Figure 2. Overview of MAGI video generation framework. MAGI receives observation frames and corresponding masked frames as inputs, enabling autoregressive video generation with Complete Teacher Forcing (Sec. 3). To address this issue and more effectively bridge the gap between training and inference, we propose a novel au￾toregressive paradigm, termed Complete Teacher Forcing (CTF), as shown on the left of [PITH_FULL_IMAGE:f… view at source ↗
Figure 4
Figure 4. Case Study of Proposed Training Techniques: This figure evaluates the impact of dynamic interval training and dynamic noise injection on CTF and MTF by: 1) visualizing CTF with and without these strategies; and 2) comparing CTF and MTF when both use them. All methods perform first-frame conditional video prediction on UCF-101 [31]. The results demonstrate the efficacy of the proposed training strategies and the supe… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: a) Speed of generation process with varying numbers of frames. MAGI achieves a significant speedup by utilizing KV Cache. b) Masked Teacher Forcing (MTF) v.s. Complete Teacher Forcing (CTF). Both methods utilize the proposed training strategies — dynamic interval train…
Figure 6
Figure 6. Figure 6: Long-term Video Prediction. MAGI predicts over 100 frames from a single input frame, maintaining reasonable motion even when trained on only 16 frames. 6. Related Work Autoregressive Visual Generation Autoregressive lan￾guage modeling [24, 25] has propelled the develop…
Figure 7
Figure 7. Figure 7: Case Study: Video Prediction on Kinetics-600. MAGI generates high-quality future frames conditioned on past frames. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Vorch-IR: Long-Form Unified Multimodal Identity Replacement Video Generation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Vorch-IR replaces one or two subjects' identities, with optional background replacement, in a driving video using indexed reference images and a textual instruction, trained on automatically synthesized pairs.

  2. VideoMAR: Autoregressive Video Generatio with Continuous Tokens

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A decoder-only autoregressive video model with continuous tokens, frame-wise causal attention, and a next-frame diffusion loss reports a higher VBench-I2V score than Cosmos I2V with a much smaller model and dataset.

Reference graph

Works this paper leans on

51 extracted references · 23 canonical work pages · cited by 2 Pith papers

  1. [1]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  2. [2]

    Ge- nie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Ge- nie: Generative interactive environments. InForty-first Inter- national Conference on Machine Learning, 2024. 1, 2, 3, 8

    work page 2024

  3. [3]

    A short note about kinetics-

    Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-

  4. [4]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315–11325, 2022. 1, 3, 8

    work page 2022

  5. [5]

    Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan

    Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704 ,

    arXiv

  6. [6]

    Diffusion forcing: Next-token prediction meets full-sequence diffu- sion

    Boyuan Chen, Diego Marti Monso, Yilun Du, Max Sim- chowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffu- sion. arXiv preprint arXiv:2407.01392, 2024. 1, 2, 3

    arXiv 2024

  7. [7]

    Pixart-α: Fast training of dif- fusion transformer for photorealistic text-to-image synthesis,

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-α: Fast training of dif- fusion transformer for photorealistic text-to-image synthesis,

  8. [8]

    Fluid: Scaling autoregressive text-to-image generative models with continuous tokens

    Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410.13863, 2024. 1

    arXiv

Show all 51 references
  1. [9]

    Model tells you what to discard: Adaptive KV cache compression for llms

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for llms. In The Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net,

    2024

  2. [10]

    Courville, and Yoshua Bengio

    Anirudh Goyal, Alex Lamb, Ying Zhang, Saizheng Zhang, Aaron C. Courville, and Yoshua Bengio. Professor forc- ing: A new algorithm for training recurrent networks. In Advances in Neural Information Processing Systems 29: An- nual Conference on Neural Information Processing Syst...

    2016

  3. [11]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 8

    2023 arXiv

  4. [12]

    Girshick

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Pi- otr Doll ´ar, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 15979–15988....

    2022

  5. [13]

    Latent video diffusion models for high-fidelity video generation with arbitrary lengths

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221, 2(3):4, 2022. 8

    2022 arXiv

  6. [14]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 5, 8

    2020

  7. [15]

    Video dif- fusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022. 7, 8

    2022

  8. [16]

    Scalable adap- tive computation for iterative generation

    Allan Jabri, David Fleet, and Ting Chen. Scalable adap- tive computation for iterative generation. arXiv preprint arXiv:2212.11972, 2022. 7

    2022 arXiv

  9. [17]

    Videopoet: A large language model for zero-shot video gen- eration

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos ´e Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vigh- nesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video gen- eration. arXiv preprint arXiv:2312.14125, 2023. 1, 2, 8

    2023 arXiv

  10. [18]

    Dart: Noise injection for robust imitation learning

    Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. In Conference on robot learning, pages 143–156. PMLR, 2017. 4, 8

    2017

  11. [19]

    Autoregressive image generation without vec- tor quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vec- tor quantization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 1, 3, 5, 11

    2024

  12. [20]

    Keep the cost down: A review on methods to opti- mize LLM’s KV-cache consumption

    Shi Luohe, Hongyi Zhang, Yao Yao, Zuchao Li, and hai zhao. Keep the cost down: A review on methods to opti- mize LLM’s KV-cache consumption. In First Conference on Language Modeling, 2024. 1, 2

    2024

  13. [21]

    Latte: Latent diffusion transformer for video generation

    Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Zi- wei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024. 5, 6, 7

    2024 arXiv

  14. [22]

    Scaling diffusion mamba with bidirectional ssms for efficient image and video gener- ation

    Shentong Mo and Yapeng Tian. Scaling diffusion mamba with bidirectional ssms for efficient image and video gener- ation. arXiv preprint arXiv:2405.15881, 2024. 7

    2024 arXiv

  15. [23]

    Cosmos tokenizer

    NVIDIA. Cosmos tokenizer. https://github.com/ NVIDIA/Cosmos-Tokenizer , 2024. Accessed: 2024- 11-15. 5, 6, 7

    2024

  16. [24]

    Improving language understanding by gener- ative pre-training

    Alec Radford. Improving language understanding by gener- ative pre-training. 2018. 2, 8 9

    2018

  17. [25]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAI blog, 1(8):9, 2019. 2, 8

    2019

  18. [26]

    Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever

    A. Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ArXiv, abs/2102.12092,

    arXiv

  19. [27]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 5, 7, 8

    2022

  20. [28]

    Generalization in generation: A closer look at exposure bias

    Florian Schmidt. Generalization in generation: A closer look at exposure bias. In Proceedings of the 3rd Workshop on Neural Generation and Translation@EMNLP-IJCNLP 2019, Hong Kong, November 4, 2019 , pages 157–167. As- sociation for Computational Linguistics, 2019. 2

    2019

  21. [29]

    Mostgan-v: Video generation with temporal motion styles

    Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Mostgan-v: Video generation with temporal motion styles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5652–5661, 2023. 7

    2023

  22. [30]

    Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2

    Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elho- seiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3626–3636, 2022. 7

    2022

  23. [31]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 ,

    arXiv

  24. [32]

    Dim: Diffusion mamba for efficient high-resolution image synthesis

    Yao Teng, Yue Wu, Han Shi, Xuefei Ning, Guohao Dai, Yu Wang, Zhenguo Li, and Xihui Liu. Dim: Diffusion mamba for efficient high-resolution image synthesis. arXiv preprint arXiv:2405.14224, 2024. 7

    2024 arXiv

  25. [33]

    A good image generator is what you need for high-resolution video synthe- sis

    Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N Metaxas, and Sergey Tulyakov. A good image generator is what you need for high-resolution video synthe- sis. arXiv preprint arXiv:2104.15069, 2021. 7

    2021 arXiv

  26. [34]

    Mocogan: Decomposing motion and content for video generation

    Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526–1535,

  27. [35]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 5

    2018 arXiv

  28. [36]

    Diffusion models are real-time game engines

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837, 2024. 2, 3, 4, 5, 8

    2024 arXiv

  29. [37]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 7, 8

    2017

  30. [38]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 4, 5

    2017

  31. [39]

    Phenaki: Variable length video generation from open domain textual descriptions

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kin- dermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Le...

    2022

  32. [40]

    Pre- dicting video with vqvae

    Jacob Walker, Ali Razavi, and A ¨aron van den Oord. Pre- dicting video with vqvae. arXiv preprint arXiv:2103.01950,

    arXiv

  33. [41]

    Omnitokenizer: A joint image-video tokenizer for visual generation

    Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng, Zux- uan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint image-video tokenizer for visual generation. arXiv preprint arXiv:2406.09399, 2024. 2, 5, 6, 7, 8

    2024 arXiv

  34. [42]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 1, 2

    2024 arXiv

  35. [43]

    Williams and David Zipser

    Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks.Neu- ral Computation, 1(2):270–280, 1989. 1, 2

    1989

  36. [44]

    N ¨uwa: Visual synthesis pre- training for neural visual world creation

    Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. N ¨uwa: Visual synthesis pre- training for neural visual world creation. In European con- ference on computer vision, pages 720–736. Springer, 2022. 1, 2

    2022

  37. [45]

    Videogpt: Video generation using vq-vae and trans- formers

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and trans- formers. arXiv preprint arXiv:2104.10157, 2021. 1, 2, 7, 8

    2021 arXiv

  38. [46]

    Magvit: Masked generative video transformer

    Lijun Yu, Yong Cheng, Kihyuk Sohn, Jos ´e Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2023

  39. [47]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, Jos ´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023. 7, 8

    arXiv

  40. [48]

    Generating videos with dynamics-aware implicit generative adversarial net- works

    Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial net- works. arXiv preprint arXiv:2202.10571, 2022. 7

    2022 arXiv

  41. [49]

    Video probabilistic diffusion models in projected latent space

    Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 18456–18466,

  42. [50]

    Bridging the gap between training and in- ference for neural machine translation

    Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. Bridging the gap between training and in- ference for neural machine translation. arXiv preprint arXiv:1906.02448, 2019. 4, 8 10 A. Additional Details A.1. Network Architecture Our spatial-temporal transformer comprises ...

    1906 arXiv

  43. [600]

    arXiv preprint arXiv:1808.01340, 2018. 5, 6, 7

    2018 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.