Pith. sign in

REVIEW 3 major objections 6 minor 51 references

Progressive Growing of Video Tokenizers for Temporally Compact Latent Spaces

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Progressive growing of a 4× video tokenizer yields better 8× and 16× temporal compression than training the high-compression model directly, and the 16× latent space trains a diffusion model to match 4× generation quality with far fewer…

desk verdict Progressive growing of a video tokenizer to 8x/16x temporal compression gives real gains over a from-scratch baseline, but the paper never compares against a directly-trained version of its own modified architecture, so the attribution of those gains is under-determined. read the letter →

arxiv 2501.05442 v2 pith:6PEGRZIU submitted 2025-01-09 cs.CV cs.AIeess.IV

Aniruddha Mahapatra , Long Mai , David Bourgin , Yitian Zhang , Feng Liu This is my paper · ORCID
classification cs.CVcs.AIeess.IV
keywords videotokenizertemporalcompressionprogressivegrowinglatentdiffusionreconstructionqualitytext-to-videogenerationresidualembeddingAdaNorm
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the right way to reach high temporal compression in a video tokenizer is to grow an already-trained low-compression model rather than train the whole high-compression model from scratch. The key observation is that a 4× tokenizer fed a temporally subsampled, low-framerate video reconstructs that video more accurately than a 16× tokenizer reconstructs the original full-framerate video. ProMAG therefore keeps a frozen 4× encoder and decoder, inserts extra bottleneck downsampling and upsampling blocks, and trains only those blocks to capture the residual information needed for in-between frames. On standard reconstruction benchmarks, this progressive growing beats directly trained baselines at 8× and 16× compression, and the resulting 16× latent space trains a video diffusion model to match 4× generation quality while using about 2.5× fewer tokens per step. The paper also claims to be the first to reach high-quality reconstruction at 16× temporal compression.

What carries the argument

The mechanism is progressive model growing with key-frame and residual embeddings. Stage I trains a base 4× tokenizer; Stage II (and Stage III) freeze the pretrained encoder and decoder blocks, insert extra bottleneck downsampling and upsampling blocks, and subsample the input by 2 to obtain key-frame embeddings through the frozen encoder. An adaptive group-normalization (AdaNorm) layer conditions the full-video intermediate features on those key-frame embeddings, the new bottleneck blocks compress the conditioned features into a residual embedding, and a 1×1×1 convolution combines the key-frame and residual latents. This forces the newly added blocks to learn only the in-between frame information, turning high compression into a guided interpolation problem rather than a from-scratch reconstruction problem.

What would settle it

Train the identical architecture at 8× or 16× temporal compression from scratch with the same data and iteration budget, then compare PSNR, LPIPS, and FVD on the same public benchmark; if the from-scratch model matches or beats ProMAG, the progressive-growing mechanism is not what carries the result. A more targeted check is to replace the frozen 4× encoder with a randomly initialized or lightly trained encoder while keeping the key-frame and AdaNorm pathway intact; if reconstruction quality stays high, the paper's core premise about reusing a well-trained lower-compression representation is unnecessary.

Watch

Extended reading notes

Core claim

The central claim is that a video tokenizer supporting 8× or 16× temporal compression can be obtained by bootstrapping a well-trained 4× temporal-compression model: freeze the pretrained encoder and decoder blocks, insert additional 2× bottleneck downsampling and upsampling layers, and train only the new layers plus a 1×1×1 bottleneck to learn the residual information that the frozen 4× representation does not already carry. A cross-level feature-mixing scheme conditions the intermediate encoder features on key-frame embeddings obtained by encoding a temporally subsampled version of the input with the frozen encoder, and the final latent is a linear combination of the key-frame and residual embeddings. The paper reports that this scheme outperforms directly extending and training MagViT-v2 at 8× and 16× compression on both reconstruction metrics and the representational quality for downstream text-to-video diffusion, and that the 16× latent space produces generation quality comparable to 4× with substantially fewer tokens.

Load-bearing premise

The method's gains rest on the premise that a frozen, well-trained 4× encoder and decoder remain informative and compatible when additional bottleneck compression blocks are inserted, so that the new blocks only need to learn residual in-between-frame information; this is an empirical premise validated only on the authors' internal training data and a single base architecture.

Editorial extensions

If this is right

  • At fixed latent channel dimension, 8× and 16× temporal compression become usable for video generation, so a given token budget produces much longer videos: 340 frames with 16× latents instead of 136 with 4× latents at the same denoising cost.
  • Training a high-compression tokenizer by progressive growing is roughly 2.7× cheaper in cumulative GPU time than training the same model directly, because the frozen blocks need no gradients and no discriminator is used during the growth stages.
  • The compact 16× latent space is compatible with standard DiT-based text-to-video training, yielding generation quality that matches 4× latents on VBench while cutting per-timestep cost by about 2.5×.
  • The reconstruction gain is not simply stage-wise training: ablations show that including the key-frame and AdaNorm residual pathway adds a further large improvement over progressive training without those components.
  • A layer-wise spatial tiling trick for the decoder removes the artifacts that arise when decoding high-resolution videos in tiles, which is needed to make the compressed model practical at larger resolutions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that the same progressive recipe may transfer to other bottlenecks, such as spatial compression or mixed spatial-temporal compression, whenever a well-trained lower-compression stage already reconstructs subsampled versions of the input well.
  • The method's encoder cost is higher than the baseline because it performs two forward passes (one on the full video and one on the subsampled video); the paper reports about 1.5× encoder time, so the total efficiency gain depends on how often encoding is amortized against diffusion training and generation.
  • A testable extension is to apply the AdaNorm-conditioned residual grow to other base tokenizers and to measure whether the reconstruction gain survives when the frozen base model was trained on a different data distribution; the paper's results are from one internal training set, so transferability is not yet established.
  • The frame-chunking and overlap-blending workaround for videos longer than 17 frames suggests that a truly streaming or temporally autoregressive decoder could be a natural next step for making high-compression latents seamless on long footage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes ProMAG, a video tokenizer that extends MagViT-v2 to 8× and 16× temporal compression by progressively growing from a pretrained 4× model. The method trains a 4× base model first, then freezes its encoder/decoder blocks, inserts additional bottleneck downsampling/upsampling blocks, and uses AdaNorm to condition the intermediate latents on key-frame embeddings from temporally subsampled video. A residual path combines the key-frame and learned residual information. The paper reports that this progressive-growing approach achieves significantly better reconstruction quality than directly training MagViT-v2 at the same high compression ratios, and the resulting 16× latent space can train a DiT for text-to-video generation at quality comparable to 4× latents while using far fewer tokens. It also introduces three modifications to the base MagViT-v2 architecture (frozen 2D image encoder initialization, discarding the first frame in decoder upsampling, and a group-norm variant without mean subtraction) and a layer-wise spatial tiling method for high-resolution decoding.

Significance. If the central claim is confirmed, this is a practically valuable contribution to efficient latent video diffusion, since temporal compression beyond 4× without increasing latent channels could significantly reduce the token budget for diffusion transformers. The paper is well motivated by the subsampling observation in Figure 2, and it contains several good experimental practices: an ablation in Table 4 separating the full method from plain progressive training, a comparison against external frame interpolation in Table 5, a training-efficiency table in the appendix, and a downstream text-to-video validation that checks the latent space is actually usable by a DiT. The main weakness is experimental: the advantage of progressive growing over direct training is not isolated from the base-model modifications, because the direct-training baseline is vanilla MagViT-v2 rather than a from-scratch ProMAG. This is a fixable omission, but it currently leaves the core novelty insufficiently supported.

major comments (3)
  1. [§4.1, Tables 2 and 4] The central claim that progressive growing outperforms direct high-compression training is not isolated from the architectural modifications introduced in Section 3.1. Table 4 compares directly trained "MagViT-v2" with "ProMAG (w/o residuals &AdaNorm)", but the latter already includes the frozen 2D image encoder, the first-frame-discarding upsampling, and the custom group norm. The observed PSNR gain of 30.35 vs. 28.51 could be due entirely to these modifications rather than to progressive training. Since the abstract and contributions state that directly training the full high-compression model is the wrong strategy, the manuscript must include a from-scratch ProMAG baseline (same architecture, trained directly at 8× and 16× with comparable compute and iterations) in Table 4 and ideally also in Table 2. Without that row, the conclusion that progressive growing is beneficial is not supported.
  2. [§4.1, Tables 1–5] All reconstruction metrics are reported as point estimates without standard errors, confidence intervals, or significance tests. Several key comparisons are close, such as the LPIPS values of 6.53 vs. 6.49 for ProMAG and MagViT-v2 at z=8 in Table 1, and the PSNR gain of about 1.7 dB in Table 4 is the main evidence for the method's advantage. The abstract and Section 4.1 repeatedly use the word "significantly" without statistical support. The authors should report variance across benchmark clips or multiple training runs, and preferably a paired test, to justify the strength of the claims.
  3. [§3.1 and Table 4] The contribution of the individual base-model modifications to the high-compression setting is not quantified. In particular, the paper states that removing mean subtraction from group normalization resolves the spot artifacts (Figure 3), but it does not evaluate whether this modification also improves direct training at 8× or 16× compression. If the custom norm and image-init changes improve direct training, then part of the gap in Table 4 is attributable to Section 3.1 rather than to progressive growing. An ablation of the base modifications on a directly trained ProMAG model would help separate these factors and strengthen the interpretation of the progressive-growing results.
minor comments (6)
  1. [§3.2, Eq. (2)–(3)] The notation is inconsistent: the superscript * appears on some z variables, and the AdaNorm output is not clearly defined. Please define all symbols explicitly and clarify the dimensions and roles of zkey, zinter, and the final latent z.
  2. [§3.1] Please specify which pretrained 2D image model is used for the frozen image encoder and whether its architecture matches the image-encoding portion of MagViT-v2; this detail is needed for reproducibility.
  3. [Figure 5] The red-bordered rectangle marking the artifact region is not conspicuous. A thicker or brighter border would make the qualitative comparison easier to see.
  4. [§5.1, Table 4 caption] The dataset name is misspelled as "MCL-MJC"; it should be "MCL-JCV".
  5. [§1 and §4.1] The claim of being "the first to achieve high-quality reconstruction with a 16× temporal compression model" is strong; please qualify it with "to our knowledge" and state the precise comparison regime (continuous tokenizer, fixed latent channels, reconstruction benchmarks) so the novelty claim is verifiable.
  6. [§3.3] The layer-wise spatial tiling technique is described only qualitatively with Figure 5. Since this tiling is used for high-resolution decoding, please provide a quantitative comparison (e.g., PSNR/LPIPS with and without tiling) or state clearly whether the reported benchmark numbers use the tiling procedure for all methods consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the progressive-growing result is an empirical training-and-evaluation claim; the only self-citation is a metric reference and does not carry the argument.

full rationale

The paper's derivation chain is an empirical training-and-evaluation pipeline, not a derivation from assumed conclusions. The motivating observation (a 4x model reconstructs subsampled video better than a directly trained 16x model on original video, Figure 2) is used only to motivate the architecture; the target claims are then established by training on internal video data and measuring PSNR/LPIPS/rFVD on standard benchmarks (MCL-JCV, DAVIS) and VBench for generation. The progressive-growing equations (Eqs. 1-3) define the model, but success is not forced by construction: the frozen 4x blocks guarantee only keyframe reconstruction, while the new bottleneck blocks and AdaNorm conditioning must learn residual in-between-frame information, and Table 4 ablates this by removing residuals and AdaNorm (Progressive training row) to show the specific contribution. The main comparison in Table 2 is ProMAG versus MagViT-v2 trained directly at the target compression; while a from-scratch ProMAG baseline is not included, this is a benchmark-fairness and ablation-design concern, not circularity, because the baseline is an external architecture with independent training and the metrics are computed on held-out standard datasets. The only self-citation, [12] (rFVD metric, co-authored by A. Mahapatra), is a metric reference and is not load-bearing to the method's correctness; the original FVD [41] is also cited, and the metric is externally defined and computable. Hyperparameters are reported transparently in Appendix B. No fitted parameter is renamed as a prediction, and no uniqueness or ansatz is imported via a self-citation chain.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or natural entities. Its central claim rests on an empirical training recipe, so the ledger records the training-dependent hyperparameters and the unproven design assumptions that the method depends on.

free parameters (3)
  • KL divergence loss weight = 1e-12
    Chosen by hand following MagViT-v2; it controls regularization strength but is not fitted to the target result.
  • GAN loss weight for 4x base model = 0.1
    Usedi only when training the 4x base model; standard adversarial weighting, not tuned against the final 16x benchmark.
  • VGG perceptual loss weight = 1.0
    Standard perceptual loss weighting from Stable Diffusion VAE; no tuning reported.
assumptions (4)
  • standard math VAE objective with KL divergence, GAN loss, and perceptual loss is a valid training signal for video tokenizers.
    The paper relies on the established VAE/GAN framework without formally justifying it (Sections 3.1 and 4.1).
  • domain assumption The internal dataset of 300M images and 15M videos is representative enough to train robust video representations.
    Training details in Section 4.1 state the dataset but release no data, so this assumption is unfalsifiable from the paper alone.
  • ad hoc to paper Removing mean subtraction from GroupNorm resolves the 'spot' artifacts without introducing other degradation.
    Section 3.1 and Figure 3 show this choice fixes visible artifacts, but no theoretical explanation or controlled ablation is provided.
  • ad hoc to paper Layer-wise spatial tiling with linear interpolation weights approximates full-context decoding well enough to avoid artifacts.
    Section 3.3 and Figure 5 demonstrate the technique empirically; there is no analysis of when it might fail.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Progressive Growing of Video Tokenizers for Temporally Compact Latent Spaces." pith.science (2026). https://pith.science/paper/6PEGRZIU

@misc{pith2026250105442,
  author       = {Pith},
  title        = {Pith review of: Progressive Growing of Video Tokenizers for Temporally Compact Latent Spaces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6PEGRZIU}},
  note         = {Machine review of arXiv:2501.05442}
}
read the original abstract

Video tokenizers are essential for latent video diffusion models, converting raw video data into spatiotemporally compressed latent spaces for efficient training. However, extending state-of-the-art video tokenizers to achieve a temporal compression ratio beyond 4x without increasing channel capacity poses significant challenges. In this work, we propose an alternative approach to enhance temporal compression. We find that the reconstruction quality of temporally subsampled videos from a low-compression encoder surpasses that of high-compression encoders applied to original videos. This indicates that high-compression models can leverage representations from lower-compression models. Building on this insight, we develop a bootstrapped high-temporal-compression model that progressively trains high-compression blocks atop well-trained lower-compression models. Our method includes a cross-level feature-mixing module to retain information from the pretrained low-compression model and guide higher-compression blocks to capture the remaining details from the full video sequence. Evaluation of video benchmarks shows that our method significantly improves reconstruction quality while increasing temporal compression compared to directly training the full model. Furthermore, the resulting compact latent space effectively trains a video diffusion model for high-quality video generation with a significantly reduced token budget.

Figures

Figures reproduced from arXiv: 2501.05442 by the authors.

Figure 1
Figure 1. (left) Our video tokenizer ProMAG is capable of much more effectively reconstructing high motion videos even at very large (16×) temporal compression, compared to baseline MagViT-v2. Dotted boxes highlight regions where the MagViT-v2 fails catastrophically, like faces and hands, which have a lot of artifacts in the reconstructions. (right) We show that our highly compressed latent space (16×) is suitable for trainin… view at source ↗
Figure 3
Figure 3. ‘spot artifacts’. Our progressive growing approach with GroupNorm leads to the ‘spot’ like artifacts (left) at the bottom right corner in reconstructed videos. (right) Removing the mean subtraction from group normalization eliminates the spot artifacts in reconstructed frames. (ProMAG) base model. In Section 3.2, we analyze why the base ProMAG, capable of doing 4× temporal compression, has difficulties extending to … view at source ↗
Figure 4
Figure 4. Methodology. Figure shows details of our method of progressive growing. In Stage I (top) we show a method of training our base 4× video tokenizer. (bottom) we illustrate the detailed method of growing the base 4× model to achieve 8× temporal compression. tion frames when generating the final output. We find this operation to be wasteful in terms of computation and be￾comes a greater problem with higher temporal comp… view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: High-Resolution Reconstruction. (left) tilting the la￾tent before passing through the decoder leads to artifacts in the reconstructed video. (right) layer-wise tiling resolves the artifacts. The red-bordered frame at the bottom left corner represents the ground-truth f…
Figure 6
Figure 6. Figure 6: Effectiveness of Progressive Growing at high temporal compression. Video reconstruction example for (left) 16× temporal compression, (right) 8× compression. MagViT-v2 causes loss of details and produces artifacts in the reconstructed frames, like motion blur in the dog…
Figure 7
Figure 7. Figure 7: Text-to-Video Evaluation. Quantitative comparison of T2V generation model train on 4× and 16× latent space of ProMAG. (top) The graph of time taken per denoising step of the diffusion model for generating N frame video. The time is computed on H100 with Flash Attention…
Figure 9
Figure 9. Figure 9: Qualitative Comparison with External Interpolation. Reconstruction comparison of our method ProMAG with 16× tem￾poral compression on a 24fps video, against a baseline where we first encode the video at low fps (with frame subsampling sf = 4), in this case 6fps, followe…
Figure 10
Figure 10. Figure 10: Methodology. Figure shows details of our complete method of progressive growing for extending the temporal compression to 16× from 4× compression. In Stage I (top) we show a method of training our base 4× video tokenizer. (middle) Stage II we illustrate the detailed m…
Figure 11
Figure 11. Figure 11 [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 16 canonical work pages

  1. [1]

    Cosmos world foundation model platform for physical ai

    Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575,

    arXiv

  2. [2]

    ediff-i: Text-to-image dif- fusion models with an ensemble of expert denoisers

    Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. ediff-i: Text-to-image dif- fusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022. 2, 3

    arXiv 2022

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2

    arXiv 2023

  4. [4]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023. 2

    2023

  5. [5]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In CVPR, 2023. 2

    2023

  6. [6]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. 2

    2024

  7. [7]

    The 2019 davis challenge on vos: Unsupervised multi-object segmenta- tion

    Sergi Caelles, Jordi Pont-Tuset, Federico Perazzi, Alberto Montes, Kevis-Kokitsi Maninis, and Luc Van Gool. The 2019 davis challenge on vos: Unsupervised multi-object segmenta- tion. arXiv preprint arXiv:1905.00737, 2019. 6

    arXiv 2019

  8. [8]

    Deep com- pression autoencoder for efficient high-resolution diffusion models

    Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang, Haotian Tang, Muyang Li, Yao Lu, and Song Han. Deep com- pression autoencoder for efficient high-resolution diffusion models. arXiv preprint arXiv:2410.10733, 2024. 2, 3

    arXiv 2024

Show all 51 references
  1. [9]

    Scaling rectified flow trans- formers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim En- tezari, Jonas M¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first International Conference on Mac...

    2024

  2. [10]

    Make-a-scene: Scene-based text-to-image generation with human priors

    Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. arXiv preprint arXiv:2203.13131, 2022. 2, 3

    2022 arXiv

  3. [11]

    Preserve your own correlation: A noise prior for video diffusion models

    Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. arXiv preprint arXiv:2305.10474, 2023. 2, 11

    2023 arXiv

  4. [12]

    On the content bias in fr ´echet video distance

    Songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Jun-Yan Zhu, and Jia-Bin Huang. On the content bias in fr ´echet video distance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 6

    2024

  5. [13]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS,

  6. [14]

    Photorealistic video generation with diffusion models

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Fei-Fei Li, Irfan Essa, Lu Jiang, and Jos ´e Lezama. Photorealistic video generation with diffusion models. In European Conference on Computer Vision, pages 393–411. Springer, 2025. 2, 11

    2025

  7. [15]

    Videocrafter: A toolkit for text-to-video generation and editing.https:// github.com/VideoCrafter/VideoCrafter, 2023

    Yingqing He, Haoxin Chen, and Menghan Xia. Videocrafter: A toolkit for text-to-video generation and editing.https:// github.com/VideoCrafter/VideoCrafter, 2023. 2

    2023

  8. [16]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. In NeurIPS, 2020. 2

    2020

  9. [17]

    Imagen video: High definition video generation with diffusion models.arXiv preprint arXiv:2210.02303, 2022

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models.arXiv preprint arXiv:2210.02303, 2022. 2, 3, 11

    arXiv

  10. [18]

    Multimodal unsupervised image-to-image translation

    Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. ECCV,

  11. [19]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat...

    2024

  12. [20]

    Progressive growing of gans for improved quality, stability, and variation

    Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In ICLR, 2018. 3

    2018

  13. [21]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019. 3

    2019

  14. [22]

    Analyzing and improving the image quality of stylegan

    Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020. 4

    2020

  15. [23]

    Elucidating the design space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565–26577, 2022. 4

    2022

  16. [24]

    Auto-encoding varia- tional bayes

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. In ICLR, 2014. 2

    2014

  17. [25]

    Hunyuanvideo: A systematic framework for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. 6

    2024 arXiv

  18. [26]

    Open-sora-plan, 2024

    PKU-Yuan Lab and Tuzhan AI etc. Open-sora-plan, 2024. 2, 8 9

    2024

  19. [27]

    Wf-vae: Enhancing video vae by wavelet-driven energy flow for latent video diffusion model

    Zongjian Li, Bin Lin, Yang Ye, Liuhan Chen, Xinhua Cheng, Shenghai Yuan, and Li Yuan. Wf-vae: Enhancing video vae by wavelet-driven energy flow for latent video diffusion model. arXiv preprint arXiv:2411.17459, 2024. 5, 6

    2024 arXiv

  20. [28]

    Geometric gan

    Jae Hyun Lim and Jong Chul Ye. Geometric gan. arXiv preprint arXiv:1705.02894, 2017. 11

    2017 arXiv

  21. [29]

    Snap video: Scaled spatiotemporal transformers for text-to-video synthe- sis

    Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, et al. Snap video: Scaled spatiotemporal transformers for text-to-video synthe- sis. In Proceedings of the IEEE/CVF Conferen...

    2024

  22. [30]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  23. [31]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  24. [32]

    xgen-videosyn-1: High- fidelity text-to-video synthesis with compressed representa- tions

    Can Qin, Congying Xia, Krithika Ramakrishnan, Michael Ryoo, Lifu Tu, Yihao Feng, Manli Shu, Honglu Zhou, Anas Awadalla, Jun Wang, et al. xgen-videosyn-1: High- fidelity text-to-video synthesis with compressed representa- tions. arXiv preprint arXiv:2408.12590, 2024. 3

    2024 arXiv

  25. [33]

    Hierarchical text-conditional image genera- tion with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image genera- tion with clip latents. arXiv preprint arXiv:2204.06125, 2022. 2

    2022 arXiv

  26. [34]

    Ul- trapixel: Advancing ultra-high-resolution image synthesis to new peaks

    Jingjing Ren, Wenbo Li, Haoyu Chen, Renjing Pei, Bin Shao, Yong Guo, Long Peng, Fenglong Song, and Lei Zhu. Ul- trapixel: Advancing ultra-high-resolution image synthesis to new peaks. arXiv preprint arXiv:2407.02158, 2024. 2, 3

    2024 arXiv

  27. [35]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022. 2, 11

    2022

  28. [36]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv...

    arXiv

  29. [37]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. arXiv preprint arXiv:2407.08608, 2024. 7

    2024 arXiv

  30. [38]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792,

    arXiv

  31. [39]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021. 2

    2021

  32. [40]

    Vidtok: A versatile and open-source video tokenizer

    Anni Tang, Tianyu He, Junliang Guo, Xinle Cheng, Li Song, and Jiang Bian. Vidtok: A versatile and open-source video tokenizer. arXiv preprint arXiv:2412.13061, 2024. 5, 6

    2024 arXiv

  33. [41]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 6

    2018 arXiv

  34. [42]

    Wan: Open and advanced large-scale video generative models

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianx- iao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jin- gren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, P...

    2025 arXiv

  35. [43]

    Mcl-jcv: a jnd-based h

    Haiqiang Wang, Weihao Gan, Sudeng Hu, Joe Yuchieh Lin, Lina Jin, Longguang Song, Ping Wang, Ioannis Katsavouni- dis, Anne Aaron, and C-C Jay Kuo. Mcl-jcv: a jnd-based h. 264/avc video quality assessment dataset. In ICIP, 2016. 6

    2016

  36. [44]

    Omnitokenizer: A joint image- video tokenizer for visual generation

    Junke Wang, Yi Jiang, Zehuan Yuan, Bingyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint image- video tokenizer for visual generation. Advances in Neural Information Processing Systems, 37:28281–28295, 2025. 5, 6

    2025

  37. [45]

    High-resolution image synthesis and semantic manipulation with conditional gans

    Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018. 3

    2018

  38. [46]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 2, 3, 5, 6, 11

    2024 arXiv

  39. [47]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, Jos´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023. 2, 3, 5, 11

    arXiv

  40. [48]

    Extracting motion and appearance via inter-frame attention for efficient video frame interpolation

    Guozhen Zhang, Yuhan Zhu, Haonan Wang, Youxin Chen, Gangshan Wu, and Limin Wang. Extracting motion and appearance via inter-frame attention for efficient video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5682–5692,

  41. [49]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 6

    2018

  42. [50]

    Cv- vae: A compatible video vae for latent generative video mod- 10 els

    Sijie Zhao, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Muyao Niu, Xiaoyu Li, Wenbo Hu, and Ying Shan. Cv- vae: A compatible video vae for latent generative video mod- 10 els. Advances in Neural Information Processing Systems, 37: 12847–12871, 2025. 5, 6

    2025

  43. [51]

    Open-sora: Democratizing efficient video production for all, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024. 2, 8 A. Progressive Growing - from 8× to 16× Tem- poral Compression In Figure 2 of the main...

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.