Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

VGDFR: Diffusion-based Video Generation with Dynamic Latent Frame Rate

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Nearly 3x faster video generation by dropping redundant latent frames

desk verdict A credible training-free video DiT accelerator whose core renoise assumption is unproven and whose advertised RoPE analysis is missing, but worth a serious referee. read the letter →

arxiv 2504.12259 v1 pith:WQLXHHEP submitted 2025-04-16 cs.CV

Zhihang Yuan , Rui Xie , Yuzhang Shang , Hanling Zhang , Siyuan Wang , Shengen Yan , Guohao Dai , Yu Wang This is my paper · ORCID
classification cs.CV
keywords dynamiclatentframeratediffusiontransformervideogenerationtraining-freeaccelerationtokenmergingrotarypositionembeddingflowmatchinginterpolation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that diffusion transformer video models waste most of their compute on temporally redundant tokens: low-motion segments of a video can be represented by fewer latent frames without changing what the video says. VGDFR is a training-free inference-time method that measures inter-frame similarity during denoising, merges redundant latent frames in low-motion segments, and keeps the full frame rate where motion is fast. On the HunyuanVideo model, the method shortens generation from about 715 seconds to about 245 seconds, a 2.92x speedup, while text-alignment and aesthetic metrics stay nearly unchanged. The practical stake is that long or high-resolution video generation can become much cheaper simply by exploiting temporal redundancy the model already contains.

What carries the argument

The central object is the VGDFR Compression Module, a five-step detour inserted into the denoising loop: one-step denoise to preview content, decode to low resolution, schedule a dynamic frame rate via an SSIM threshold $\theta$, encode the merged preview back to latent space, and renoise by linear interpolation with the original noise. The dynamic frame rate scheduler is the decision mechanism: it marks a segment as mergeable only if every pair of frames in the segment exceeds the similarity threshold. DyRoPE is the supporting positional encoding machinery that keeps the compressed token sequence compatible with the pretrained transformer, alternating Global-RoPE, which deletes positions of merged frames, and Local-RoPE, which keeps a continuous positional span, according to each layer's sensitivity. Together these reduce the number of latent tokens the DiT processes in the later denoising steps.

What would settle it

Run VGDFR on prompts with uniform high motion, such as a fast camera pan, a shaking camera, or crowd movement, and compare outputs to the base model; if the scheduler still merges many frames or quality drops sharply where motion is constant, the motion estimation is not actually tracking content. More directly, compute the distribution of renoised latents $X^k_{dy}$ versus the latents the base model encounters at the same timestep; a large divergence would show that Eq. (6) leaves the learned trajectory.

Watch

Extended reading notes

Core claim

VGDFR establishes that a pretrained DiT video model can generate inside a dynamic latent frame rate space without any fine-tuning. After $k$ initial denoising steps, the method predicts the clean latent in one step, decodes it through a lightweight VAE decoder stripped of most upsampling layers, and compares frame pairs in low-resolution video space. Any contiguous segment whose pairwise similarity exceeds a threshold $\theta$ is merged by averaging; an equally light encoder maps the compressed preview back to latent space, and the result is linearly renoised as $X^k_{dy} = (1 - k/T)X^0_{dy} + (k/T)X^T_{dy,pre}$ so that denoising can resume on the flow-matching trajectory. A layer-wise dynamic rotary position embedding (DyRoPE) then keeps positional semantics: some layers drop the RoPE entries of merged frames (Global-RoPE), while others keep a continuous span of the original encodings (Local-RoPE). The paper's measured claim is up to a 3x speedup with minimal quality degradation, with the caveat that the preview-encode-decode loop makes frames differ in low-level similarity metrics from the uncompressed baseline.

Load-bearing premise

The whole pipeline assumes that a latent formed by compressing a decoded preview and linearly renoising it with noise still lies on the same flow-matching trajectory the DiT learned, even though the VAE weights were never exposed to noisy compressed latents; the paper's own ablation shows that removing the renoise step makes the video collapse.

Editorial extensions

If this is right

  • Applied to any flow-matching DiT video model with a latent VAE, the same compression loop should yield speedups without retraining, since the modified encoder and decoder reuse pretrained weights.
  • Raising the similarity threshold $\theta$ preserves more tokens and shrinks the speedup; at $\theta=0.9$ the reported gain falls to about 1.2-1.3x, so $\theta$ is a direct quality-efficiency dial.
  • Starting compression earlier in denoising (smaller $k$) buys larger speedups but risks losing local detail, because early latents are still noise-dominated; the reported 2.92x case starts at $k=5$.
  • The generated videos remain semantically aligned with the prompt even when low-level pixel metrics such as PSNR, SSIM, and LPIPS differ from the baseline, since merging changes surface appearance more than content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The renoise step is the load-bearing risk: if a compressed, re-encoded latent falls outside the support of latents the model saw in training, the flow-matching trajectory assumption silently breaks; a targeted check would measure the distribution gap of $X^k_{dy}$ against latents the model normally sees at that timestep.
  • The speedup ceiling should depend strongly on prompt content: videos dominated by static scenes could compress far more than the reported 2.92x average, while uniformly fast motion would push the gain toward the 1.2x floor, making the headline number prompt-dependent.
  • The same frame-merging idea could extend to spatial tokens in low-texture image regions, though the paper only compresses the temporal dimension.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The manuscript presents VGDFR, a training-free inference acceleration method for diffusion-transformer video generation. The core idea is to exploit temporal redundancy by adaptively lowering the latent frame rate in low-motion segments. After k denoising steps, the method generates a clean preview with a one-step denoise, decodes it to a low-resolution video with a truncated VAE decoder, segments frames by inter-frame similarity, merges redundant frames, re-encodes the result with a truncated VAE encoder, and renoises the compressed latent by linear interpolation with the original noise before resuming the denoising loop. The positional encodings are adjusted layer-wise (DyRoPE) to account for the compressed temporal indices. Experiments on HunyuanVideo report up to 2.92x wall-clock speedup at k=5, theta=0.5, with absolute quality metrics (CLIP-SIM, CLIP-Temp, VQA, flicker) close to the original, while relative-difference metrics (PSNR, SSIM, LPIPS) show large deviations. The paper's central assumption is that the renoised compressed latent in Eq. (6) stays on the flow-matching trajectory, enabling the pretrained DiT to continue denoising without finetuning.

Significance. VGDFR targets a real and important bottleneck: the high inference cost of DiT-based video generation. If the reported speedups and quality measurements are robust, the method would be a practically useful, training-free addition to the video-generation toolbox. The paper's strengths are its clear high-level idea, the use of wall-clock latency rather than FLOP counts, and the explicit layer-wise RoPE analysis. The contribution is, however, an incremental extension of the authors' prior DLFR-VAE concept, and the main novelty here is the decode-merge-encode-renoise loop applied to HunyuanVideo. The 'minimal quality degradation' claim is not yet fully supported: the evaluation lacks baselines, error bars, and a precisely specified prompt subset, and the load-bearing renoise step is justified by assertion rather than by distributional or ablative evidence. I therefore view the work as promising but in need of substantive revision.

major comments (3)
  1. [Section 3.2, Step 5 (Eq. 6)] The central validity claim that X_k_dy lies 'on the same trajectory between the original noise and the compressed clean latent' is not established. X_T_dy,pre is the output of a one-step denoising approximation followed by a decode-merge-encode cycle, so it is not a typical clean latent from the training distribution of the DiT; the paper itself states in Sec. 4.3 that the VAE 'has not been exposed to distributions with noisy latents.' Since the remaining denoising steps operate on this interpolated latent, a failure of the trajectory assumption would directly undermine the quality and generalizability claims. Please add a quantitative test of this assumption, such as a distributional distance of X_T_dy,pre to real VAE latents, an experiment continuing from the same X_k with and without compression, or a measure of denoising error accumulation over the remaining timesteps, and discuss the result.
  2. [Table 1] The relative-difference metrics at the headline setting (PSNR 15.4, SSIM 0.513, LPIPS 0.398) are large enough to conflict with the phrase 'minimal quality degradation,' yet the paper does not explain whether the comparison to the original model uses the same initial noise and seed. If the comparison is between independently sampled generations, these metrics mostly measure stochasticity and should not be used as distortion measures; if it is same-seed, the numbers need a direct interpretation. Please specify the protocol, add error bars over multiple seeds, and define the exact VBench subset, including the number and categories of prompts and the output resolution and length.
  3. [Section 4.2 and Related Work] The evaluation does not include any comparison with existing training-free acceleration methods for DiT video generation, such as the token-merging and caching baselines cited in Section 2. A speedup method should be positioned on the speed/quality Pareto frontier relative to at least one or two such baselines on the same prompt set and hardware; without that, the claimed advantage of VGDFR over the general idea of temporal token reduction is not demonstrated.
minor comments (4)
  1. [Abstract and Introduction] The abstract contains a typo: 'tailered' should be 'tailored'; a full proofread of the introduction and method sections is needed.
  2. [Equation (2)] Equation (2) should define the time convention explicitly: the text uses both normalized time and total step count T, and the subscript of f_theta(X_t,t) is inconsistent with the ODE in Eq. (1).
  3. [Steps 2 and 4] Please specify which upsampling/downsampling layers are removed from the VAE encoder and decoder, and report the resulting low-resolution shape, so that the compression module is reproducible.
  4. [Section 4.2] The qualitative claim that 'these videos fully comply with the prompt in terms of content' should be supported by a user study or by quantitative prompt-alignment numbers, since the example of the speaker's gender changing suggests non-negligible semantic drift.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the speedup and quality claims are measured outcomes, and the only same-author citation is background rather than load-bearing.

full rationale

The paper's central results are empirical measurements rather than derivations. The claimed speedup (up to 3x, Table 1) is a wall-clock latency comparison between the original HunyuanVideo and VGDFR at different values of k and theta; no equation is fitted to those latency or quality numbers. The compression pipeline described by Eqs. (2)-(6) is an algorithmic construction with an explicit assumption in Eq. (6) that the renoised latent lies on the flow-matching trajectory. An unsupported assumption is a correctness or robustness risk, not a circular reduction, because Eq. (6) is not defined in terms of the quality metrics it is later used to support. The same-author citation [39] appears in background discussion about temporal information density, but the paper independently motivates feasibility with its own HunyuanVideo-VAE experiment and reports external benchmark numbers; no load-bearing claim is justified solely by that self-citation. The DyRoPE configuration is an empirical layer-wise preference analysis, not a preselected outcome forced by a prior result. Therefore, no step in the derivation chain reduces by construction to its own inputs.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method depends on two swept hyperparameters (theta, k), a hand-selected RoPE layer list, and several unproven distributional assumptions about the flow-matching trajectory, VAE latent support, and decoded-frame similarity. There are no new physical or conceptual entities introduced, but the renoise step is the most fragile assumption because it asserts that a nonlinear compression operation preserves the learned latent distribution.

free parameters (3)
  • similarity threshold theta = 0.5 to 0.9 (0.5 for headline 2.92x)
    Controls which frame segments are merged via Eq. (4). Swept in Table 1 rather than derived from any principle.
  • starting compression timestep k = 5, 10, 15 (5 for headline)
    Determines when dynamic frame rate compression begins. Swept in Table 1 and set based on the heuristic that early compression destroys structure.
  • Global-RoPE layer list = [4, 19, 23, 31, 35, 36, 37, 40]
    Hand-selected layers where Global-RoPE is applied. The promised preference analysis is not shown, so the selection is effectively a free choice.
assumptions (5)
  • domain assumption One-step denoised preview X_T_pre from Eq. (2) contains enough content information to identify low-motion segments
    The whole scheduler relies on the preview decoded from Eq. (2) at an early timestep k. If the preview is still noise-dominated, the similarity measure in Eq. (4) may not reflect true motion.
  • ad hoc to paper Renoised compressed latent X_k_dy in Eq. (6) lies on the same flow-matching trajectory as the original ODE path, so the pretrained DiT can denoise it
    No proof is given that nonlinear VAE encode/decode plus renoise keeps the latent inside the training distribution. The Sec. 4.3 ablation says VAE weights were not exposed to noisy latents.
  • domain assumption Pairwise frame similarity in low-resolution decoded video space is a valid proxy for compressibility in latent space
    Eq. (4) merges segments when all pairwise similarities exceed theta. This assumes decoded low-res similarity correlates with latent redundancy and semantic importance.
  • ad hoc to paper The layer-wise split between Global-RoPE and Local-RoPE is known and beneficial
    The split [4, 19, 23, 31, 35, 36, 37, 40] is presented without the promised preference analysis or layer-wise ablations.
  • domain assumption External frame interpolation network restores merged frames without semantic drift
    RIFE is used in Eq. (7). The final video quality depends on its ability to synthesize intermediate frames after merging.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VGDFR: Diffusion-based Video Generation with Dynamic Latent Frame Rate." pith.science (2026). https://pith.science/paper/WQLXHHEP

@misc{pith2026250412259,
  author       = {Pith},
  title        = {Pith review of: VGDFR: Diffusion-based Video Generation with Dynamic Latent Frame Rate},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WQLXHHEP}},
  note         = {Machine review of arXiv:2504.12259}
}
read the original abstract

Diffusion Transformer(DiT)-based generation models have achieved remarkable success in video generation. However, their inherent computational demands pose significant efficiency challenges. In this paper, we exploit the inherent temporal non-uniformity of real-world videos and observe that videos exhibit dynamic information density, with high-motion segments demanding greater detail preservation than static scenes. Inspired by this temporal non-uniformity, we propose VGDFR, a training-free approach for Diffusion-based Video Generation with Dynamic Latent Frame Rate. VGDFR adaptively adjusts the number of elements in latent space based on the motion frequency of the latent space content, using fewer tokens for low-frequency segments while preserving detail in high-frequency segments. Specifically, our key contributions are: (1) A dynamic frame rate scheduler for DiT video generation that adaptively assigns frame rates for video segments. (2) A novel latent-space frame merging method to align latent representations with their denoised counterparts before merging those redundant in low-resolution space. (3) A preference analysis of Rotary Positional Embeddings (RoPE) across DiT layers, informing a tailored RoPE strategy optimized for semantic and local information capture. Experiments show that VGDFR can achieve a speedup up to 3x for video generation with minimal quality degradation.

Figures

Figures reproduced from arXiv: 2504.12259 by the authors.

Figure 1
Figure 1. VGDFR: A training-free approach that accelerates video generation by dynamically determines the frame rate based on content already generated in denoising steps. By introducing VGDFR Compression Module, we apply various compression ra￾tio of different time segments in order to reduce tokens for de￾creasing the computation cost of following DiT inference. eration computationally expensive and inefficient in terms of … view at source ↗
Figure 2
Figure 2. VGDFR Compression Module dynamically adjusts the frame rate for different time segments in a video to reduce the number of [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualization of Text-to-Video Generation on VBench Prompt Set. Three rows of the figure displays frames extracted from the start, middle, and latter of the generated videos. The leftmost column shows videos generated by the original Hunyuan Video model, while the subsequent columns present videos generated by VGDFR under different parameter configurations. Here, θ denotes the threshold, and k represents the denoisi… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Contribution of components. Left column: Video generated by VGDFR. Middle two column: Video generated by VGDFR with Compression Module completely removed. Right column: Video generated by omitting denoising and renoising op￾erations in the VGDFR Compression Module. we …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Temporal Concentration from Rollout Errors: Implicit Preference Optimization for Text-to-Video Diffusion

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Implicit DPO pairs from real-vs-reconstruction rollouts plus concentration on high latent-error temporal windows improve video authenticity and coherence without annotations or reward models.

Reference graph

Works this paper leans on

44 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2

    arXiv 2023

  2. [2]

    Token merging for fast sta- ble diffusion

    Daniel Bolya and Judy Hoffman. Token merging for fast sta- ble diffusion. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4599–4603,

  3. [3]

    To- ken merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. To- ken merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022. 2

    arXiv 2022

  4. [4]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, et al. Video generation models as world simulators. 2024. URL https://openai. com/research/video- generation-models-as-world-simulators, 3:1, 2024. 1

    work page 2024

  5. [5]

    Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. 2

    arXiv

  6. [6]

    Deep compression autoencoder for efficient high-resolution diffu- sion models

    Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang, Haotian Tang, Muyang Li, Yao Lu, and Song Han. Deep compression autoencoder for efficient high-resolution diffu- sion models. arXiv preprint arXiv:2410.10733, 2024. 2

    arXiv 2024

  7. [7]

    Structure and content-guided video synthesis with diffusion models

    Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7346–7356, 2023. 5

    work page 2023

  8. [8]

    Emu video: Factorizing text-to-video generation by explicit image con- ditioning (2023)

    R Girdhar, M Singh, A Brown, Q Duval, S Azadi, SS Ramb- hatla, A Shah, X Yin, D Parikh, and I Misra. Emu video: Factorizing text-to-video generation by explicit image con- ditioning (2023). arXiv preprint arXiv:2311.10709. 1

    arXiv 2023

Show all 44 references
  1. [9]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2014. 2

    2014

  2. [10]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 1

    2020

  3. [11]

    Real-time intermediate flow estimation for video frame interpolation

    Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In Proceedings of the European Conference on Computer Vision (ECCV), 2022. 5

    2022

  4. [12]

    Vbench: Comprehensive bench- mark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive bench- mark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2024

  5. [13]

    Scale-adaptive feature aggregation for efficient space-time video super-resolution

    Zhewei Huang, Ailin Huang, Xiaotao Hu, Chen Hu, Jun Xu, and Shuchang Zhou. Scale-adaptive feature aggregation for efficient space-time video super-resolution. In Winter Con- ference on Applications of Computer Vision (WACV), 2024. 5

    2024

  6. [14]

    Adaptive caching for faster video generation with diffu- sion transformers

    Kumara Kahatapitiya, Haozhe Liu, Sen He, Ding Liu, Menglin Jia, Chenyang Zhang, Michael S Ryoo, and Tian Xie. Adaptive caching for faster video generation with diffu- sion transformers. arXiv preprint arXiv:2411.02397, 2024. 3

    2024 arXiv

  7. [15]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 4401–4410, 2019. 2

    2019

  8. [16]

    Auto-encoding vari- ational bayes, 2013

    Diederik P Kingma, Max Welling, et al. Auto-encoding vari- ational bayes, 2013. 2

    2013

  9. [17]

    Hunyuanvideo: A systematic framework for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. 1, 2, 5

    2024 arXiv

  10. [18]

    Black Forest Labs. Flux. https://github.com/ black-forest-labs/flux, 2024. 2

    2024

  11. [19]

    Vidtome: Video token merging for zero-shot video editing

    Xirui Li, Chao Ma, Xiaokang Yang, and Ming-Hsuan Yang. Vidtome: Video token merging for zero-shot video editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7486–7495, 2024. 2

    2024

  12. [20]

    Open-sora plan: Open-source large video generation model

    Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Li- uhan Chen, et al. Open-sora plan: Open-source large video generation model. arXiv preprint arXiv:2412.00131, 2024. 2

    2024 arXiv

  13. [21]

    Sora: A review on background, technology, limitations, and opportunities of large vision models

    Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jian- feng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177, 2024. 1

    2024 arXiv

  14. [22]

    A study of subjective video quality at various frame rates

    Alex Mackin, Fan Zhang, and David R Bull. A study of subjective video quality at various frame rates. In2015 IEEE International Conference on Image Processing (ICIP), pages 3407–3411. IEEE, 2015. 3

    2015

  15. [23]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 4195–4205,

  16. [24]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 1

    2023 arXiv

  17. [25]

    Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petro- vic, and Yuming Du

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, Kiran Jagadeesh, Kunpeng Li, L...

    2025

  18. [26]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 2, 3

    2022

  19. [27]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  20. [28]

    Mediconfusion: Can you trust your ai radiologist? probing the reliability of multimodal medical foundation models

    Mohammad Shahab Sepehri, Zalan Fabian, Maryam Soltanolkotabi, and Mahdi Soltanolkotabi. Mediconfusion: Can you trust your ai radiologist? probing the reliability of multimodal medical foundation models. arXiv preprint arXiv:2409.15477, 2024. 1

    2024 arXiv

  21. [29]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International confer- ence on machine learning, pages 2256–2265. pmlr, 2015. 1

    2015

  22. [30]

    Rate control for low-bit-rate video via variable-encoding frame rates

    Hwangjun Song and C-CJ Kuo. Rate control for low-bit-rate video via variable-encoding frame rates. IEEE Transactions on Circuits and Systems for Video Technology , 11(4):512– 521, 2001. 3

    2001

  23. [31]

    Asymrnr: Video diffusion transformers ac- celeration with asymmetric reduction and restoration

    Wenhao Sun, Rong-Cheng Tu, Jingyi Liao, Zhao Jin, and Dacheng Tao. Asymrnr: Video diffusion transformers ac- celeration with asymmetric reduction and restoration. arXiv preprint arXiv:2412.11706, 2024. 3

    2024 arXiv

  24. [32]

    Mocogan: Decomposing motion and content for video generation

    Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526–1535,

  25. [33]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 1

    2017

  26. [34]

    Omnitokenizer: A joint image- video tokenizer for visual generation

    Junke Wang, Yi Jiang, Zehuan Yuan, Bingyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint image- video tokenizer for visual generation. Advances in Neural Information Processing Systems, 37:28281–28295, 2025. 2

    2025

  27. [35]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si- moncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 6

    2004

  28. [36]

    Godiva: Gen- erating open-domain videos from natural descriptions

    Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Gen- erating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.14806, 2021. 5

    2021 arXiv

  29. [37]

    Fast- vqa: Efficient end-to-end video quality assessment with frag- ment sampling

    Haoning Wu, Chaofeng Chen, Jingwen Hou, Liang Liao, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Fast- vqa: Efficient end-to-end video quality assessment with frag- ment sampling. In Proceedings of European Conference of Computer Vision (ECCV), 2022. 5

    2022

  30. [38]

    Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity

    Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, et al. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity. arXiv preprint arXiv:2502.01776, 2025. 5

    2025 arXiv

  31. [39]

    Dlfr-vae: Dynamic latent frame rate vae for video generation

    Zhihang Yuan, Siyuan Wang, Rui Xie, Hanling Zhang, Tongcheng Fang, Yuzhang Shang, Shengen Yan, Guohao Dai, and Yu Wang. Dlfr-vae: Dynamic latent frame rate vae for video generation. arXiv preprint arXiv:2502.11897,

    arXiv

  32. [40]

    Training-free and hardware-friendly acceleration for diffu- sion models via similarity-based token pruning

    Evelyn Zhang, Jiayi Tang, Xuefei Ning, and Linfeng Zhang. Training-free and hardware-friendly acceleration for diffu- sion models via similarity-based token pruning. 2

  33. [41]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 6

    2018

  34. [42]

    Cross- attention makes inference cumbersome in text-to-image dif- fusion models

    Wentian Zhang, Haozhe Liu, Jinheng Xie, Francesco Fac- cio, Mike Zheng Shou, and J ¨urgen Schmidhuber. Cross- attention makes inference cumbersome in text-to-image dif- fusion models. arXiv e-prints, pages arXiv–2404, 2024. 3

    2024

  35. [43]

    Real-time video generation with pyramid attention broad- cast

    Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. Real-time video generation with pyramid attention broad- cast. arXiv preprint arXiv:2408.12588, 2024. 3

    2024 arXiv

  36. [44]

    Open-sora: Democratizing efficient video production for all

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:2412.20404, 2024. 1, 2

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.