Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Video diffusion transformers can run up to 1.85x faster by replacing dense attention with fixed sparse patterns, no retraining required.

desk verdict A useful, incremental acceleration for video DiTs, but the offline pattern-transfer claim is under-evidenced and the pseudocode/objective conflict needs fixing. read the letter →

arxiv 2506.03065 v1 pith:XLHAIQ4W submitted 2025-06-03 cs.CV cs.AIcs.LG

Pengtao Chen , Xianfang Zeng , Maosen Zhao , Peng Ye , Mingzhu Shen , Wei Cheng , Gang Yu , Tao Chen This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords videodiffusiontransformerssparseattentionstructuredsparsityheadskippingofflinesearchinferenceaccelerationpatterns
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the attention maps inside video diffusion transformers are not dense in practice: most heads settle into one of a few recurring shapes—a diagonal band, several evenly spaced diagonals, or a vertical stripe—and a small percentage of heads can be dropped entirely. Building on that observation, Sparse-vDiT assigns each head an offline-chosen sparse computation mode, fuses heads that share a mode, and replaces dense attention with pattern-specific kernels. On CogVideoX1.5, HunyuanVideo, and Wan2.1 the authors report 2.09x, 2.38x, and 1.67x theoretical FLOP reductions and 1.76x, 1.85x, and 1.58x end-to-end speedups while keeping output close to the original model, with PSNR values of 24.13, 27.09, and 22.59. If the pattern-stability claim holds, production video generators can run roughly 40 to 45 percent faster with no retraining and with little visible change in quality.

What carries the argument

The central mechanism is a per-head attention mode drawn from the fixed set {full attention, skip, diagonal, multi-diagonal, vertical-stripe}, selected by an offline sparse diffusion search and executed by pattern-optimized kernels. During the search, each candidate mode's output is compared with full attention by mean squared error plus a sparsity penalty $\lambda(1-S_i)$; if every sparse candidate exceeds a threshold $\epsilon$, the head keeps full attention. After the search, heads in the same layer that share the same mode are fused into a single operation, reducing kernel-launch and memory overhead. The multi-diagonal pattern is handled by a token rearrangement that converts it into a diagonal structure usable by a window-attention kernel.

What would settle it

Run the offline search on one prompt distribution, then evaluate on a diverse set of out-of-distribution prompts while tracking per-head attention-pattern agreement across denoising steps; if heads switch pattern class on unseen prompts or at later steps (for example, a searched diagonal head becomes full-attention-like at step 30), the frozen configuration either wastes compute or drops fidelity, contradicting the input-invariance claim.

Watch

Extended reading notes

Core claim

The core discovery is that the sparsity structure of vDiT attention is largely determined by architectural position rather than by the video or text content. Across prompts, the same layer and head index tends to show the same pattern: some heads follow a diagonal pattern capturing within-frame structure, some follow a multi-diagonal pattern capturing cross-frame consistency, some form vertical stripes attending to global tokens, and a few heads contribute so little that zeroing their output costs almost nothing. Because the patterns are fixed and input-invariant, the expensive configuration step can be done once on a small prompt set instead of at every denoising step. Sparse-vDiT operationalizes this by scoring five candidate modes per head—full attention, skip, and three sparse kernels—with an MSE-plus-sparsity-penalty loss relative to the full-attention output, then freezing a per-layer, per-head configuration for deployment. As reported, this configuration matches or beats existing sparse-attention baselines on fidelity while delivering larger speedups on all three tested models.

Load-bearing premise

The load-bearing premise is that the sparsity patterns and the fixed sparsity levels chosen offline remain correct for unseen prompts at every denoising step, so the frozen configuration never needs to adapt at inference time.

Editorial extensions

If this is right

  • Production deployment becomes a one-time per-model search: after the offline configuration is fixed, inference needs no runtime pattern decisions and no extra memory for attention-map analysis.
  • Head fusion converts part of the theoretical FLOP reduction into real end-to-end speedup, so the 2.09x to 2.38x FLOP cuts translate into 1.58x to 1.85x wall-clock speedups across the three tested models.
  • The threshold $\epsilon$ acts as a quality-speedup dial: larger values raise sparsity and speed (up to 1.91x on CogVideoX1.5 in the ablation) at a measurable fidelity cost, letting practitioners choose an operating point.
  • The method composes with quantization: the appendix shows that adding FP8 on Wan2.1 lifts the end-to-end speedup to 1.78x with a VBench drop below 0.5 percent.
  • Because the approach is training-free and was validated on both an MM-DiT architecture and a Self-Attn plus Cross-Attn architecture, it is positioned as a general accelerator for current vDiT families.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the patterns are truly input-invariant, a configuration searched at one resolution or frame count may transfer to other settings with minimal re-validation, which would make per-deployment search unnecessary in practice.
  • Beyond the paper: the MSE-based mode selection is a proxy that could mis-rank modes at intermediate denoising steps; a per-timestep stability test would either confirm the fixed-config assumption or expose a need for lightweight adaptation.
  • Beyond the paper: the diagonal and multi-diagonal classes both say that frame tokens mostly talk to same-frame and nearby-frame tokens, which suggests cache-friendly or tile-based implementations beyond the reported window kernels could push speedups further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Sparse-vDiT, a training-free framework to accelerate video diffusion transformers (vDiTs) by exploiting structured sparsity in attention. It identifies three recurring sparse attention patterns (diagonal, multi-diagonal, vertical-stripe) and a small fraction of skippable heads, and argues these patterns are largely input-invariant and depend mainly on layer depth and head position. A hardware-aware offline search assigns one of five computation modes (full, skip, three sparse kernels) to each layer-head pair using a small search set; after search the configuration is fixed and heads sharing a mode are fused. Experiments on CogVideoX1.5, HunyuanVideo, and Wan2.1 report theoretical FLOP reductions of 2.09x, 2.38x, and 1.67x and end-to-end speedups of 1.76x, 1.85x, and 1.58x with modest fidelity loss (e.g., PSNR 24.13, 27.09, 22.59 against the original model).

Significance. If the central claim holds, Sparse-vDiT offers a practical, retraining-free speedup of roughly 40-45% for several popular open-source vDiTs, with quality close to the original. The paper has several notable strengths: it evaluates on three different vDiT families, reports both theoretical and wall-clock results, compares against strong baselines (SVG, WinAttn, MInference, PAB), provides a public code link, and its offline-fixed-configuration approach enables head fusion and custom kernel optimization. The observation of recurring, position-dependent attention patterns in vDiTs is interesting and could inform future acceleration designs. However, the paper's reproducibility and the strength of its invariance claim are currently undermined by an internal inconsistency in the search objective and an underspecified aggregation of per-timestep decisions.

major comments (4)
  1. [Sec. 4.2, Eq. (2) vs Appendix A, Algorithm 1] The sparsity penalty in the search objective is defined inconsistently. Equation (2) defines L_i = MSE(O_i - O_0) + λ(1 - S_i), which penalizes denser modes more (since higher S_i means sparser computation, 1-S_i is smaller). Algorithm 1 instead computes loss = MSE(x_i^t, x_gt^t) + λ S_i, which penalizes sparser modes more. Because S_i=1 for skip-head and higher S_i indicates more sparsity, the two formulations favor opposite configurations. The manuscript does not state which objective was actually used to produce the reported tables. This ambiguity directly affects reproducibility of the searched configurations and, consequently, the speed/quality numbers.
  2. [Algorithm 1 and Sec. 4.2 (Offline Sparse Diffusion Search)] Algorithm 1 appends a decision f for every head at every timestep and layer, returning a list whose length grows with T·N·H. In contrast, the text states that 'once the search is completed, the sparse modes for the entire inference process are fixed.' The algorithm does not describe how per-timestep decisions are aggregated into a single global configuration (e.g., majority vote, first-timestep decision, or a separate selection step). Without this aggregation rule, the method is underspecified and the reader cannot reproduce the reported fixed configuration.
  3. [Sec. 4.1.3, Fig. 4] The load-bearing premise that attention patterns are input-invariant across prompts and across denoising timesteps is supported only by a qualitative t-SNE visualization on 50 VBench prompts. The figure shows clustering along the layer dimension but does not quantify per-head pattern stability, prompt-to-prompt variation, or timestep-to-timestep variation. Since the final configuration is fixed offline and applied to all inference steps, the paper should report a quantitative stability measure, for example the fraction of heads whose selected mode is unchanged across a held-out set of prompts or across timesteps, or the distribution of per-timestep decisions. The paper's own Limitation (Sec. 6) admits that the predefined sparsity may under- or over-sparsify, which further underscores the need for quantitative evidence of invariance.
  4. [Sec. 4.2, Sec. 5.2] The offline search uses the MSE to full attention as its selection criterion, and the final quality is measured on held-out prompts, so the evaluation is not fully circular. However, the paper does not specify the size, diversity, or selection procedure of the search set, nor does it report sensitivity of the final configuration to the search set. Given that the claim is that 'a small number of samples' suffices, the authors should provide the number of prompts used for search, the number of timesteps sampled, and ideally repeat the search on different random subsets and report the resulting variance in speedup and fidelity. Without this, it is unclear how robust the reported numbers are to the choice of search samples.
minor comments (5)
  1. [Appendix A, Algorithm 1] There are typos in the pseudocode: 'Pretraine' should be 'Pretrained' and 'accoding' should be 'according.'
  2. [Table 2 and Figs. 7-9] The table header appears to have overlapping or duplicated metric columns ('Against Original' and then separate SSIM/PSNR/LPIPS/ImageQual/SubConsist columns); please clarify the exact column layout. Additionally, in Figures 7-9, 'Temperal' should be 'Temporal.'
  3. [Fig. 6 caption] The caption says 'The green box indicates the ground truth,' but the green-boxed image appears to be the output of the pretrained (original) model, not a ground-truth video. Please clarify the terminology to avoid confusion between original model outputs and ground-truth content.
  4. [Sec. 4.1.2, Table 1] The head-skipping results are said to be based on a 'minimum mean squared error (MSE) criterion,' but the criterion that ranks heads for skipping is not defined. Please specify how the heads to be skipped are selected (e.g., by per-head output sensitivity, by attention-map energy, or by another importance measure).
  5. [Throughout] The manuscript uses 'FLOPs' and 'FLOPS' interchangeably; in Table 2, 'PFLOPS(↓)' appears to denote total floating-point operations, which should be stated consistently (e.g., 'PFLOPS' as total FLOPs and 'FLOPS' as rate).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the offline search is a fit, but reported quality and speedups are measured on held-out inputs and are not forced by construction.

full rationale

The paper's derivation chain is self-contained and does not reduce to its own inputs. The offline sparse diffusion search in Section 4.2 selects per-head attention modes by minimizing MSE to full-attention outputs on a small search set, but the reported fidelity metrics (PSNR, SSIM, LPIPS) and VBench scores in Table 2 are computed on held-out evaluation prompts against the pretrained model, so the quality numbers are not defined by the search criterion. The FLOP reductions are arithmetic consequences of the fixed predefined sparsity ratios and head-skipping choices, while the end-to-end speedups are measured latencies. The self-citations (Delta-DiT, MD-DiT, DiTFastAttnV2, 4-bit FP quantization) appear only in related work or as prior context and are not load-bearing. The paper's own Limitation explicitly acknowledges that the predefined sparsity may under- or over-sparsify on unseen inputs, which is a generalization risk rather than a circular dependence. One internal inconsistency exists: Eq. (2) defines the loss as MSE + λ(1 - S_i), while Algorithm 1 writes loss = MSE + λS_i; this is a correctness/reproducibility concern about the search objective's sign, not a circularity. Overall, no step in the claimed acceleration derivation is equivalent to its inputs by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework rests on three kinds of assumptions: input-invariant attention patterns, MSE as a quality proxy, and predefined kernel sparsity levels that match real attention. Free parameters are the trade-off weights lambda and epsilon and the kernel sparsity ratios S_2 through S_4. There are no invented physical entities.

free parameters (3)
  • lambda (sparsity penalty weight) = 0.5 (default)
    Tuned via ablation on CogVideoX1.5 VBench in Section 5.3; balances MSE fidelity against sparsity preference and affects which per-head mode is selected.
  • epsilon (sparsity threshold) = 1 (default)
    Tuned via ablation on CogVideoX1.5 in Section 5.3; directly controls how many heads are assigned a sparse mode, and therefore the final speedup and quality.
  • Per-pattern sparsity ratios S_2, S_3, S_4 = Not reported
    Predefined kernel sparsity levels for the three sparse patterns (Section 4.2). Values are fixed constants but not stated in the paper; they directly determine theoretical FLOP reduction.
assumptions (4)
  • domain assumption Attention maps of pretrained vDiTs exhibit input-invariant structured sparsity patterns.
    Core premise introduced in Section 4.1.3 and supported only by t-SNE clustering on 50 VBench prompts. If prompts or denoising steps break the pattern, the offline fixed configuration fails.
  • domain assumption MSE between sparse and full attention hidden states is a good proxy for final generation quality.
    Used as the selection loss in Eq. 2 and Algorithm 1. The paper does not establish a quantitative link between this per-layer MSE and downstream PSNR, SSIM, or VBench scores.
  • domain assumption A configuration selected via per-timestep decisions can be frozen into one global config for all inference steps.
    Algorithm 1 accumulates decisions across denoising timesteps but the paper states the final config is fixed. The aggregation mechanism is not specified, and no experiment validates that a single config works across all timesteps.
  • domain assumption The predefined diagonal, multi-diagonal, and vertical-stripe kernels cover the actual attention structure.
    Stated in Section 4.2 and acknowledged as a limitation in Section 6: if the predefined sparsity level does not match real attention maps, under- or over-sparsification occurs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers." pith.science (2026). https://pith.science/paper/XLHAIQ4W

@misc{pith2026250603065,
  author       = {Pith},
  title        = {Pith review of: Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XLHAIQ4W}},
  note         = {Machine review of arXiv:2506.03065}
}
abstract

While Diffusion Transformers (DiTs) have achieved breakthroughs in video generation, this long sequence generation task remains constrained by the quadratic complexity of attention mechanisms, resulting in significant inference latency. Through detailed analysis of attention maps in Video Diffusion Transformer (vDiT), we identify three recurring sparsity patterns: diagonal, multi-diagonal, and vertical-stripe structures. And even 3-6\% attention heads can be skipped. Crucially, these patterns exhibit strong layer-depth and head-position correlations but show limited dependence on the input content. Leveraging these findings, we propose Sparse-vDiT, a sparsity acceleration framework for vDiT comprising: 1) Pattern-optimized sparse kernels that replace dense attention with computationally efficient implementations for each identified sparsity pattern. 2) An offline sparse diffusion search algorithm that selects the optimal sparse computation strategy per layer and head via hardware-aware cost modeling. After determining the optimal configuration, we fuse heads within the same layer that share the same attention strategy, enhancing inference efficiency. Integrated into state-of-the-art vDiT models (CogVideoX1.5, HunyuanVideo, and Wan2.1), Sparse-vDiT achieves 2.09$\times$, 2.38$\times$, and 1.67$\times$ theoretical FLOP reduction, and actual inference speedups of 1.76$\times$, 1.85$\times$, and 1.58$\times$, respectively, while maintaining high visual fidelity, with PSNR values reaching 24.13, 27.09, and 22.59. Our work demonstrates that latent structural sparsity in vDiTs can be systematically exploited for long video synthesis.

Figures

Figures reproduced from arXiv: 2506.03065 by the authors.

Figure 1
Figure 1. The architecture of vDiT and inference latency analysis of its two variants, CogVideoX1.5 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Visualization of the vDiT attention map showing four interaction regions. The dominant [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the four recurring attention patterns in vDiT. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: t-SNE visualization of attention patterns along the head dimension on a VBench subset, [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Overview of the Sparse-vDiT. We first predefine five types of attention mode M0:4. Then, using an offline sparse diffusion search algorithm, we select the best attention mode for each layer and head in vDiT. After the search, for heads set to skip attention, we set the…
Figure 6
Figure 6. Figure 6: Visual comparison between the proposed Sparse-vDiT and the baseline method. The green [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: More visual comparison between the proposed Sparse-vDiT and the baseline method. Our [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: More visual comparison between the proposed Sparse-vDiT and the baseline method. Our [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: More visual comparison between the proposed Sparse-vDiT and the baseline method. Our [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: More visual comparison between the proposed Sparse-vDiT and the pretrain model. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: More visual comparison between the proposed Sparse-vDiT and the pretrain model. [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformers

    cs.CV 2026-01 conditional novelty 6.0 of 10

    MOD-DiT accelerates video DiTs by extrapolating fitted attention-pattern intensities across denoising steps to build dynamic block masks, achieving 1.8–2.3× speedups on CogVideoX, HunyuanVideo, and Wan2.1.

  2. Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers

    cs.CV 2025-12 conditional novelty 6.0 of 10

    LLSA uses hierarchical top-K selection plus coarse-token enrichment to make diffusion-transformer attention O(N log N), giving about 6x faster training at 256x256 pixel resolution with FID comparable to full attention.

  3. Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformer

    cs.CV 2025-11 conditional novelty 6.0 of 10

    Expressing all RoPE positions on the query's grid ('one attention, one scale') plus a small boundary content-exchange step restores mixed-resolution diffusion generation that naive position interpolation destroys.

  4. SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference

    cs.CV 2026-08 conditional novelty 5.0 of 10

    SPADE combines static, semi-static, and dynamic block-sparse attention with a cheap SICS-based blocking heuristic to speed up video diffusion inference by up to 1.80x end-to-end.

Reference graph

Works this paper leans on

51 extracted references · 8 canonical work pages · cited by 4 Pith papers

  1. [1]

    Longformer: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

    arXiv 2004

  2. [2]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  3. [3]

    Ld-pruner: Ef- ficient pruning of latent diffusion models using task-agnostic insights

    Thibault Castells, Hyoung-Kyu Song, Bo-Kyeong Kim, and Shinkook Choi. Ld-pruner: Ef- ficient pruning of latent diffusion models using task-agnostic insights. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 821–830, 2024

    2024

  4. [4]

    Delta-dit: A training-free acceleration method tailored for diffusion transformers.arXiv preprint arXiv:2406.01125, 2024

    Pengtao Chen, Mingzhu Shen, Peng Ye, Jianjian Cao, Chongjun Tu, Christos-Savvas Bouganis, Yiren Zhao, and Tao Chen. Delta-dit: A training-free acceleration method tailored for diffusion transformers.arXiv preprint arXiv:2406.01125, 2024

    arXiv 2024

  5. [5]

    Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019

    arXiv 1904

  6. [6]

    Efficient-vdit: Efficient video diffusion transformers with attention tile.arXiv preprint arXiv:2502.06155, 2025

    Hangliang Ding, Dacheng Li, Runlong Su, Peiyuan Zhang, Zhijie Deng, Ion Stoica, and Hao Zhang. Efficient-vdit: Efficient video diffusion transformers with attention tile.arXiv preprint arXiv:2502.06155, 2025

    arXiv 2025

  7. [7]

    Scaling rectified flow trans- formers for high-resolution image synthesis, 2024.URL https://arxiv

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis, 2024.URL https://arxiv. org/abs/2403.03206, 2

    arXiv 2024

  8. [8]

    Structural pruning for diffusion models

    Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY , USA, 2023. Curran Associates Inc

    work page 2023

Show all 51 references
  1. [9]

    Neighborhood attention transformer

    Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6185–6194, 2023

    2023

  2. [10]

    Pre-trained video generative models as world simulators.arXiv preprint arXiv:2502.07825, 2025

    Haoran He, Yang Zhang, Liang Lin, Zhongwen Xu, and Ling Pan. Pre-trained video generative models as world simulators.arXiv preprint arXiv:2502.07825, 2025

    2025 arXiv

  3. [11]

    Animate-a-story: Storytelling with retrieval-augmented video generation.arXiv preprint arXiv:2307.06940, 2023

    Yingqing He, Menghan Xia, Haoxin Chen, Xiaodong Cun, Yuan Gong, Jinbo Xing, Yong Zhang, Xintao Wang, Chao Weng, Ying Shan, et al. Animate-a-story: Storytelling with retrieval-augmented video generation.arXiv preprint arXiv:2307.06940, 2023

    2023 arXiv

  4. [12]

    Animate anyone: Consistent and controllable image-to-video synthesis for character animation

    Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8153–8163, 2024. 10

    2024

  5. [13]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  6. [14]

    Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention.Advances in Neural Information Processing Systems, 37:52481–52515, 2024

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention.Advances in Neural Information Processing S...

    2024

  7. [15]

    Adaptive caching for faster video generation with diffusion transformers

    Kumara Kahatapitiya, Haozhe Liu, Sen He, Ding Liu, Menglin Jia, Chenyang Zhang, Michael S Ryoo, and Tian Xie. Adaptive caching for faster video generation with diffusion transformers. arXiv preprint arXiv:2411.02397, 2024

    2024 arXiv

  8. [16]

    Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  9. [17]

    Flexprefill: A context- aware sparse attention mechanism for efficient long-sequence inference.arXiv preprint arXiv:2502.20766, 2025

    Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. Flexprefill: A context- aware sparse attention mechanism for efficient long-sequence inference.arXiv preprint arXiv:2502.20766, 2025

    2025 arXiv

  10. [18]

    Open-sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024

    Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024

    2024 arXiv

  11. [19]

    Diffusion adversarial post-training for one-step video generation.arXiv preprint arXiv:2501.08316, 2025

    Shanchuan Lin, Xin Xia, Yuxi Ren, Ceyuan Yang, Xuefeng Xiao, and Lu Jiang. Diffusion adversarial post-training for one-step video generation.arXiv preprint arXiv:2501.08316, 2025

    2025

  12. [20]

    Timestep embedding tells: It’s time to cache for video diffusion model.arXiv preprint arXiv:2411.19108, 2024

    Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, and Fang Wan. Timestep embedding tells: It’s time to cache for video diffusion model.arXiv preprint arXiv:2411.19108, 2024

    2024 arXiv

  13. [21]

    Clear: Conv-like linearization revs pre-trained diffusion transformers up.arXiv preprint arXiv:2412.16112, 2024

    Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Clear: Conv-like linearization revs pre-trained diffusion transformers up.arXiv preprint arXiv:2412.16112, 2024

    2024 arXiv

  14. [22]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InProceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021

    2021

  15. [23]

    Fastercache: Training-free video diffusion model acceleration with high quality.arXiv preprint arXiv:2410.19355, 2024

    Zhengyao Lv, Chenyang Si, Junhao Song, Zhenyu Yang, Yu Qiao, Ziwei Liu, and Kwan-Yee K Wong. Fastercache: Training-free video diffusion model acceleration with high quality.arXiv preprint arXiv:2410.19355, 2024

    2024 arXiv

  16. [24]

    Deepcache: Accelerating diffusion models for free

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15762–15772, 2024

    2024

  17. [25]

    Towards world simulator: Crafting physical commonsense- based benchmark for video generation.arXiv preprint arXiv:2410.05363, 2024

    Fanqing Meng, Jiaqi Liao, Xinyu Tan, Wenqi Shao, Quanfeng Lu, Kaipeng Zhang, Yu Cheng, Dianqi Li, Yu Qiao, and Ping Luo. Towards world simulator: Crafting physical commonsense- based benchmark for video generation.arXiv preprint arXiv:2410.05363, 2024

    arXiv

  18. [26]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  19. [27]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  20. [28]

    Post-training quantization on diffusion models

    Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1972–1981, 2023. 11

    1972

  21. [29]

    MD-dit: Step-aware mixture-of-depths for efficient diffusion transformers

    Mingzhu Shen, Pengtao Chen, Peng Ye, Guoxuan Xia, Tao Chen, Christos-Savvas Bouganis, and Yiren Zhao. MD-dit: Step-aware mixture-of-depths for efficient diffusion transformers. In Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning, 2024

    2024

  22. [30]

    Reducio! generating 1024 times1024 video within 16 seconds using extremely compressed motion latents.arXiv preprint arXiv:2411.13552, 2024

    Rui Tian, Qi Dai, Jianmin Bao, Kai Qiu, Yifan Yang, Chong Luo, Zuxuan Wu, and Yu- Gang Jiang. Reducio! generating 1024 times1024 video within 16 seconds using extremely compressed motion latents.arXiv preprint arXiv:2411.13552, 2024

    2024 arXiv

  23. [31]

    Triton: an intermediate language and compiler for tiled neural network computations

    Philippe Tillet, Hsiang-Tsung Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. InProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10–19, 2019

    2019

  24. [32]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  25. [33]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  26. [34]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    2025 arXiv

  27. [35]

    Taming rectified flow for inversion and editing.arXiv preprint arXiv:2411.04746, 2024

    Jiangshan Wang, Junfu Pu, Zhongang Qi, Jiayi Guo, Yue Ma, Nisha Huang, Yuxin Chen, Xiu Li, and Ying Shan. Taming rectified flow for inversion and editing.arXiv preprint arXiv:2411.04746, 2024

    2024 arXiv

  28. [36]

    A universal image quality index.IEEE signal processing letters, 9(3):81–84, 2002

    Zhou Wang and Alan C Bovik. A universal image quality index.IEEE signal processing letters, 9(3):81–84, 2002

    2002

  29. [37]

    Ptq4dit: Post-training quantization for diffusion transformers.arXiv preprint arXiv:2405.16005, 2024

    Junyi Wu, Haoxuan Wang, Yuzhang Shang, Mubarak Shah, and Yan Yan. Ptq4dit: Post-training quantization for diffusion transformers.arXiv preprint arXiv:2405.16005, 2024

    2024 arXiv

  30. [38]

    Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity.arXiv preprint arXiv:2502.01776, 2025

    Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, et al. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity.arXiv preprint arXiv:2502.01776, 2025

    2025 arXiv

  31. [39]

    Duoattention: Efficient long-context llm inference with retrieval and streaming heads.arXiv preprint arXiv:2410.10819, 2024

    Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context llm inference with retrieval and streaming heads.arXiv preprint arXiv:2410.10819, 2024

    arXiv

  32. [40]

    Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453, 2023

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453, 2023

    2023 arXiv

  33. [41]

    Sana: Efficient high-resolution image synthesis with linear diffusion transformers.arXiv preprint arXiv:2410.10629, 2024

    Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffusion transformers.arXiv preprint arXiv:2410.10629, 2024

    arXiv

  34. [42]

    Dynamicrafter: Animating open-domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. InEuropean Conference on Computer Vision, pages 399–417. Springer, 2024

    2024

  35. [43]

    Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    2024 arXiv

  36. [44]

    Ditfastattn: Attention compression for diffusion transformer models.Advances in Neural Information Processing Systems, 37:1196–1219, 2024

    Zhihang Yuan, Hanling Zhang, Lu Pu, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. Ditfastattn: Attention compression for diffusion transformer models.Advances in Neural Information Processing Systems, 37:1196–1219, 2024. 12

    2024

  37. [45]

    Motion consistency model: Accelerating video diffusion with disentangled motion-appearance distillation.Advances in Neural Information Processing Systems, 37:111000–111021, 2024

    Yuanhao Zhai, Kevin Lin, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Chung-Ching Lin, David Doermann, Junsong Yuan, and Lijuan Wang. Motion consistency model: Accelerating video diffusion with disentangled motion-appearance distillation.Advances in Neural Information Processing ...

    2024

  38. [46]

    Instructvedit: A holistic approach for instructional video editing.arXiv preprint arXiv:2503.17641, 2025

    Chi Zhang, Chengjian Feng, Feng Yan, Qiming Zhang, Mingjin Zhang, Yujie Zhong, Jing Zhang, and Lin Ma. Instructvedit: A holistic approach for instructional video editing.arXiv preprint arXiv:2503.17641, 2025

    2025 arXiv

  39. [47]

    Ditfastattnv2: Head-wise attention compression for multi- modality diffusion transformers.arXiv preprint arXiv:2503.22796, 2025

    Hanling Zhang, Rundong Su, Zhihang Yuan, Pengtao Chen, Mingzhu Shen Yibo Fan, Shengen Yan, Guohao Dai, and Yu Wang. Ditfastattnv2: Head-wise attention compression for multi- modality diffusion transformers.arXiv preprint arXiv:2503.22796, 2025

    2025 arXiv

  40. [48]

    The unrea- sonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018

    2018

  41. [49]

    Pioneering 4-bit fp quantization for diffusion models: Mixup-sign quantization and timestep-aware fine-tuning, 2025

    Maosen Zhao, Pengtao Chen, Chong Yu, Yan Wen, Xudong Tan, and Tao Chen. Pioneering 4-bit fp quantization for diffusion models: Mixup-sign quantization and timestep-aware fine-tuning, 2025

    2025

  42. [50]

    Real-time video generation with pyramid attention broadcast.arXiv preprint arXiv:2408.12588, 2024

    Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. Real-time video generation with pyramid attention broadcast.arXiv preprint arXiv:2408.12588, 2024

    2024 arXiv

  43. [51]

    Dig: Scalable and efficient diffusion models with gated linear attention.arXiv preprint arXiv:2405.18428, 2024

    Lianghui Zhu, Zilong Huang, Bencheng Liao, Jun Hao Liew, Hanshu Yan, Jiashi Feng, and Xinggang Wang. Dig: Scalable and efficient diffusion models with gated linear attention.arXiv preprint arXiv:2405.18428, 2024. 13 Appendix for SPARSE-VDIT A Algorithm Implementation Figure 5 ...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.