Pith. sign in

REVIEW 3 major objections 4 minor 6 cited by

Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EFFICIENT-vDiT claims that a repetitive, input-independent "Attention Tile" pattern makes most 3D attention in video DiTs redundant, so fixed sparse masks plus consistency distillation yield 7.4x-7.8x faster 720p video generation with…

desk verdict Real speedups and a genuinely new sparsity pattern, but the 'data-independent' mask claim rests on two prompts and the quality trade-off is less marginal than the aggregate VBench score suggests. read the letter →

arxiv 2502.06155 v2 pith:4CC5WIBQ submitted 2025-02-10 cs.CV

Hangliang Ding , Dacheng Li , Runlong Su , Peiyuan Zhang , Zhijie Deng , Ion Stoica , Hao Zhang This is my paper · ORCID
classification cs.CV
keywords videodiffusiontransformers3DfullattentionTilesparsemasksmulti-stepconsistencydistillationknowledgeOpen-Sora-PlanCogX
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the expensive 3D full attention in video diffusion transformers is largely redundant, because the attention maps organize into a repetitive tile pattern in which each latent frame strongly attends to a few nearby frames and a small set of global reference frames. It claims that this "Attention Tile" structure is stable across inputs, so a fixed sparse mask chosen once per layer can replace the full attention map, cutting the cost from quadratic to linear in the number of frames. Combined with a multi-step consistency distillation that splits the sampling trajectory into segments and a final knowledge-distillation step, the pipeline turns a pretrained Open-Sora-Plan-1.2 model into one that generates 720p video 7.4x to 7.8x faster, using only 0.1% of the pretraining data, while keeping VBench final scores within about one percentage point of the base model. The practical stake is that high-quality video generation could run at much lower compute cost or higher throughput without a visible quality drop.

What carries the argument

The load-bearing object is the Attention Tile, a repetitive block structure in the 3D attention map of a video DiT in which each tile block records attention between two latent frames. The paper's sparse mask family, denoted $k{:}F{-}k$, keeps the main-diagonal tile blocks and the attention to $k$ global reference frames chosen uniformly from the $F$ latent frames, so the per-layer cost is linear in $F$. The load-bearing procedure is Algorithm 1, a layer-by-layer greedy search that, for each layer, picks the sparsest mask whose final hidden-state MSE against the teacher stays below threshold $r$, followed by knowledge distillation that matches the student's attention outputs, MLP outputs, and diffusion loss to the teacher.

What would settle it

Run the layer-wise search on 100 or more diverse prompts and measure the overlap of the top-attention positions across them; the paper's data-independence claim predicts roughly 90% overlap, so a large drop (say below 70%) would falsify the fixed-mask design. A complementary check is to apply the $r=0.400$ mask to prompts with rapid scene cuts or strong camera motion and compare CD-FVD against the reported 231.68: a large jump would show the mask is content-dependent after all.

Watch

Extended reading notes

Core claim

The central discovery is the Attention Tile: in a 3D full-attention video DiT, attention maps decompose into uniformly repeated tile blocks, one per pair of latent frames, and these tiles show three measurable properties. Diagonal tiles carry roughly 2.8x the attention weight of off-diagonal tiles; off-diagonal similarity falls off as frames get farther apart; and the positions of high attention overlap at about 90% across different prompts. From these observations the paper argues that a mask keeping only main-diagonal tiles plus attention to a constant number $k$ of uniformly chosen global reference frames preserves the essential computation while making attention complexity linear in the frame count. It then claims that a three-stage procedure -- multi-step consistency distillation, a layer-wise greedy search over mask sparsity, and knowledge distillation from the full-attention teacher -- produces a 7.4x-7.8x faster Open-Sora-Plan-1.2 model, with VBench final score 75.30% at threshold $r=0.400$ versus 76.12% for the base model and Content-Debiased Frechet Video Distance (CD-FVD) 231.68 versus 172.64.

Load-bearing premise

The load-bearing premise is that the Attention Tile pattern stays essentially the same across different prompts, video contents, layers, and after consistency distillation, so one fixed sparse mask per layer works for every input with roughly the reported quality.

Editorial extensions

If this is right

  • Longer videos benefit disproportionately: full attention cost grows quadratically with frame count while the fixed sparse mask grows linearly, so the 93-frame model gains as much as the 29-frame model does.
  • The two acceleration sources multiply: the MLCD checkpoint alone reaches about a 5.00x speedup at 20 inference steps, and the added sparsity at $r=0.400$ lifts the total to 7.80x.
  • Because the mask is fixed and layer-specific, no per-input attention selection or inference-time search is needed, which keeps the overhead low and makes the method compatible with static compilation and CUDA graphs.
  • The approach transfers to a different attention architecture: on CogVideoX-5B (MM-DiT), the same search-and-distill recipe keeps the VBench final score within one percentage point while speeding up the attention kernel by 1.34x.
  • The sparse mask is orthogonal to sequence parallelism: running on 4 GPUs with all-to-all sequence parallel attention gives an additional 3.68x-3.91x speedup.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Attention Tile pattern persists at longer horizons and higher resolutions, the same fixed-mask design should give even larger relative gains on future models, since attention dominates more as sequences lengthen; this extrapolation is not tested in the paper.
  • The data-independence claim is currently supported by only two prompts and one model family, so a broader stress test across diverse prompts, camera motion, and distilled checkpoints would determine whether the 90% overlap ratio holds when content changes drastically.
  • The VBench trade-off is uneven: imaging quality and object-class accuracy drop more than motion smoothness, so applications that prioritize photorealism or fine-grained semantics may notice the degradation more than the composite score suggests.
  • The same static mask could be applied during training rather than only at inference, reducing memory and FLOPs for fine-tuning or distillation of video DiTs; the paper does not explore this direction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Efficient-vDiT, a three-stage pipeline to accelerate video diffusion transformers. It first identifies an empirically observed "Attention Tile" structure in 3D attention maps of video DiTs, characterized by large diagonal blocks, locality, and a claimed data-independent repetition pattern. Based on this, it replaces full attention with a fixed family of sparse masks that keep diagonal blocks and a constant number k of global reference frames, reducing attention complexity from quadratic to linear in the number of frames. Stage 1 applies multi-step latent consistency distillation (MLCD) to reduce sampling steps. Stage 2 performs a layer-wise greedy search over sparse masks using an MSE threshold r. Stage 3 applies knowledge distillation from the MLCD teacher to the sparse student. Experiments on Open-Sora-Plan-1.2 (29-frame and 93-frame 720p) report 7.4x--7.8x end-to-end speedups with VBench final scores within about 1% of the base model, plus a 3.68x--3.91x further speedup with sequence parallelism on 4 GPUs. The method is also applied to CogVideoX-5B with a smaller speedup.

Significance. If the central claims hold, the paper makes a useful contribution to efficient video diffusion inference. The core observations about repetitive, locally structured 3D attention are interesting and could inform future sparse-attention designs. The paper also ships a concrete end-to-end system with measured kernel-level and end-to-end speedups, which is valuable. The three-stage pipeline, including the layer-wise mask search and knowledge distillation, is a plausible recipe for converting a pretrained dense model into a fast sparse model with limited data. The main caveats are that the "data-independent" claim rests on very thin evidence and that the "marginal performance trade-off" is supported mainly by an aggregate score while several per-dimension metrics degrade substantially. The results are measured and internally consistent, but the generality of the fixed-mask assumption and the interpretation of quality preservation need stronger support before the central claims are fully convincing.

major comments (3)
  1. [Section 3.1, Figure 1d] The claim that the Attention Tile pattern is "data independent" is load-bearing because it justifies using one fixed sparse mask per layer at inference time, and it is supported only by Figure 1d, which compares top-90/95/99% attention positions for two prompts. Two samples cannot establish stability across the prompt/content distribution, across diffusion timesteps, or after the MLCD and knowledge-distillation stages that alter the model weights. The paper should either provide a quantitative study over a larger and more diverse prompt set, report overlap statistics across timesteps and across the distilled model, or explicitly weaken the claim to "stable on the evaluated distribution" and discuss the risk of per-prompt quality drops.
  2. [Section 4.3, Table 2] The abstract and Section 4.3 describe the quality degradation as "marginal," but this is supported only by the aggregate VBench final score. In Table 2, Object Class drops from 64.72% to 54.98% at r=0.400, and CD-FVD degrades from 172.64 to 231.68, a relative increase of about 34%. The paper should report per-dimension VBench results with error bars or a per-prompt worst-case analysis, and should directly address whether the fixed-mask approach produces large failures on a subset of prompts. Without this, the reader cannot distinguish "fixed masks work generally" from "fixed masks happen to work on the average of the evaluated prompts."
  3. [Algorithm 1 and Table 2] The threshold r in Algorithm 1 is a free parameter that directly controls the speed/quality trade-off, yet the paper does not describe how r is chosen or whether the reported quality values are selected post hoc. Because the same VBench evaluation is used to motivate the threshold values, the reported trade-off curve should be validated on a held-out prompt set or with a clear protocol for selecting r. Otherwise the claim that a particular speedup comes with a "marginal" drop is partly a result of tuning r on the evaluation benchmark.
minor comments (4)
  1. [Throughout] There are several typos and grammatical errors, e.g., "seaminglessly" in Section 4.2.2, "qualitvative" in Section 4.4, and "simliar" in the caption of Figure 6. These should be corrected.
  2. [Section 4.2.2] The sentence "If reported 29 frames generation on multi-GPUs, Ours r=0.100 can achieve 25.8x speedup on 4 GPUs and 13.0x speedup on 2 GPUs" is unclear; the derivation of these numbers from the per-step speedups in Table 3 should be explained.
  3. [Section 3.3, Algorithm 1] Algorithm 1 is described as greedily selecting the largest k (or sparsest mask) within threshold r, but the mask list is indexed from dense to sparse and the loop breaks when the threshold is exceeded. The pseudocode should clarify whether the "best mask" is the last mask that satisfies the threshold or the first one that violates it.
  4. [Appendix A] The Lagrangian relaxation appendix assumes that the impact of different layers on image quality is additive, but this assumption is not tested. Since this appendix is not used in the main results, it could be shortened or marked as preliminary.

Circularity Check

0 steps flagged · score 2.0 of 10

No material circularity: the central speed/quality claims are measured against external benchmarks; only minor non-load-bearing self-citations are present.

full rationale

The paper's central derivations are not circular. The Attention Tile pattern is presented as an empirical observation about 3D DiT attention maps (Fig. 1a-d), and the sparse mask family is a design consequence of that observation, not defined in terms of the target VBench or CD-FVD scores. Stage 2's layer-wise search (Algorithm 1) chooses masks by a threshold r on the MSE of final hidden states; this is a standard validation-style knob, and the resulting quality is then measured independently on VBench and CD-FVD in Tables 2, 7, and 8. The three-stage pipeline (MLCD, mask search, KD) is evaluated against the external benchmarks, so the final claim is not forced by construction. The self-citations to Xie et al. (MLCM), Li et al. (DistFlashAttn), and Xue et al. (LongVILA) are method adoptions, not load-bearing support for the paper's predictions; the MLCD baseline itself is independently evaluated in Table 2 (76.81% VBench, 5.00x speedup), and the sequence-parallel speedup is measured on the authors' own system. The paper also explicitly acknowledges a limitation at Table 2: 'the imaging quality and subject class are lower than those of the base model.' The main concern, that data-independence of the Attention Tile is supported by only two prompts in Fig. 1d and that the fixed-mask approach may not transfer across content or after distillation, is a robustness/correctness risk rather than a circularity. Score 2 reflects the minor self-citations, none of which is load-bearing for the central external benchmark claims.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the empirical generality of the Attention Tile pattern, the proxy quality metric used in the mask search, and the transferability of MLCD and FlexAttention behavior to the video setting. These are domain assumptions rather than derived theorems.

free parameters (5)
  • Mask search threshold r = 0.025, 0.05, 0.1, 0.2, 0.4
    User-chosen threshold in Algorithm 1 that controls sparsity, speed, and quality; the reported speedups correspond to a post hoc sweep over r, not a predicted value.
  • Number of global reference frames k = Varies per layer (e.g., 2:6, 3:5, 1:7)
    Design choice of the mask family; the layerwise search picks among masks with different k to satisfy the threshold.
  • Number of MLCD inference steps = 20
    The authors write that they select the checkpoint with 20 inference steps because it gives the best qualitative result, an empirical choice.
  • Diffusion loss scale lambda = 100 for Open-Sora-Plan, 1 for CogVideoX
    Tuned to balance the diffusion loss against attention and MLP distillation losses in Equation (2).
  • Number of MLCD segments S = Not reported
    A hyperparameter inherited from Xie et al. (MLCM); the paper does not state the value used, which affects the distillation trajectory split.
assumptions (5)
  • domain assumption The Attention Tile pattern is data-independent and stable across inputs
    Load-bearing for using a fixed mask at inference; supported only by the two-prompt overlap analysis in Figure 1d, an insufficient sample.
  • domain assumption Hidden-state MSE is a valid proxy for final video quality in the mask search
    Algorithm 1 selects masks using MSE of final hidden states; the paper does not validate that this correlates with VBench or CD-FVD.
  • domain assumption Multi-step consistency distillation transfers to video DiTs without modification
    The MLCD loss is adopted from image latent diffusion and applied to video latents; no proof of convergence or stability in the video setting is given.
  • domain assumption FlexAttention skips masked blocks exactly, so reported sparsity translates to wall-clock speedup
    Kernel speedups rely on FlexAttention's block-skip behavior; if the backend materializes or partially computes masked blocks, the speedup numbers would not hold.
  • domain assumption Open-Sora-Plan-1.2 is representative of 3D full-attention video DiTs
    The Attention Tile pattern was studied mainly on this model family (plus CogVideoX), yet the paper generalizes the finding to 3D DiTs broadly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile." pith.science (2026). https://pith.science/paper/4CC5WIBQ

@misc{pith2026250206155,
  author       = {Pith},
  title        = {Pith review of: Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4CC5WIBQ}},
  note         = {Machine review of arXiv:2502.06155}
}
read the original abstract

Despite the promise of synthesizing high-fidelity videos, Diffusion Transformers (DiTs) with 3D full attention suffer from expensive inference due to the complexity of attention computation and numerous sampling steps. For example, the popular Open-Sora-Plan model consumes more than 9 minutes for generating a single video of 29 frames. This paper addresses the inefficiency issue from two aspects: 1) Prune the 3D full attention based on the redundancy within video data; We identify a prevalent tile-style repetitive pattern in the 3D attention maps for video data, and advocate a new family of sparse 3D attention that holds a linear complexity w.r.t. the number of video frames. 2) Shorten the sampling process by adopting existing multi-step consistency distillation; We split the entire sampling trajectory into several segments and perform consistency distillation within each one to activate few-step generation capacities. We further devise a three-stage training pipeline to conjoin the low-complexity attention and few-step generation capacities. Notably, with 0.1% pretraining data, we turn the Open-Sora-Plan-1.2 model into an efficient one that is 7.4x -7.8x faster for 29 and 93 frames 720p video generation with a marginal performance trade-off in VBench. In addition, we demonstrate that our approach is amenable to distributed inference, achieving an additional 3.91x speedup when running on 4 GPUs with sequence parallelism.

Figures

Figures reproduced from arXiv: 2502.06155 by the authors.

Figure 1
Figure 1. We observe the Attention Tile pattern in 3D DiTs. (a) the attention map can be broken down into smaller repetitive blocks. (b) These blocks can be classified into two types, where attention weights on the diagonal blocks are noticeably larger than on off-diagonal ones. (c) These blocks exhibit locality, where the attention score differences between the first frame and later frames gradually increases. (d) The block … view at source ↗
Figure 2
Figure 2. EFFICIENT-VDIT takes in a pre-trained 3D Full Attention video diffusion transformer(DiT), with slow inference speed and high fidelity. It then operates on three stages to greatly accelerate the inference while maintaining the fidelity. In Stage 1, we modify the multi-step consistency distillation framework from (Heek et al., 2024) to the video domain, which turned a DiT model to a CM model with stable training. In S… view at source ↗
Figure 3
Figure 3. Exemplar attention mask (2 : 6). It maintains the attention in the main diagonals and against 2 global refer￾ence latent frames. Tile blocks in white are not computed. of sparse attention masks. Large Diagonals Tile blocks on the main diagonals has higher attention scores than off-diagonal ones. In Fig￾ure 1(b), we plot the attention scores at the main diagonal tile blocks, compared to attention scores at the off-di… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Search results for Open-Sora-Plan v1.2 model (29 frames). We verify that different layers have different sparsity in 3D video DiTs. Layer-wise Searching For Attention Masks Previous 4 [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Qualitative samples of our models. We com [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Qualitative samples of ablation of distillation order. sampled from VBench prompts. We show that both MLCD [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Qualitative samples of CogvideoX-5B (Yang et al., 2024b) distillation from its sample prompts. We show that [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Based on Open-Sora’s examples (Zheng et al., 2024) , we selected dynamic prompts featuring centralized explo [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Qualitative samples of dynamic scenes from VBench prompts. We show that both MLCD and E [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Qualitative samples of dynamic scenes from VBench prompts. We show that both MLCD and E [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers

    cs.CV 2026-07 conditional novelty 6.5 of 10

    A hybrid NoPE visual diffusion backbone with HeteroP scaling yields ~7× pretraining compute efficiency versus matched full attention and near-stable zero-shot 5s→30s video extrapolation.

  2. CODA: Algorithm-Hardware Co-design for Edge Video Diffusion via NMP-Enabled Compute-Cache Operator Disaggregation

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Disaggregating cache operators from compute and overlapping them across the two classifier-free-guidance branches turns cross-timestep caching into up to 1.80x real end-to-end speedup on edge GPUs when the cache overf...

  3. Kaleido: Algorithm-Hardware Co-Design for Video Diffusion Transformers by Exploiting Latent Space Correlations

    cs.AR 2026-07 conditional novelty 6.0 of 10

    A channel-wise reuse algorithm plus a reconfigurable systolic accelerator skips redundant vDiT attention and MLP computation, achieving up to 5.9x speedup and 16x energy savings.

  4. Timeripple: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Space

    cs.AR 2025-11 conditional novelty 6.0 of 10

    Timeripple cuts vDiT self-attention compute by up to 85% by reusing partial attention scores of spatially and temporally correlated tokens across channels, with VBench quality essentially unchanged.

  5. SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference

    cs.CV 2026-08 conditional novelty 5.0 of 10

    SPADE combines static, semi-static, and dynamic block-sparse attention with a cheap SICS-based blocking heuristic to speed up video diffusion inference by up to 1.80x end-to-end.

  6. Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Sparse-vDiT replaces dense attention with fixed per-head sparse patterns chosen offline, achieving 1.58-1.85x end-to-end speedups on CogVideoX1.5, HunyuanVideo, and Wan2.1 with minimal quality loss.

Reference graph

Works this paper leans on

65 extracted references · 20 canonical work pages · cited by 6 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Y., Suk, H., Suo, M., Tillet, P., Wang, E., Wang, X., Wen, W., Zhang, S., Zhao, X., Zhou, K., Zou, R., Mathews, A., Chanan, G., Wu, P., and Chintala, S

    Ansel, J., Yang, E., He, H., Gimelshein, N., Jain, A., Voznesensky, M., Bao, B., Bell, P., Berard, D., Burovski, E., Chauhan, G., Chourdia, A., Constable, W., Desmaison, A., DeVito, Z., Ellison, E., Feng, W., Gong, J., Gschwind, M., Hirsh, B., Huang, S., Kalambarkar, K., Kirsch, L., Lazos, M., Lezcano, M., Liang, Y., Liang, J., Lu, Y., Luk, C., Maher, B.,...

    arXiv 2024

  3. [3]

    M., Du, Y., Simchowitz, M., Tedrake, R., and Sitzmann, V

    Chen, B., Monso, D. M., Du, Y., Simchowitz, M., Tedrake, R., and Sitzmann, V. Diffusion forcing: Next-token prediction meets full-sequence diffusion. arXiv preprint arXiv:2407.01392, 2024 a

    arXiv 2024

  4. [4]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models

    Chen, H., Zhang, Y., Cun, X., Xia, M., Wang, X., Weng, C., and Shan, Y. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 7310--7320, 2024 b

    work page 2024

  5. [5]

    E., Fang, Y., Lee, H.-Y., Ren, J., Yang, M.-H., and Tulyakov, S

    Chen, T.-S., Siarohin, A., Menapace, W., Deyneka, E., Chao, H.-w., Jeon, B. E., Fang, Y., Lee, H.-Y., Ren, J., Yang, M.-H., and Tulyakov, S. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. arXiv preprint arXiv:2402.19479, 2024 c

    arXiv 2024

  6. [6]

    Asyncdiff: Parallelizing diffusion models by asynchronous denoising

    Chen, Z., Ma, X., Fang, G., Tan, Z., and Wang, X. Asyncdiff: Parallelizing diffusion models by asynchronous denoising. arXiv preprint arXiv:2406.06911, 2024 d

    arXiv 2024

  7. [7]

    Structure and content-guided video synthesis with diffusion models

    Esser, P., Chiu, J., Atighehchian, P., Granskog, J., and Germanidis, A. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 7346--7356, 2023

    work page 2023

  8. [8]

    Preserve your own correlation: A noise prior for video diffusion models

    Ge, S., Nah, S., Liu, G., Poon, T., Tao, A., Catanzaro, B., Jacobs, D., Huang, J.-B., Liu, M.-Y., and Balaji, Y. Preserve your own correlation: A noise prior for video diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 22930--22941, 2023 a

    work page 2023

Show all 65 references
  1. [9]

    Model tells you what to discard: Adaptive kv cache compression for llms

    Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., and Gao, J. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801, 2023 b

    2023 arXiv

  2. [10]

    On the content bias in fréchet video distance

    Ge, S., Mahapatra, A., Parmar, G., Zhu, J.-Y., and Huang, J.-B. On the content bias in fréchet video distance. arXiv preprint arXiv:2404.12391, 2024

    2024 arXiv

  3. [11]

    Minillm: Knowledge distillation of large language models

    Gu, Y., Dong, L., Wei, F., and Huang, M. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  4. [12]

    Sparsectrl: Adding sparse controls to text-to-video diffusion models

    Guo, Y., Yang, C., Rao, A., Agrawala, M., Lin, D., and Dai, B. Sparsectrl: Adding sparse controls to text-to-video diffusion models. arXiv preprint arXiv:2311.16933, 2023

    2023 arXiv

  5. [13]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Guo, Y., Yang, C., Rao, A., Liang, Z., Wang, Y., Qiao, Y., Agrawala, M., Lin, D., and Dai, B. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. International Conference on Learning Representations, 2024

    2024

  6. [14]

    Latent video diffusion models for high-fidelity long video generation

    He, Y., Yang, T., Zhang, Y., Shan, Y., and Chen, Q. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221, 2022

    2022 arXiv

  7. [15]

    Multistep consistency models

    Heek, J., Hoogeboom, E., and Salimans, T. Multistep consistency models. arXiv preprint arXiv:2403.06807, 2024

    2024 arXiv

  8. [16]

    Streamingt2v: Consistent, dynamic, and extendable long video generation from text

    Henschel, R., Khachatryan, L., Hayrapetyan, D., Poghosyan, H., Tadevosyan, V., Wang, Z., Navasardyan, S., and Shi, H. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. arXiv preprint arXiv:2403.14773, 2024

    2024 arXiv

  9. [17]

    Distilling the knowledge in a neural network

    Hinton, G. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  10. [18]

    Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J. Video diffusion models. Advances in Neural Information Processing Systems, 35: 0 8633--8646, 2022

    2022

  11. [19]

    Vbench: Comprehensive benchmark suite for video generative models

    Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 21807--21818, 2024

    2024

  12. [20]

    A., Tanaka, M., Zhang, C., Zhang, M., Song, S

    Jacobs, S. A., Tanaka, M., Zhang, C., Zhang, M., Song, S. L., Rajbhandari, S., and He, Y. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309.14509, 2023

    2023 arXiv

  13. [21]

    Tinybert: Distilling bert for natural language understanding

    Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., and Liu, Q. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351, 2019

    1909 arXiv

  14. [22]

    Consistency trajectory models: Learning probability flow ode trajectory of diffusion

    Kim, D., Lai, C.-H., Liao, W.-H., Murata, N., Takida, Y., Uesaka, T., He, Y., Mitsufuji, Y., and Ermon, S. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023

    arXiv

  15. [23]

    Kling, 2024

    Kuaishou. Kling, 2024. URL https://kling.kuaishou.com/en. Accessed: [2024]

    2024

  16. [24]

    and etc., T

    Lab, P.-Y. and etc., T. A. Open-sora-plan, April 2024. URL https://doi.org/10.5281/zenodo.10948109

    2024 doi

  17. [25]

    P., Ma, X., Stoica, I., Gonzalez, J

    Li, D., Shao, R., Xie, A., Xing, E. P., Ma, X., Stoica, I., Gonzalez, J. E., and Zhang, H. Distflashattn: Distributed memory-efficient attention for long-context llms training. In First Conference on Language Modeling, 2024 a

    2024

  18. [26]

    Li, J., Feng, W., Fu, T.-J., Wang, X., Basu, S., Chen, W., and Wang, W. Y. T2v-turbo: Breaking the quality bottleneck of video consistency model with mixed reward feedback. arXiv preprint arXiv:2405.18750, 2024 b

    2024 arXiv

  19. [27]

    Gan compression: Efficient architectures for interactive conditional gans

    Li, M., Lin, J., Ding, Y., Liu, Z., Zhu, J.-Y., and Han, S. Gan compression: Efficient architectures for interactive conditional gans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 5284--5294, 2020

    2020

  20. [28]

    Efficient spatially sparse inference for conditional gans and diffusion models

    Li, M., Lin, J., Meng, C., Ermon, S., Han, S., and Zhu, J.-Y. Efficient spatially sparse inference for conditional gans and diffusion models. Advances in neural information processing systems, 35: 0 28858--28873, 2022

    2022

  21. [29]

    Distrifusion: Distributed parallel inference for high-resolution diffusion models

    Li, M., Cai, T., Cao, J., Zhang, Q., Cai, H., Bai, J., Jia, Y., Li, K., and Han, S. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 7183--7193, 2024 c

    2024

  22. [30]

    Ring attention with blockwise transformers for near-infinite context

    Liu, H., Zaharia, M., and Abbeel, P. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889, 2023 a

    2023 arXiv

  23. [31]

    Scott: Accelerating diffusion models with stochastic consistency distillation

    Liu, H., Xie, Q., Deng, Z., Chen, C., Tang, S., Fu, F., Zha, Z.-j., and Lu, H. Scott: Accelerating diffusion models with stochastic consistency distillation. arXiv preprint arXiv:2403.01505, 2024

    2024 arXiv

  24. [32]

    Instaflow: One step is enough for high-quality diffusion-based text-to-image generation

    Liu, X., Zhang, X., Ma, J., Peng, J., et al. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. In The Twelfth International Conference on Learning Representations, 2023 b

    2023

  25. [33]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps

    Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35: 0 5775--5787, 2022 a

    2022

  26. [34]

    Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models

    Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022 b

    2022 arXiv

  27. [35]

    Videofusion: Decomposed diffusion models for high-quality video generation

    Luo, Z., Chen, D., Zhang, Y., Huang, Y., Wang, L., Shen, Y., Zhao, D., Zhou, J., and Tan, T. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023

    2023

  28. [36]

    Latte: Latent diffusion transformer for video generation

    Ma, X., Wang, Y., Jia, G., Chen, X., Liu, Z., Li, Y.-F., Chen, C., and Qiao, Y. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024

    2024 arXiv

  29. [37]

    and Lee, W

    Mirsky, Y. and Lee, W. The creation and detection of deepfakes: A survey. ACM Computing Surveys, 2020. doi:10.1145/3425780

    2020 doi

  30. [38]

    Sora, 2024

    OpenAI. Sora, 2024. URL https://openai.com/index/sora/. Accessed: [2024]

    2024

  31. [39]

    and Xie, S

    Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4195--4205, 2023

    2023

  32. [40]

    High-resolution image synthesis with latent diffusion models

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 10684--10695, June 2022 a

    2022

  33. [41]

    High-resolution image synthesis with latent diffusion models

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10684--10695, 2022 b

    2022

  34. [42]

    and Ho, J

    Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022

    2022 arXiv

  35. [43]

    Adversarial diffusion distillation

    Sauer, A., Lorenz, D., Blattmann, A., and Rombach, R. Adversarial diffusion distillation. arXiv preprint arXiv:2311.17042, 2023

    2023 arXiv

  36. [44]

    Denoising diffusion implicit models

    Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  37. [45]

    Consistency models

    Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consistency models. arXiv preprint arXiv:2303.01469, 2023

    2023 arXiv

  38. [46]

    Roformer: Enhanced transformer with rotary position embedding

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  39. [47]

    Diffusion models are real-time game engines

    Valevski, D., Leviathan, Y., Arar, M., and Fruchter, S. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837, 2024

    2024 arXiv

  40. [48]

    Cuttlefish: Low-rank model training without all the tuning

    Wang, H., Agarwal, S., Tanaka, Y., Xing, E., Papailiopoulos, D., et al. Cuttlefish: Low-rank model training without all the tuning. Proceedings of Machine Learning and Systems, 5: 0 578--605, 2023 a

    2023

  41. [49]

    Pipefusion: Displaced patch pipeline parallelism for inference of diffusion transformer models

    Wang, J., Fang, J., Li, A., and Yang, P. Pipefusion: Displaced patch pipeline parallelism for inference of diffusion transformer models. arXiv preprint arXiv:2405.14430, 2024 a

    2024 arXiv

  42. [50]

    Qihoo-t2x: An efficiency-focused diffusion transformer via proxy tokens for text-to-any-task

    Wang, J., Ma, A., Feng, J., Leng, D., Yin, Y., and Liang, X. Qihoo-t2x: An efficiency-focused diffusion transformer via proxy tokens for text-to-any-task. arXiv preprint arXiv:2409.04005, 2024 b

    2024 arXiv

  43. [51]

    Videolcm: Video latent consistency model

    Wang, X., Zhang, S., Zhang, H., Liu, Y., Zhang, Y., Gao, C., and Sang, N. Videolcm: Video latent consistency model. arXiv preprint arXiv:2312.09109, 2023 b

    2023 arXiv

  44. [52]

    Lavie: High-quality video generation with cascaded latent diffusion models

    Wang, Y., Chen, X., Ma, X., Zhou, S., Huang, Z., Wang, Y., Yang, C., He, Y., Yu, J., Yang, P., et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023 c

    2023 arXiv

  45. [53]

    Pandora: Towards general world model with natural language actions and video states

    Xiang, J., Liu, G., Gu, Y., Gao, Q., Ning, Y., Zha, Y., Feng, Z., Tao, T., Hao, S., Shi, Y., et al. Pandora: Towards general world model with natural language actions and video states. arXiv preprint arXiv:2406.09455, 2024

    2024 arXiv

  46. [54]

    Efficient streaming language models with attention sinks

    Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023

    2023 arXiv

  47. [55]

    Mlcm: Multistep consistency distillation of latent diffusion model

    Xie, Q., Liao, Z., Deng, Z., Tang, S., Lu, H., et al. Mlcm: Multistep consistency distillation of latent diffusion model. arXiv preprint arXiv:2406.05768, 2024

    2024 arXiv

  48. [56]

    Longvila: Scaling long-context visual language models for long videos

    Xue, F., Chen, Y., Li, D., Hu, Q., Zhu, L., Li, X., Fang, Y., Tang, H., Yang, S., Liu, Z., et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188, 2024

    2024 arXiv

  49. [57]

    mt5: A massively multilingual pre-trained text-to-text transformer

    Xue, L. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020

    2010 arXiv

  50. [58]

    Pyramidinfer: Pyramid kv cache compression for high-throughput llm inference

    Yang, D., Han, X., Gao, Y., Hu, Y., Zhang, S., and Zhao, H. Pyramidinfer: Pyramid kv cache compression for high-throughput llm inference. arXiv preprint arXiv:2405.12532, 2024 a

    2024 arXiv

  51. [59]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng, G., et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024 b

    2024 arXiv

  52. [60]

    T., and Park, T

    Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., and Park, T. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 6613--6623, 2024

    2024

  53. [61]

    S., Kim, G.-W., Kim, S., and Chun, B.-G

    Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G. Orca: A distributed serving system for \ Transformer-Based \ generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp.\ 521--538, 2022

    2022

  54. [62]

    Q-hitter: A better token oracle for efficient llm inference via sparse-quantized kv cache

    Zhang, Z., Liu, S., Chen, R., Kailkhura, B., Chen, B., and Wang, A. Q-hitter: A better token oracle for efficient llm inference via sparse-quantized kv cache. Proceedings of Machine Learning and Systems, 6: 0 381--394, 2024 a

    2024

  55. [63]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., R \'e , C., Barrett, C., et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024 b

    2024

  56. [64]

    Real-time video generation with pyramid attention broadcast

    Zhao, X., Jin, X., Wang, K., and You, Y. Real-time video generation with pyramid attention broadcast. arXiv preprint arXiv:2408.12588, 2024

    2024 arXiv

  57. [65]

    Open-sora: Democratizing efficient video production for all, March 2024

    Zheng, Z., Peng, X., Yang, T., Shen, C., Li, S., Liu, H., Zhou, Y., Li, T., and You, Y. Open-sora: Democratizing efficient video production for all, March 2024. URL https://github.com/hpcaitech/Open-Sora

    2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.