Pith. sign in

REVIEW 36 cited by

Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.01776 v2 pith:XKSBDS2Q submitted 2025-02-03 cs.CV cs.LG

Haocheng Xi , Shuo Yang , Yilong Zhao , Chenfeng Xu , Muyang Li , Xiuyu Li , Yujun Lin , Han Cai
show 6 more authors
Jintao Zhang Dacheng Li Jianfei Chen Ion Stoica Kurt Keutzer Song Han
This is my paper
classification cs.CVcs.LG
keywords attentionsparsedominateheadvideocomputationaldiffusionfull
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Diffusion Transformers (DiTs) dominate video generation but their high computational cost severely limits real-world applicability, usually requiring tens of minutes to generate a few seconds of video even on high-performance GPUs. This inefficiency primarily arises from the quadratic computational complexity of 3D Full Attention with respect to the context length. In this paper, we propose a training-free framework termed Sparse VideoGen (SVG) that leverages the inherent sparsity in 3D Full Attention to boost inference efficiency. We reveal that the attention heads can be dynamically classified into two groups depending on distinct sparse patterns: (1) Spatial Head, where only spatially-related tokens within each frame dominate the attention output, and (2) Temporal Head, where only temporally-related tokens across different frames dominate. Based on this insight, SVG proposes an online profiling strategy to capture the dynamic sparse patterns and predicts the type of attention head. Combined with a novel hardware-efficient tensor layout transformation and customized kernel implementations, SVG achieves up to 2.28x and 2.33x end-to-end speedup on CogVideoX-v1.5 and HunyuanVideo, respectively, while preserving generation quality. Our code is open-sourced and is available at https://github.com/svg-project/Sparse-VideoGen

Discussion (0). Sign in to comment.

Forward citations

Cited by 36 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation

    cs.CV 2026-02 conditional novelty 7.0 of 10

    Causal Forcing uses an autoregressive teacher for ODE initialization in diffusion distillation to close the causal attention gap and deliver better real-time video generation than Self Forcing.

  2. FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Models

    cs.CV 2025-09 conditional novelty 7.0 of 10

    FG-Attn accelerates video DiT inference by computing attention over only the relevant Mx1 key slices per query group, using an asynchronous gather-load kernel, and reports up to 1.65x end-to-end speedup with negligibl...

  3. Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers

    cs.CV 2026-07 conditional novelty 6.5 of 10

    A hybrid NoPE visual diffusion backbone with HeteroP scaling yields ~7× pretraining compute efficiency versus matched full attention and near-stable zero-shot 5s→30s video extrapolation.

  4. ContextMaster: Interactive Multi-Shot Video Creation via Fixed-Budget Sparse Context Routing

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A unified video model with role-aware context and fixed-budget sparse attention enables interactive multi-shot generation, reference conditioning, and editing at 16 FPS on a single GPU.

  5. Token Radius Attention for Efficient Video Generation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Video diffusion transformers can run ~1.5-2x faster with competitive quality by converting each query's attention entropy into a spatially decayed retention radius instead of dense attention.

  6. EchoCache: Energy-Guided Cross-Modal Caching for Efficient Audio-Driven Video Generation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Audio time-frequency energy guides which video latents get recomputed during diffusion denoising, yielding up to 2.46x faster audio-driven video generation with competitive quality.

  7. SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A 5B/14B video diffusion transformer using 75% linear + 25% softmax attention with block attention residuals generates 480p/720p video on one GPU faster than full-softmax models at comparable VBench quality.

  8. Surprise Forcing: What to Remember, When to Skip in Long Video Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A training-free 'surprise' controller decides which old frames to keep in memory and which chunks need fewer denoising steps, improving long-video consistency at real-time speed.

  9. DiTango: Cost-Effective Parallel Diffusion Generation with Selective Attention State Reuse

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Selective reuse of composed attention states across denoising steps lets DiTango skip both remote KV communication and attention compute for low-contribution sequence partitions, cutting multi-GPU diffusion latency by...

  10. Kaleido: Algorithm-Hardware Co-Design for Video Diffusion Transformers by Exploiting Latent Space Correlations

    cs.AR 2026-07 conditional novelty 6.0 of 10

    A channel-wise reuse algorithm plus a reconfigurable systolic accelerator skips redundant vDiT attention and MLP computation, achieving up to 5.9x speedup and 16x energy savings.

  11. Controlling Motion Transfer in Diffusion Transformers via Attention Heads

    cs.CV 2026-07 accept novelty 6.0 of 10

    Video DiTs encode motion and structure in separate attention-head subsets; selecting and guiding those heads yields training-free motion transfer with higher fidelity and structural alignment than existing methods.

  12. SAF3R: Dynamic Sparse Attention for Feed-Forward 3D Reconstruction Transformers

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Training-free head-wise dynamic sparse attention preserves F3R pose and reconstruction quality while delivering multi-fold end-to-end speedups on long image sequences.

  13. Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attention

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Sparse attention with chunk-aware sparsity growth and hierarchical frame/block selection accelerates autoregressive video diffusion at ~1.3x with VBench quality on par with dense attention.

  14. Transition Matching Distillation for Fast Video Generation

    cs.CV 2026-01 conditional novelty 6.0 of 10

    Splitting a video diffusion model into a fixed feature extractor and a small recurrent flow head lets TMD generate videos in one to two effective steps with better VBench scores than prior distilled models.

  15. Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformers

    cs.CV 2026-01 conditional novelty 6.0 of 10

    MOD-DiT accelerates video DiTs by extrapolating fitted attention-pattern intensities across denoising steps to build dynamic block masks, achieving 1.8–2.3× speedups on CogVideoX, HunyuanVideo, and Wan2.1.

  16. SnapGen++: Unleashing Diffusion Transformers for Efficient High-Fidelity Image Generation on Edge Devices

    cs.CV 2026-01 conditional novelty 6.0 of 10

    A compact elastic diffusion transformer with adaptive sparse attention and knowledge-guided distribution-matching distillation achieves 4-step 1K image generation on a phone in roughly 1.8 seconds.

  17. TinyHistory: Lightweight Video History Embeddings via Two-Stage Context Learning

    cs.CV 2025-12 conditional novelty 6.0 of 10

    TinyHistory compresses long video history into a ~5k-token context via a two-stage learning scheme, achieving consistency on par with heavier baselines at lower memory cost.

  18. Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers

    cs.CV 2025-12 conditional novelty 6.0 of 10

    LLSA uses hierarchical top-K selection plus coarse-token enrichment to make diffusion-transformer attention O(N log N), giving about 6x faster training at 256x256 pixel resolution with FID comparable to full attention.

  19. End-to-End Training for Autoregressive Video Diffusion via Self-Resampling

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Resampling Forcing trains autoregressive video diffusion models on self-resampled degraded histories with a causal mask, achieving stable long-horizon generation without a teacher or discriminator.

  20. Timeripple: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Space

    cs.AR 2025-11 conditional novelty 6.0 of 10

    Timeripple cuts vDiT self-attention compute by up to 85% by reusing partial attention scores of spatially and temporally correlated tokens across channels, with VBench quality essentially unchanged.

  21. OmniCache: A Trajectory-Oriented Global Perspective on Training-Free Cache Reuse for Diffusion Transformer Models

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    A training-free cache-reuse scheme that spreads computation across the full diffusion trajectory and subtracts estimated noise, accelerating DiT sampling with claimed competitive quality.

  22. Waver: Wave Your Way to Lifelike Video Generation

    cs.CV 2025-08 conditional novelty 6.0 of 10

    Waver unifies text-to-video, image-to-video, and text-to-image generation in a single 12B-parameter DiT with a hybrid dual/single-stream architecture and a cascade refiner, claiming top-three public leaderboard performance.

  23. FreeLong++: Training-Free Long Video Generation via Multi-band SpectralFusion

    cs.CV 2025-06 conditional novelty 6.0 of 10

    FreeLong++ extends short-video diffusion models to 4x to 8x longer clips, without retraining, by fusing multiple windowed attention branches through frequency-domain filters and a spectral noise initialization.

  24. VMoBA: Mixture-of-Block Attention for Video Diffusion Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    VMoBA is a sparse attention mechanism for video diffusion models that combines cyclic 1D-2D-3D block partitioning with global and threshold-based block selection to reduce training FLOPs while keeping generation quality.

  25. TurboVSR: Fantastic Video Upscalers and Where to Find Them

    cs.CV 2025-06 conditional novelty 6.0 of 10

    TurboVSR uses a high-compression video autoencoder with factorized conditioning and non-uniform shortcut sampling to achieve near-state-of-the-art perceptual video super-resolution at roughly 100x lower compute cost.

  26. Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Chipmunk speeds up diffusion transformer inference by recomputing, at each step, only the top percent of attention and MLP activation columns that change most between steps, caching the rest in column-sparse GPU kernels.

  27. Dual-Expert Consistency Model for Efficient and High-Quality Video Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    By training a semantic expert and a LoRA-based detail expert, DCM reaches nearly teacher-level VBench scores with 4-step video sampling on HunyuanVideo and CogVideoX.

  28. SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference

    cs.CV 2026-08 conditional novelty 5.0 of 10

    SPADE combines static, semi-static, and dynamic block-sparse attention with a cheap SICS-based blocking heuristic to speed up video diffusion inference by up to 1.80x end-to-end.

  29. ACID: Adaptive Caching for vIDeo generation

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Adaptive threshold switching on the drift signal's rate of change expands the quality-vs-speed Pareto frontier of TeaCache, EasyCache, and DiCache across HunyuanVideo, Wan 2.1, and CogVideoX.

  30. MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models

    cs.AI 2026-06 conditional novelty 5.0 of 10

    By sampling variance-inflated query vectors during prefilling, MM-ShiftKV selects prompt KV caches that better match decoding-time attention and outperforms prior prefill-only KV compression on multimodal benchmarks a...

  31. UltraImageGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attention

    cs.CV 2025-10 conditional novelty 5.0 of 10

    A pretrained FLUX diffusion model is adapted with local-window attention plus low-resolution global guidance, allowing 4K text-to-image generation from 1K-only training data at about 2x lower cost.

  32. NABLA: Neighborhood Adaptive Block-Level Attention

    cs.CV 2025-07 conditional novelty 5.0 of 10

    NABLA builds content-aware block-sparse attention masks by pooling queries and keys and thresholding their softmax CDF, giving near-baseline video quality at up to 2.7x inference speedup and 1.46x pretraining speedup.

  33. Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Sparse-vDiT replaces dense attention with fixed per-head sparse patterns chosen offline, achieving 1.58-1.85x end-to-end speedups on CogVideoX1.5, HunyuanVideo, and Wan2.1 with minimal quality loss.

  34. RainFusion: Adaptive Video Generation Acceleration via Multi-Dimensional Visual Redundancy

    cs.CV 2025-05 conditional novelty 5.0 of 10

    Training-free sparse attention that classifies each head as spatial, temporal, or textural and applies a matched mask or token reduction, giving about 1.9x attention speedup with small VBench losses.

  35. PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models

    cs.CV 2025-06 conditional novelty 4.0 of 10

    PAROAttention permutes tokens along frame, height, and width axes to make visual attention block-wise, enabling sparse and INT8/INT4 quantized attention with near-baseline generation quality.

  36. SageAttention2++: A More Efficient Implementation of SageAttention2

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Using the FP16-accumulator FP8 matmul instruction with narrowed quantization ranges for P and V yields up to 3.9x speedup over FlashAttention2 with negligible accuracy loss.

Pith tools