Pith. sign in

REVIEW 5 cited by

DiG: Scalable and Efficient Diffusion Models with Gated Linear Attention

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.18428 v2 pith:57RDJMXZ submitted 2024-05-28 cs.CV cs.AI

Lianghui Zhu , Zilong Huang , Bencheng Liao , Jun Hao Liew , Hanshu Yan , Jiashi Feng , Xinggang Wang This is my paper · ORCID
classification cs.CVcs.AI
keywords diffusionresolutionmodelstimesattentionefficiencyfastergated
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Diffusion models with large-scale pre-training have achieved significant success in the field of visual content generation, particularly exemplified by Diffusion Transformers (DiT). However, DiT models have faced challenges with quadratic complexity efficiency, especially when handling long sequences. In this paper, we aim to incorporate the sub-quadratic modeling capability of Gated Linear Attention (GLA) into the 2D diffusion backbone. Specifically, we introduce Diffusion Gated Linear Attention Transformers (DiG), a simple, adoptable solution with minimal parameter overhead. We offer two variants, i,e, a plain and U-shape architecture, showing superior efficiency and competitive effectiveness. In addition to superior performance to DiT and other sub-quadratic-time diffusion models at $256 \times 256$ resolution, DiG demonstrates greater efficiency than these methods starting from a $512$ resolution. Specifically, DiG-S/2 is $2.5\times$ faster and saves $75.7\%$ GPU memory compared to DiT-S/2 at a $1792$ resolution. Additionally, DiG-XL/2 is $4.2\times$ faster than the Mamba-based model at a $1024$ resolution and $1.8\times$ faster than DiT with FlashAttention-2 at a $2048$ resolution. We will release the code soon. Code is released at https://github.com/hustvl/DiG.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Exploring Diffusion Transformer Designs via Grafting

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Grafting uses activation distillation and lightweight fine-tuning to edit pretrained diffusion transformers into hybrid architectures with near-baseline quality at under 2% pretraining compute.

  2. Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Direct3D-S2 uses a new Spatial Sparse Attention mechanism to train a sparse-volume diffusion transformer at 1024^3 resolution on 8 GPUs.

  3. REPA Works Until It Doesn't: Early-Stopped, Holistic Alignment Supercharges Diffusion Training

    cs.CV 2025-05 conditional novelty 6.0 of 10

    HASTE trains diffusion transformers faster by aligning student features and attention maps with a DINOv2 teacher early in training and then switching the alignment off, matching vanilla SiT quality on ImageNet 256x256...

  4. ENA: Efficient N-dimensional Attention

    cs.LG 2025-08 conditional novelty 5.0 of 10

    ENA combines linear recurrence with hardware-friendly sliding tile attention to model images and videos efficiently, claiming Transformer-level accuracy at roughly 70% attention sparsity.

  5. Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Sparse-vDiT replaces dense attention with fixed per-head sparse patterns chosen offline, achieving 1.58-1.85x end-to-end speedups on CogVideoX1.5, HunyuanVideo, and Wan2.1 with minimal quality loss.

Pith tools