Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformers

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GRAT claims diffusion transformers can run up to 35.8x faster in attention by grouping queries and restricting key-value lookups to structured regions, with quality close to full attention.

desk verdict GRAT is a genuinely useful training-free DiT attention approximation with big speedups; the quality claims are mostly believable but statistically under-supported, and the fixed pattern across layers/steps is the softest spot. read the letter →

arxiv 2505.14687 v1 pith:HWI7UAYC submitted 2025-05-20 cs.CV

Sucheng Ren , Qihang Yu , Ju He , Alan Yuille , Liang-Chieh Chen This is my paper · ORCID
classification cs.CV
keywords diffusiontransformersattentionaccelerationstructuredsparsetraining-freeimagegenerationvideoFluxHunyuan
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GRAT is a training-free plug-in that replaces full self-attention in pretrained diffusion transformers with grouped, structured sparse attention. The paper claims this speeds up attention by 35.8x when generating 8192x8192 images with Flux and by 15.8x on HunyuanVideo, while keeping sample quality close to the full-attention baseline. The central insight is that learned attention maps are locally concentrated, so queries in one contiguous group can share a single set of key-value tokens drawn from surrounding blocks or whole rows and columns. If the claim holds, order-of-magnitude faster high-resolution image and video generation does not require retraining or fine-tuning.

What carries the argument

The load-bearing mechanism is the two-stage grouping-then-attending scheme: first, contiguous tokens are partitioned into non-overlapping groups, and then every query in a group shares one attendable set of key and value tokens. GRAT-B uses surrounding blocks, giving $O(b^2g^2N)$ complexity, and GRAT-X uses criss-cross rows and columns, giving $O(gHN+gWN)$ complexity. This converts quadratic attention into near-linear cost while keeping memory access coalesced and the effective receptive field large, which is what turns theoretical FLOPs savings into real GPU speedups.

What would settle it

A direct test would generate images whose composition requires long-range, off-axis relationships, such as two objects placed on opposite diagonal corners, and compare object-count and position accuracy under GRAT-X versus full attention; a material accuracy drop on such prompts would show that the criss-cross pattern does not preserve the long-range context the paper claims.

Watch

Extended reading notes

Core claim

The discovery is that a hand-designed, inference-time sparsity pattern can replace dense attention in pretrained diffusion transformers almost losslessly. GRAT partitions tokens into non-overlapping groups; all queries in a group attend to the same key and value groups. GRAT-B restricts attention to surrounding blocks, while GRAT-X adds whole rows and columns to preserve long-range context. On Flux at 8192x8192, GRAT-B cuts attention latency from 4.081 seconds to 0.114 seconds, a 35.8x speedup, and reduces end-to-end generation from 5480 to 598 seconds, while GRAT-X matches the full-attention GenEval score of 0.66 with 0.65. On HunyuanVideo, GRAT-B gives a 15.8x attention speedup, and GRAT-X reaches a VBench total of 83.82% versus 82.71% for full attention while running 2.4x faster per attention call.

Load-bearing premise

The whole method rests on assuming that one fixed sparsity pattern, chosen from average attention statistics and one ablation, works well enough for every layer, denoising step, prompt, and resolution.

Editorial extensions

If this is right

  • For 8192x8192 Flux generation, GRAT-B reduces attention latency from 4.081 to 0.114 seconds per image, a 35.8x speedup, and total inference time from 5480 to 598 seconds, a 9.2x speedup, with no fine-tuning.
  • GRAT-X preserves full-attention quality on GenEval (0.65 vs 0.66) and on VBench total (83.82% vs 82.71%) while running 2.4x faster per attention call than full attention in HunyuanVideo.
  • Because the method is training-free, it can be swapped into already-deployed pretrained models without retraining, unlike sparse attention schemes that require fine-tuning to recover quality.
  • The criss-cross variant keeps a large effective receptive field, attending to tokens as far as 512 positions away in an 8192 image and 81 positions away in video, avoiding the long-range failure mode of purely local windows.
  • GRAT-B beats prior sparse attention baselines such as CLEAR, neighborhood attention, and sliding tile attention in both speed and quality, indicating that shared grouped key-value sets are what make the sparsity pattern GPU-efficient.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors leave implicit is making group size or sparsity pattern depend on the denoising step, since early diffusion steps may need more global context than the fixed pattern provides.
  • Because GRAT is orthogonal to step-count reduction and weight quantization, combining it with those techniques could compound the speedups beyond what the paper reports.
  • The shared-key-value grouping idea should transfer to other quadratic-attention settings, such as autoregressive visual generation or long-context vision-language models, wherever attention is also locally concentrated.
  • The paper's own GenEval sub-scores show counting remains the weakest skill for GRAT-X (0.62 versus 0.76 for full attention), which points to a concrete stress test for long-range compositional reasoning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes GRAT, a training-free acceleration method for attention in pretrained Diffusion Transformers. GRAT partitions tokens into non-overlapping groups and lets all queries in a group share a common set of key/value tokens restricted to structured regions: surrounding blocks (GRAT-B) or criss-cross rows/columns (GRAT-X). The method is validated on Flux for text-to-image generation and HunyuanVideo for text-to-video generation, reporting large attention speedups (up to 35.8x for Flux at 8192x8192 and 15.8x for HunyuanVideo) while claiming generation quality comparable to full attention. The paper includes efficiency comparisons against CLEAR, Neighborhood Attention, and STA, quantitative benchmarks (COCO2014, MJHQ-30K, GenEval, VBench), ablations over group size and block size, and qualitative examples.

Significance. If the claims hold, GRAT would be practically valuable: it is training-free, requires no fine-tuning, is built on hardware-friendly grouping that improves GPU utilization, and demonstrates large real-world attention speedups on current state-of-the-art image and video diffusion models. The paper also releases code, reports detailed latency and FLOPs comparisons, includes ablations, and explicitly discusses limitations in Appendix E. These are genuine strengths. However, the central quality-preservation claim is currently supported only by single-run aggregate metrics, and the reported numbers themselves show non-negligible degradations for GRAT-B on several benchmarks, so the strength of the conclusion is not yet commensurate with the evidence.

major comments (5)
  1. [Tab. 2 and Tab. 4] The paper's central claim that GRAT preserves the quality of full attention is not supported for GRAT-B, the variant that delivers the headline speedups. In Table 2, GRAT-B has COCO FID 35.99 vs. 33.89 for full attention, and GenEval overall 0.61 vs. 0.66; in Table 4, GRAT-B has VBench total 80.51% vs. 82.71%. These are consistent quality drops, not matches. Since the abstract and Section 4.1 emphasize 'maintaining the performance of full attention', the paper should either reposition GRAT-B as a quality-speed tradeoff or provide evidence (e.g., human evaluation, repeated runs, or tighter metrics) that these gaps are practically insignificant.
  2. [Section 4, Tables 2 and 4] All quality comparisons rest on single-run metrics with no error bars, confidence intervals, or significance tests. Differences such as GRAT-X VBench total 83.82% vs. 82.71% for full attention, or GRAT-B VBench 80.51% vs. STA 80.46%, are within the range where run-to-run variability in diffusion sampling and benchmark evaluation could flip the comparison. The authors should report multiple seeds or a bootstrap uncertainty estimate before making claims like 'matching' or 'outperforming'.
  3. [Appendix B, Table 6] The default group sizes and block sizes are selected by maximizing GenEval performance in the ablation, and GenEval is then reported as a headline result in Table 2. This selection-on-the-evaluation-metric protocol weakens the claim that GRAT-X (0.65) matches full attention (0.66) on GenEval. The authors should either use a separate held-out split for configuration selection, or explicitly acknowledge that GenEval scores are partially tuned and provide independent validation on another reasoning-heavy benchmark.
  4. [Fig. 2 and Eqs. (5)-(8)] The sparsity motivation is an aggregate statistic averaged over 100 generated images, but GRAT applies one fixed sparsity pattern to every attention layer and every denoising step. Figure 2 does not show how attention locality varies across layers or timesteps, so the paper does not establish that a single group/block/criss-cross pattern is a faithful approximation in all replaced attention modules. Given that Appendix E itself states the patterns are 'manually defined and fixed at inference time', the authors should provide per-layer or per-timestep attention-mass analysis, or at least an ablation measuring attention-mass capture, to address this mismatch between the aggregate motivation and the uniform application.
  5. [Eqs. (7) and (8)] The formal definition of GRAT-X is incorrect as written: the attendable groups are defined as { (m,n) | m=p ∨ n=p }, which uses p for both coordinates and would give row p and column p rather than row p and column q. The intended criss-cross region should be { (m,n) | m=p ∨ n=q }. This indexing error affects the reproducibility of a central variant and should be corrected.
minor comments (5)
  1. [Section 4.2, Qualitative Comparison] The text says 'Fig. 5 presents a qualitative comparison' for the video results, but the video comparison is in Fig. 6; Section 4.1 already uses Fig. 5 for images.
  2. [Table 3] The Neighborhood Attention row cites 'NA [32]' but the correct reference is [11] (Hassani et al.).
  3. [Abstract and Fig. 3] The 35.8x speedup is for the attention operation only, not end-to-end generation (which the paper reports as 9.2x in Table 1). The abstract and Figure 3 should state this explicitly to avoid overclaiming.
  4. [Tables 1 and 3] The 'FLOPs sparsity' column is described only in the caption; the paper should define it explicitly as the percentage reduction in attention FLOPs relative to full attention, since it is used as a primary efficiency measure.
  5. [Section 4.2, Table 4] The sentence claiming GRAT-B 'surpasses' STA with a VBench total score of 80.51% vs. 80.46% should be softened; a 0.05-point gap is within likely measurement noise, and this is exactly the kind of comparison that needs error bars.

Circularity Check

1 steps flagged · score 2.0 of 10

No equation-level circularity; one mild selection-bias step where GenEval-guided defaults are later reported on GenEval.

  1. fitted input called prediction [Appendix B (Table 6 and surrounding text) vs. Section 4.1 (Table 2)]
    "We evaluate the design choices of GRAT using Flux [22], reporting FLOPs sparsity, attention latency, and performance on the GenEval benchmark [10]. ... Our default configurations are highlighted in gray. ... By default, GRAT-B uses a group size of 16 × 16 and b = 1, balancing speed and performance effectively. ... Similarly, for GRAT-X, we adopt a default group size of 16 × 16, which achieves a balanced tradeoff between speed and generation quality."

    The default GRAT-B/GRAT-X configurations are selected in Table 6 using GenEval performance as the criterion. The same GenEval benchmark is then reported in Table 2 and Section 4.1 as evidence that GRAT-X matches full attention (0.65 vs. 0.66). Thus the headline GenEval comparison partly reports the selection criterion rather than an out-of-sample prediction. This is a mild selection-bias circularity, not a derivation-level reduction: the speedups and the COCO, MJHQ-30K, and VBench results are independent of this choice.

full rationale

GRAT's central claim is not circular in the equation sense. The method is an explicit, training-free sparse-attention mask defined by Eqs. (2)-(15) and applied to frozen pretrained models; the reported attention speedups are wall-clock measurements, and quality is assessed on external benchmarks (COCO2014, MJHQ-30K, VBench) that do not depend on the GRAT hyperparameter selection. The Fig. 2 locality statistic (69% of attention mass within 0.2 normalized distance) is empirical motivation for the design, not a fitted parameter from which the final quality numbers are derived. Self-citations in the reference list are related-work or backbone citations and are not load-bearing for the acceleration claim. The paper's own limitation section honestly states that the grouping and structured patterns are 'manually defined and fixed at inference time,' which is a robustness concern rather than a circularity. The only notable circularity-adjacent issue is that the default group/block sizes are chosen by ablation on GenEval (Table 6) and then GenEval is used as a headline quality benchmark (Table 2), giving that particular comparison a mild selection-bias component. This does not undermine the independent speed and other-benchmark evidence, so the overall circularity score is 2.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities; the grouping and structured regions are algorithmic choices rather than postulated objects. The main burdens are the empirical assumptions about attention sparsity and the fixed pattern's sufficiency, plus the hyperparameters chosen by ablation.

free parameters (3)
  • image group size g = 16x16
    Default for Flux; chosen from ablation on GenEval (Table 6) balancing speed and quality; larger groups increase GPU parallelism but reduce attention flexibility.
  • GRAT-B block radius b = 1
    Default; ablation shows b=2 or 3 improves GenEval by 0.02 at modest latency cost (Table 6).
  • video group size = 4x8x8
    Used for HunyuanVideo (temporal x height x width); no ablation reported for this choice.
assumptions (3)
  • domain assumption Pretrained diffusion transformers have attention maps that are predominantly sparse and local, with roughly 69% of attention mass within 0.2 normalized distance.
    Sect. 1, Fig. 2. This motivates replacing full attention with fixed local and structured patterns; if false, the method would lose essential information.
  • domain assumption All query tokens within a g x g spatial group can share a single key-value set without meaningful quality loss.
    Sect. 3.2, Eq. 2-8. This is the core grouping assumption; not derived from first principles, only validated empirically.
  • domain assumption A fixed pattern of surrounding blocks or criss-cross rows and columns captures the long-range dependencies needed for global coherence across all layers and denoising steps.
    Sect. 3.2, GRAT-X definition; acknowledged in Appendix E as a limitation (manually defined, fixed at inference time).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformers." pith.science (2026). https://pith.science/paper/HWI7UAYC

@misc{pith2026250514687,
  author       = {Pith},
  title        = {Pith review of: Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HWI7UAYC}},
  note         = {Machine review of arXiv:2505.14687}
}
abstract

Diffusion-based Transformers have demonstrated impressive generative capabilities, but their high computational costs hinder practical deployment, for example, generating an $8192\times 8192$ image can take over an hour on an A100 GPU. In this work, we propose GRAT (\textbf{GR}ouping first, \textbf{AT}tending smartly), a training-free attention acceleration strategy for fast image and video generation without compromising output quality. The key insight is to exploit the inherent sparsity in learned attention maps (which tend to be locally focused) in pretrained Diffusion Transformers and leverage better GPU parallelism. Specifically, GRAT first partitions contiguous tokens into non-overlapping groups, aligning both with GPU execution patterns and the local attention structures learned in pretrained generative Transformers. It then accelerates attention by having all query tokens within the same group share a common set of attendable key and value tokens. These key and value tokens are further restricted to structured regions, such as surrounding blocks or criss-cross regions, significantly reducing computational overhead (e.g., attaining a \textbf{35.8$\times$} speedup over full attention when generating $8192\times 8192$ images) while preserving essential attention patterns and long-range context. We validate GRAT on pretrained Flux and HunyuanVideo for image and video generation, respectively. In both cases, GRAT achieves substantially faster inference without any fine-tuning, while maintaining the performance of full attention. We hope GRAT will inspire future research on accelerating Diffusion Transformers for scalable visual generation.

Figures

Figures reproduced from arXiv: 2505.14687 by the authors.

Figure 1
Figure 1. Fast high-resolution image and video generation enabled by equipping Flux [22] and HunyuanVideo [21] with the proposed GRAT, a training-free attention acceleration strategy. GRAT significantly improves inference speed without any fine-tuning or degradation in visual quality. Abstract Diffusion-based Transformers have demonstrated impressive generative capabili￾ties, but their high computational costs hinder practica… view at source ↗
Figure 2
Figure 2. Attention Visualization of Flux [22]. Left: A single query token (marked in red) attends only to sparse, local regions. Middle: Visualization of attention maps across all query tokens in an image (each row corresponds to a query). The patterns consistently demonstrate sparsity and locality. Right: Distribution of attention scores, averaged over 100 generated images, plotted as a function of the normalized spatial di… view at source ↗
Figure 3
Figure 3. Comparison of Attention Schemes. The comparison is based on Flux [22] with various attention mechanisms, including Full Attention [47], Neighborhood Attention [39, 11], and the proposed GRAT-B and GRAT-X. FLOPs Sparsity measures the theoretical reduction in compute relative to Full Attention (0% indicates no reduction). Inference Speedup reflects real-world speedup on an A100 GPU, relative to Full Attention (1× mean… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Illustration of Attention Operations. Query tokens are shown in red, and their corre￾sponding attended regions (key and value tokens) are highlighted in light blue. (a) Full Attention [47]: each query attends to the entire feature map. (b) Neighborhood Attention [39, 1…
Figure 5
Figure 5. Figure 5: Generated Images by Flux [22] with Different Attention Mechanisms. We compare the visual results of Full Attention [47] (i.e., the original Flux), Neighborhood Attention (NA) [11], CLEAR [32], and our proposed GRAT-B and GRAT-X. (NA) [11]—exhibit significant quality de…
Figure 6
Figure 6. Figure 6: Generated Videos by HunyuanVideo [21] with Different Attention Mechanisms. We compare the visual results of Full Attention [47] (i.e., the original HunyuanVideo), STA [59], and our proposed GRAT-B and GRAT-X. Quantitative Comparison. Tab. 4 presents quantitative result…
Figure 7
Figure 7. Figure 7: Generated Images by Flux [22] Enhanced with Our GRAT Attention. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Generated Images by Flux [22] Enhanced with Our GRAT Attention. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Generated Videos by HunyuanVideo [21] Enhanced with Our GRAT Attention. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ACID: Adaptive Caching for vIDeo generation

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Adaptive threshold switching on the drift signal's rate of change expands the quality-vs-speed Pareto frontier of TeaCache, EasyCache, and DiCache across HunyuanVideo, Wan 2.1, and CogVideoX.

Reference graph

Works this paper leans on

59 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    All are worth words: A vit backbone for diffusion models

    Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In CVPR, 2023

    2023

  2. [2]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  3. [3]

    Pixart- σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In ECCV, 2024

    work page 2024

  4. [4]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023

    arXiv 2023

  5. [5]

    Coconut-pancap: Joint panoptic segmentation and grounded captions for fine-grained understanding and generation

    Xueqing Deng, Qihang Yu, Ali Athar, Chenglin Yang, Linjie Yang, Xiaojie Jin, Xiaohui Shen, and Liang-Chieh Chen. Coconut-pancap: Joint panoptic segmentation and grounded captions for fine-grained understanding and generation. arXiv preprint arXiv:2502.02589, 2025

    arXiv 2025

  6. [6]

    Flex attention: A programming model for generating optimized attention kernels

    Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A programming model for generating optimized attention kernels. arXiv preprint arXiv:2412.05496, 2024

    arXiv 2024

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021

    2021

  8. [8]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024

    2024

Show all 59 references
  1. [9]

    Structural pruning for diffusion models

    Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. NeurIPS, 2023

    2023

  2. [10]

    Geneval: An object-focused framework for evaluating text-to-image alignment

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. NeurIPS, 2023

    2023

  3. [11]

    Neighborhood attention transformer

    Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In CVPR, 2023

    2023

  4. [12]

    Maxtron: Mask transformer with trajectory attention for video panoptic segmentation

    Ju He, Qihang Yu, Inkyu Shin, Xueqing Deng, Alan Yuille, Xiaohui Shen, and Liang-Chieh Chen. Maxtron: Mask transformer with trajectory attention for video panoptic segmentation. arXiv preprint arXiv: 2311.18537, 2023

    2023 arXiv

  5. [13]

    Flowtok: Flowing seamlessly across text and image tokens

    Ju He, Qihang Yu, Qihao Liu, and Liang-Chieh Chen. Flowtok: Flowing seamlessly across text and image tokens. arXiv preprint arXiv:2503.10772, 2025

    2025

  6. [14]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022

    2022

  7. [15]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS, 2017

    2017

  8. [16]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020

    2020

  9. [17]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In CVPR, 2024

    2024

  10. [18]

    On architectural compression of text-to-image diffusion models, 2024

    Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells, and Shinkook Choi. On architectural compression of text-to-image diffusion models, 2024. URL https://arxiv.org/abs/2305.15798

    2024 arXiv

  11. [19]

    Democratizing text-to-image masked generative models with compact text-aware one-dimensional tokens

    Dongwon Kim, Ju He, Qihang Yu, Chenglin Yang, Xiaohui Shen, Suha Kwak, and Liang-Chieh Chen. Democratizing text-to-image masked generative models with compact text-aware one-dimensional tokens. arXiv preprint arXiv:2501.07730, 2025

    2025 arXiv

  12. [20]

    Auto-encoding variational bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014. 10

    2014

  13. [21]

    Hunyuanvideo: A systematic framework for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  14. [22]

    Flux: Official inference repository for flux.1 models, 2024

    Black Forest Labs. Flux: Official inference repository for flux.1 models, 2024. URL https://github. com/black-forest-labs/flux . Accessed: 2024-11-12

    2024

  15. [23]

    Set transformer: A framework for attention-based permutation-invariant neural networks

    Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In ICML, 2019

    2019

  16. [24]

    Koala: Empirical lessons toward memory-efficient and fast diffusion models for text-to-image synthesis

    Youngwan Lee, Kwanyong Park, Yoorhim Cho, Yong-Ju Lee, and Sung Ju Hwang. Koala: Empirical lessons toward memory-efficient and fast diffusion models for text-to-image synthesis. NeurIPS, 2024

    2024

  17. [25]

    Playground v2

    Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402.17245, 2024

    2024 arXiv

  18. [26]

    Q-diffusion: Quantizing diffusion models

    Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In ICCV, 2023

    2023

  19. [27]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014

    2014

  20. [28]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In ICLR, 2023

    2023

  21. [29]

    Pseudo numerical methods for diffusion models on manifolds

    Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In ICLR, 2022

    2022

  22. [30]

    Alleviating distortion in image generation via multi-resolution diffusion models and time-dependent layer normalization.NeurIPS, 2024

    Qihao Liu, Zhanpeng Zeng, Ju He, Qihang Yu, Xiaohui Shen, and Liang-Chieh Chen. Alleviating distortion in image generation via multi-resolution diffusion models and time-dependent layer normalization.NeurIPS, 2024

    2024

  23. [31]

    Revision: High-quality, low-cost video generation with explicit 3d physics modeling for complex motion and interaction

    Qihao Liu, Ju He, Qihang Yu, Liang-Chieh Chen, and Alan Yuille. Revision: High-quality, low-cost video generation with explicit 3d physics modeling for complex motion and interaction. arXiv preprint arXiv:2504.21855, 2025

    2025

  24. [32]

    Clear: Conv-like linearization revs pre-trained diffusion transformers up

    Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Clear: Conv-like linearization revs pre-trained diffusion transformers up. arXiv preprint arXiv:2412.16112, 2024

    2024 arXiv

  25. [33]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR, 2023

    2023

  26. [34]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021

    2021

  27. [35]

    Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers

    Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In ECCV, 2024

    2024

  28. [36]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 2019

    2019

  29. [37]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023

    2023

  30. [38]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021

    2021

  31. [39]

    Stand-alone self-attention in vision models

    Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. NeurIPS, 2019

    2019

  32. [40]

    Beyond next-token: Next-x prediction for autoregressive visual generation

    Sucheng Ren, Qihang Yu, Ju He, Xiaohui Shen, Alan Yuille, and Liang-Chieh Chen. Beyond next-token: Next-x prediction for autoregressive visual generation. arXiv preprint arXiv:2502.20388, 2025

    2025 arXiv

  33. [41]

    Flowar: Scale-wise autoregressive image generation meets flow matching

    Sucheng Ren, Qihang Yu, Ju He, Xiaohui Shen, Alan Yuille, and Liang-Chieh Chen. Flowar: Scale-wise autoregressive image generation meets flow matching. In ICML, 2025. 11

    2025

  34. [42]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022

    2022

  35. [43]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022

    2022

  36. [44]

    Post-training quantization on diffusion models

    Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. In CVPR, 2023

    2023

  37. [45]

    Deeply supervised flow-based generative models

    Inkyu Shin, Chenglin Yang, and Liang-Chieh Chen. Deeply supervised flow-based generative models. arXiv preprint arXiv:2503.14494, 2025

    2025 arXiv

  38. [46]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020

    2011 arXiv

  39. [47]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017

    2017

  40. [48]

    Wan: Open and advanced large-scale video generative models

    Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pin...

    2025 arXiv

  41. [49]

    Axial- deeplab: Stand-alone axial-attention for panoptic segmentation

    Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Axial- deeplab: Stand-alone axial-attention for panoptic segmentation. In ECCV, 2020

    2020

  42. [50]

    Linformer: Self-attention with linear complexity, 2020

    Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, 2020. URL https://arxiv.org/abs/2006.04768

    2020 arXiv

  43. [51]

    Ppt: Token pruning and pooling for efficient vision transformers

    Xinjian Wu, Fanhu Zeng, Xiudong Wang, and Xinghao Chen. Ppt: Token pruning and pooling for efficient vision transformers. arXiv preprint arXiv:2310.01812, 2023

    2023 arXiv

  44. [52]

    Revealing the dark secrets of masked image modeling

    Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. In CVPR, 2023

    2023

  45. [53]

    Imagereward: Learning and evaluating human preferences for text-to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. NeurIPS, 2023

    2023

  46. [54]

    1.58-bit flux

    Chenglin Yang, Celong Liu, Xueqing Deng, Dongwon Kim, Xing Mei, Xiaohui Shen, and Liang-Chieh Chen. 1.58-bit flux. arXiv preprint arXiv:2412.18653, 2024

    2024 arXiv

  47. [55]

    Randomized autoregressive visual generation

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Randomized autoregressive visual generation. arXiv preprint arXiv:2411.00776, 2024

    2024 arXiv

  48. [56]

    An image is worth 32 tokens for reconstruction and generation

    Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. NeurIPS, 2024

    2024

  49. [57]

    Ditfastattn: Attention compression for diffusion transformer models

    Zhihang Yuan, Hanling Zhang, Lu Pu, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. Ditfastattn: Attention compression for diffusion transformer models. NeurIPS, 2024

    2024

  50. [58]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. NeurIPS, 2020

    2020

  51. [59]

    Hello GRAT

    Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhenghong Liu, and Hao Zhang. Fast video generation with sliding tile attention. arXiv preprint arXiv:2502.04507, 2025. 12 Appendix In the appendix, we provide the following additional information: • Detailed ...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.