Pith. sign in

REVIEW 5 major objections 5 minor 70 references

PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that reordering tokens to make attention maps block-wise lets sparse and quantized attention match full-precision generation quality while cutting latency by 1.9-2.7x.

desk verdict A practical, well-tested method for unifying visual attention patterns via offline-chosen axis permutations — worth reviewing, but the missing SparseVideoGen2 comparison and “lossless” wording need fixing. read the letter →

arxiv 2506.16054 v1 pith:2INBEHDX submitted 2025-06-19 cs.CV cs.GR

Tianchen Zhao , Ke Hong , Xinhao Yang , Xuefeng Xiao , Huixia Li , Feng Ling , Ruiqi Xie , Siqi Chen
show 3 more authors
Hongyu Zhu Yichong Zhang Yu Wang
This is my paper
classification cs.CVcs.GR
keywords attentionsparsificationquantizationtokenreorderingdiffusiontransformersvideogenerationimageblocksparsityinteger
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the real obstacle to compressing attention in visual generation models is the layout of the attention map, not the absence of clever sparse masks or quantizers. In diffusion transformers, 3D image and video features are flattened into 1D token sequences, so heads that aggregate along different spatial axes produce multi-diagonal and dispersed attention patterns that are hard to sparsify or quantize. The proposed method, PARO, reorders tokens per head so that all these patterns collapse into a single block-wise structure, chosen offline from six axis permutations; the resulting static block mask and block-wise INT8/INT4 quantization then operate at 20-30% density without the quality collapse seen in prior baselines. On CogVideoX and Flux the authors report outputs nearly identical to the FP16 full-attention baseline, with end-to-end latency speedups of 1.9-2.7x. A sympathetic reader would care because it converts a hard pattern-matching engineering problem into a cheap layout change that any sparsity or quantization scheme can exploit.

What carries the argument

The load-bearing mechanism is PARO (Pattern-Aware token ReOrdering), a per-head permutation of the flattened token sequence restricted to the six axis orders of $[F,H,W]$, chosen offline by a metric that balances block sparsifiability (fraction of $64 \times 64$ blocks that are mostly below a small threshold) against quantization incoherence (max over mean absolute value inside each block). It converts multi-diagonal attention into block-wise attention; the companion machinery is a static, timestep-aware sparse mask (block-sum thresholding, shared after early timesteps, stored as bitmasks with prefetch) and block-wise INT8/INT4 quantization aligned to the FlashAttention block size. The permutation is fused into preceding kernels so its runtime overhead stays below 1%.

What would settle it

Use a held-out prompt set deliberately different from the calibration prompts (e.g., text-heavy scenes or unusual camera motion), run PAROAttention at 20-30% density or INT4, and compare outputs to the FP16 baseline. If the attention maps under the static permutation no longer concentrate into block-sparse regions (measured by a drop in the sparse-block fraction or a rise in block incoherence), and PSNR or cosine similarity degrade far below the values reported in the paper's tables, the generalization claim is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that the apparently diverse attention patterns in visual generation models—block-wise, multi-diagonal, diagonal-in-block—are the same phenomenon: local aggregation of neighboring pixels along different axes of a feature volume that has been flattened into a 1D sequence. By permuting the token order so that the aggregation axis becomes contiguous, the paper unifies all patterns into a regular block-wise form. This one reorganization makes both compression techniques simple: a static block mask selected by block-sum thresholding (shared across prompts, and across later timesteps), and block-wise INT8/INT4 quantization of the post-softmax attention map, whose block incoherence falls from roughly 200-1200 to 12-20 after reordering. The paper's headline evidence is that on CogVideoX-5B and Flux, PAROAttention keeps quality metrics and pixel/feature-space similarity close to the FP16 dense baseline at 20-30% density and INT8/INT4 bitwidth, while the CUDA kernels deliver 1.9-2.7x end-to-end speedups.

Load-bearing premise

Everything rests on the offline choice of one permutation per head staying correct for prompts, timesteps, and CFG settings that were not seen at calibration time; if that permutation—and the static masks built on it—stops matching the true attention, the reported quality and speedup numbers do not transfer.

Editorial extensions

If this is right

  • Sparse and quantized attention for visual DiTs can be designed around one unified block pattern rather than per-head specialized masks; the change is a layout swap, not a new compression algorithm.
  • At 20-30% density, PAROAttn matches or beats the quality of 50%-density baselines, so practical video generation moves to much lower density.
  • Quantizing the attention map to INT8/INT4 becomes viable, letting attention run on integer tensor cores and non-GPU accelerators rather than staying at FP8/FP16.
  • Because permutation also helps dynamic sparsity (SpargeAttn with PARO at 30% matches its 50% quality), reordering is a drop-in preprocessing step for other compression methods.
  • End-to-end inference of CogVideoX and Flux gains 1.9-2.7x with under 1% overhead and no metric loss, making long-video and high-resolution generation more deployable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An immediate extension is adaptive permutation: a small predictor on prompt or timestep features could pick the order per head, extending the same gains to distributions where pattern type varies more than the calibration suggests.
  • Because the pattern diversity is partly an artifact of token layout, training models with axis-aligned attention heads might produce attention that is natively block-sparse and low-incoherence, without any post-training permutation.
  • The reported speedups are tied to 64x64 block alignment on the target GPU; on other hardware or tile sizes the optimal density and bitwidth would need re-tuning, and end-to-end gains could differ.
  • The incoherence reduction from roughly 200-1200 to 12-20 suggests a general recipe: any compression scheme struggling with outliers inside quantization groups can first permute data to group similar values, which may transfer to KV-cache quantization in other models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes PAROAttention, a post-training compression method for diffusion transformers in visual generation. It observes that visual attention maps have diverse, dispersed patterns that make block sparsity and PV quantization difficult, and proposes an offline per-head token permutation chosen from the six axis orders to unify these patterns into hardware-friendly block-wise forms. On top of this reordering, it applies static timestep-aware sparse masks and block-wise INT8/INT4 quantization, and implements fused CUDA kernels. Experiments on CogVideoX, Wan 2.1, and Flux.1.Dev report near-baseline quality at 20-50% density and INT8/INT4 bitwidth, with 1.9-2.7x end-to-end and up to 9.28x attention speedups. Appendices provide additional results on Wan, an ablation of SparseVideoGen's skipping scheme, overhead measurements, and visualizations of attention stability across timesteps and prompts.

Significance. If the stability premise holds, the contribution is significant: it converts irregular visual attention patterns into a simple block-wise form, making both static sparsity and block-wise quantization easier and more hardware-friendly, and the permutation idea is compatible with dynamic sparsity methods. The paper's strengths include the analysis of why visual attention patterns hinder sparsity and quantization, the combined sparse/quantization permutation metric in Eq. (3), detailed overhead measurements (permutation 0.03%, prefetch 0.33%), and consistent empirical trends across three models. However, the core generalization claim for offline-chosen permutations is supported mainly by qualitative or underspecified evidence, the 'lossless' wording overstates the reported FP-difference metrics, and the efficiency-accuracy trade-off is not accompanied by error bars or released artifacts. The idea is novel and likely useful, but the paper needs a quantitative stability analysis and more careful claims before acceptance.

major comments (5)
  1. [Sec. 4.1, Eq. (3); Appendix F] The central premise that one offline-determined permutation per head remains optimal across unseen prompts, all timesteps, CFG branches, and the altered attention distribution is not quantitatively established. Sec. 4.2 reports cosine similarity >= 0.99 for image-token attention across prompts, but does not state the number of prompts, heads, or timesteps, nor whether the similarity is computed on raw attention maps, block-sparsity patterns, or the M metric of Eq. (3). Appendix F says only that the 'type' of pattern remains consistent while 'detailed attention pattern may vary,' which does not show that the argmin permutation is stable. If the optimal permutation drifts for even a fraction of heads or timesteps, the static masks and fused permutation misalign with actual attention, and the reported PSNR values in Tables 1 and 4 would not transfer. I request a quantitative analysis: for held-out prompts, report the fraction of heads and timesteps where the selected permutation is not the argmin of M, and the resulting quality drop when the static permutation is enforced.
  2. [Abstract; Sec. 5.2, Tables 1 and 2] The term 'lossless metrics' overstates the reported results. In Table 1, PAROAttn(0.3) has PSNR 22.89, SSIM 0.829, CosSim 0.984, and FVD-FP16 0.153 relative to the FP16 baseline, while PAROAttn(0.2) has PSNR 19.39, SSIM 0.744, and CosSim 0.962. These are substantial pixel-level differences, not lossless. The quality metrics (CLIPSIM, VQA) are nearly unchanged, but the abstract's 'lossless metrics' and Sec. 1's 'generation with lossless metrics' should be qualified as 'no significant degradation on quality metrics' with the FP-difference metrics reported explicitly and with error bars or multiple seeds.
  3. [Sec. 5.2, Table 1; Appendix B, Fig. 10] The main comparison handicaps SparseVideoGen by disabling its skipping scheme. Appendix B shows that SparseVideoGen(0.5) with skipping achieves PSNR 25.37, while the main table reports PSNR 18.50 without skipping. Disabling skipping may be a legitimate fair-comparison choice given that PAROAttn applies to all timesteps, but the main text should present both configurations and explicitly justify the choice, because the claimed superiority over SparseVideoGen at 50% density partly reflects this ablation rather than the sparse-mask design itself. As written, a reader comparing Table 1 with the SparseVideoGen paper could draw a misleading conclusion about relative performance.
  4. [Sec. 4.1, Eq. (3); Sec. 5.1] Key hyperparameters are under-specified, preventing reproduction and robustness assessment. The paper gives example values for epsilon (1e-3) and sigma (90%) in Sec. 4.2, but does not report the exact values used for each model, the weighting coefficient alpha in Eq. (3), the block size b (stated only as 'chosen as 64'), or the precise timestep split for mask sharing (described only as 'first half'). The calibration set is the first 2 prompts of the CogVideo example dataset, and the evaluation uses a subset of VBench prompts and 1024 COCO prompts, with no error bars across prompts or seeds. Please provide a hyperparameter table and report variance across evaluation prompts.
  5. [Sec. 5.1; App. C] No code, kernels, masks, or permutation files are released, and the CUDA kernel description is high-level. Given that the central speedup claims depend on a custom fused permutation, prefetching, and block-skipping scheme, the absence of artifacts makes the efficiency numbers difficult to verify. At minimum, the per-head permutation orders and sparse masks for each model should be released, or the kernel pseudocode should be detailed enough to reimplement independently.
minor comments (5)
  1. [Sec. 1] The word 'subpotimal' in the third paragraph should be 'suboptimal'.
  2. [Eq. (1)] The notation n^{<epsilon}_{ij} is introduced without explaining the superscript; please clarify that this counts entries below epsilon and connect it to the block-sum threshold used for static mask generation.
  3. [Table 3] LPIPS appears in the ablation table but is not defined in Sec. 5.1; please add a definition or citation for it.
  4. [Appendix D, Table 4] The omission of SpargeAttn results for Wan 2.1 due to NaN outputs should be noted in the main table or in a footnote to Table 4, not only in the appendix, so that readers do not misread the comparison as more complete than it is.
  5. [Fig. 4 and Fig. 5] The qualitative claims of 'nearly identical' frames would be strengthened by difference maps or per-image error metrics; the current visual comparisons are not quantified.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular reduction found: the permutation, masks, and quantization settings are calibrated on a disjoint prompt set and evaluated on held-out prompts; the only self-citation (Vidit-Q for the incoherence metric) is not load-bearing.

full rationale

The derivation chain is self-contained in the relevant sense. The per-head permutation is chosen offline by minimizing the metric M in Eq. (3) on a calibration set consisting of the first 2 prompts of the CogVideo example dataset (Sec. 5.1), and the reported quality numbers are measured on held-out prompts (a VBench subset for video, the first 1024 COCO prompts for image). No fitted constant targets the reported PSNR, SSIM, CosSim, FVD, or FID metrics; the static masks and INT8/INT4 scales are generated from calibration-time post-softmax attention maps rather than by inverting the evaluation metrics. The self-citation to Vidit-Q [69] for the 'incoherence' quantization metric is shared with the external QuaRot work [4] and functions as an explanatory design metric, not as an axiom that defines the result; its effect is independently verified through measured incoherence reduction (App. E) and end-to-end generation quality. The stability of the offline permutation across timesteps, prompts, and CFG settings is asserted with visual evidence (Sec. 4.1, App. F), and a generalization failure would be a correctness or robustness risk rather than a circularity, because the central claim is not defined in terms of that stability assumption. Accordingly, no step in the claimed derivation reduces by construction to its own inputs.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method rests on an empirical assumption that visual attention is locally aggregated along one of the three axes, and that a single offline permutation per head stays valid across prompts and timesteps. Several hyperparameters (block size, epsilon, sigma, alpha, timestep split, calibration count) are chosen manually. No invented entities are introduced.

free parameters (6)
  • block size b = 64
    Set to match FlashAttention block size; not optimized, chosen by design.
  • sparse threshold epsilon = 1e-3
    Used in Eq. 1 to classify a block as sparse; manually chosen small value.
  • sparse fraction sigma = 90%
    Fraction of values below epsilon required for a block to be marked sparse; manually chosen.
  • metric weight alpha = not reported
    Balances sparse metric and quant metric in Eq. 3; value not stated in the text.
  • timestep split for mask sharing = first half vs second half
    Distinct masks in early timesteps, one shared mask later; split chosen without sensitivity analysis.
  • calibration prompt count = 2
    Permutation and masks determined from first 2 prompts of CogVideo example dataset.
assumptions (5)
  • standard math Softmax attention is equivariant under a simultaneous permutation of token rows and columns, so permuting tokens before QKV projection and inverse-permuting the output is exact.
    Justifies the claim that reordering causes no approximation beyond mask and quantization; used in Sec 4.1.
  • domain assumption Visual attention is locally aggregated: attention heads tend to connect neighboring pixels, so diverse patterns reflect local aggregation along different dimensions.
    Motivates restricting reorderings to the six axis permutations; Sec 1 and Sec 4.1.
  • ad hoc to paper One fixed permutation per head, chosen offline, remains optimal across prompts, timesteps, and CFG settings.
    Load-bearing for static masks and fused permutation; Sec 4.1 and Appendix F.
  • ad hoc to paper Post-softmax attention maps from one or two calibration prompts are representative enough to generate masks for all inference prompts.
    Calibration uses first 2 prompts; generalization claimed from cosine similarity >=0.99 without distributional analysis.
  • domain assumption Fusing permutation into preceding kernels adds negligible overhead and does not change numerical results.
    Overhead measured 0.03% for RoPE fusion, but relies on specific kernel implementation not released.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models." pith.science (2026). https://pith.science/paper/2INBEHDX

@misc{pith2026250616054,
  author       = {Pith},
  title        = {Pith review of: PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2INBEHDX}},
  note         = {Machine review of arXiv:2506.16054}
}
read the original abstract

In visual generation, the quadratic complexity of attention mechanisms results in high memory and computational costs, especially for longer token sequences required in high-resolution image or multi-frame video generation. To address this, prior research has explored techniques such as sparsification and quantization. However, these techniques face significant challenges under low density and reduced bitwidths. Through systematic analysis, we identify that the core difficulty stems from the dispersed and irregular characteristics of visual attention patterns. Therefore, instead of introducing specialized sparsification and quantization design to accommodate such patterns, we propose an alternative strategy: *reorganizing* the attention pattern to alleviate the challenges. Inspired by the local aggregation nature of visual feature extraction, we design a novel **Pattern-Aware token ReOrdering (PARO)** technique, which unifies the diverse attention patterns into a hardware-friendly block-wise pattern. This unification substantially simplifies and enhances both sparsification and quantization. We evaluate the performance-efficiency trade-offs of various design choices and finalize a methodology tailored for the unified pattern. Our approach, **PAROAttention**, achieves video and image generation with lossless metrics, and nearly identical results from full-precision (FP) baselines, while operating at notably lower density (~20%-30%) and bitwidth (**INT8/INT4**), achieving a **1.9x** to **2.7x** end-to-end latency speedup.

Figures

Figures reproduced from arXiv: 2506.16054 by the authors.

Figure 1
Figure 1. PAROAttention unifies the diverse attention patterns through token reorder, which benefits both the sparsification and quantization. It achieves nearly identical generation result from full￾precision baseline without metrics degradation, under lower density (20%-30%) and bitwidth (INT8/INT4), achieving a 1.9∼2.7× end-to-end latency speedup. Abstract In visual generation, the quadratic complexity of attention mechani… view at source ↗
Figure 2
Figure 2. The Motivation of PAROAttention. (a) The computational flow of transformer. (b) The challenge for sparsification and quantization caused by visual attention pattern, and how PAROAtten￾tion addresses it. (c) The illustration of 3D feature, and 1D token sequence with different orders. (FP) baselines, while operating at notably lower density (20%-30%) and bitwidth (INT8/INT4), achieving a 1.9∼2.7× end-to-end latency sp… view at source ↗
Figure 3
Figure 3. The overall framework of PAROAttention. The pattern-aware token reordering (PARO) is applied to unify the attention pattern into hardware-friendly block pattern. Sparse attention and quantization techniques are designed tailored for this pattern. Metric for Permutation Order: As discussed above, sparsification and quantization prefer different distributions. We design separate metrics and combine them to to determin… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Qualitative Results of CogVideoX generated videos for PAROAttention and baselines. 5 Experiments 5.1 Experimental Setup Video and Image Generation: For video generation, we apply PAROAttn to the CogVideoX-5B [56] and Wan [41] (see Appendix Sec.1) model for 720P 6/10-se…
Figure 5
Figure 5. Figure 5: Qualitative Results of Flux generated images for PAROAttention and baseline methods. (2) Relative Difference Metrics: They quantify the difference between FP16 generation. For both video and image generation, PSNR and cosine similarity are used to measure low-level pix…
Figure 6
Figure 6. Figure 6: Normalized latency speedup (bar plot) and PSNR (line plot) trade-off under different (a) quantization and (b) sparse configurations. the P V to lower-bit integers (e.g., PAROAttn (INT8) vs. SageAttn, and PAROAttn (INT4) vs. SageAttnV2). (5) The PAROAttn sparsification …
Figure 7
Figure 7. Figure 7: Attention patterns under dif￾ferent permute orders [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Qualitative results of Wan 2.1 model video generation. PSNR: - VQA: 92.53 Speedup: 1.00x FP16 Full Attn. (100%) PSNR: 29.14 VQA: 92.56 Speedup: 1.73x PAROAttn. (50%) PSNR: 22.90 VQA: 92.66 Speedup: 2.71x PAROAttn. (30%) PSNR: 16.80 VQA: 87.72 Speedup: 1.67x SpargeAttn.…
Figure 9
Figure 9. Figure 9: Additional qualitative results of sparsification for CogVideoX. Ablation study on skipping scheme in SparseVideoGen. In the original SparseVideoGen paper and its official code release, sparsification is deliberately omitted for the first two Transformer blocks and the …
Figure 10
Figure 10. Figure 10: Ablation of skipping timestep and transformer blocks for SparseVideoGen. PSNR: - VQA: 92.53 Speedup: 1.00x FP16 Full Attn. (100%) PSNR: 18.50 VQA: 90.14 Speedup: 1.42x SparseVideoGen. (50%, w. Skip) SparseVideoGen. (50%, w.o. Skip) PSNR: 25.37 VQA: 91.89 Speedup: 1.42…
Figure 12
Figure 12. Figure 12: Comparison of attention masks for PAROAttention and baseline sparse attention methods. We present the relative difference metrics (L1 Norm, RMSE, CosSim) to measure the difference between the original and masked attention map. E The Effectiveness of PAROAttention Quan…
Figure 13
Figure 13. Figure 13: Incoherence for data within the quantization group before and after permutation [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Quantization error with respect to FP for quantization of the attention map P. The red “FP” stands for the FP8 quantization error. leading to a notable decrease in quantization error. As shown in [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Visualization of post softmax attention pattern for different timesteps, prompts, and classifier-free-guidance (CFG). The metric scores are calculated relative to the attention pattern with red text. G Additional Visualization of Permutation for Flux We present the at…
Figure 16
Figure 16. Figure 16: Visualization of the attention pattern for flux under different permutation. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

70 extracted references · 37 canonical work pages

  1. [54]

    Sparse videogen2: Accelerate video generation with sparse attention via semantic-aware permutation

    Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Jintao Zhang, Han Cai, Yujun Lin, Xiuyu Li, Chenfeng Xu, Kelly Peng, et al. Sparse videogen2: Accelerate video generation with sparse attention via semantic-aware permutation. arXiv preprint arXiv:2505.18875, 2025. 3

    arXiv 2025

  2. [1]

    https://resources.nvidia.com/en-us-tensor-core ,

    Tensor core. https://resources.nvidia.com/en-us-tensor-core ,. 19

    work page

  3. [2]

    https://docs.amd.com/r/2021.2-English/ug1483-model-composer-sys-gen-user-guide/ DSP48E,

    Xilinx dsp. https://docs.amd.com/r/2021.2-English/ug1483-model-composer-sys-gen-user-guide/ DSP48E,. 19

    work page 2021

  4. [3]

    Moba: Mixture of block attention

    Moonshot AI. Moba: Mixture of block attention. Technical report, MoonshotAI, 2025. 2, 3

    work page 2025

  5. [4]

    Quarot: Outlier-free 4-bit inference in rotated llms

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456, 2024. 6

    arXiv 2024

  6. [5]

    Rainfusion: Adaptive video generation acceleration via multi-dimensional visual redundancy

    Aiyue Chen, Bin Dong, Jingru Li, Jing Lin, Yiwu Yao, and Gongyi Wang. Rainfusion: Adaptive video generation acceleration via multi-dimensional visual redundancy. arXiv preprint arXiv:2505.21036, 2025. 3

    arXiv 2025

  7. [6]

    Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023. 3

    work page 2023

  8. [7]

    Hardware acceleration of sparse and irregular tensor computations of ml models: A survey and insights

    Shail Dave, Riyadh Baghdadi, Tony Nowatzki, Sasikanth Avancha, Aviral Shrivastava, and Baoxin Li. Hardware acceleration of sparse and irregular tensor computations of ml models: A survey and insights. Proceedings of the IEEE, 109(10):1706–1752, 2021. 4

    work page 2021

Show all 70 references
  1. [8]

    Nested sparse attention

    DeepSeek. Nested sparse attention. arXiv preprint arXiv:2502.11089, 2025. 2, 3

    2025 arXiv

  2. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  3. [10]

    R2r: Efficiently navigating divergent reasoning paths with small-large model token routing

    Tianyu Fu, Yi Ge, Yichen You, Enshu Liu, Zhihang Yuan, Guohao Dai, Shengen Yan, Huazhong Yang, and Yu Wang. R2r: Efficiently navigating divergent reasoning paths with small-large model token routing. arXiv preprint arXiv:2505.21600, 2025. 21

    2025

  4. [11]

    Moa: Mixture of sparse attention for automatic large language model compression, 2024

    Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Moa: Mixture of sparse attention for automatic large language model compression, 2024. 3

    2024

  5. [12]

    Framefusion: Combining similarity and importance for video token reduction on large visual language models

    Tianyu Fu, Tengxuan Liu, Qinghao Han, Guohao Dai, Shengen Yan, Huazhong Yang, Xuefei Ning, and Yu Wang. Framefusion: Combining similarity and importance for video token reduction on large visual language models. arXiv preprint arXiv:2501.01986, 2024. 21

    2024 arXiv

  6. [13]

    Clipscore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Bras, and Choi Yejin. Clipscore: A reference-free evaluation metric for image captioning. pages 7514–7528, 01 2021. 7

    2021

  7. [14]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS’17,...

    2017

  8. [15]

    Open-Sora

    HPC-AI. Open-Sora. https://github.com/hpcaitech/Open-Sora, 2024. 3

    2024

  9. [16]

    Qvgen: Pushing the limit of quantized video generative models

    Yushi Huang, Ruihao Gong, Jing Liu, Yifu Ding, Chengtao Lv, Haotong Qin, and Jun Zhang. Qvgen: Pushing the limit of quantized video generative models. arXiv preprint arXiv:2505.11497, 2025. 3 10

    2025

  10. [17]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. arXiv preprint arXiv:2311.17982, 2023. 7

    2023 arXiv

  11. [18]

    Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention, 2024. 2, 3, 8, 17

    2024

  12. [19]

    Flux.1: A high-quality text-to-image model

    Black Forest Labs. Flux.1: A high-quality text-to-image model. https://github.com/ black-forest-labs/flux , 2024. Accessed [current date]. 3, 7

    2024

  13. [20]

    Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference, 2025

    Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference, 2025. 2

    2025

  14. [21]

    Svdqunat: Absorbing outliers by low-rank components for 4-bit diffusion models

    Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun-Yan Zhu, and Song Han. Svdqunat: Absorbing outliers by low-rank components for 4-bit diffusion models. arXiv preprint arXiv:2411.05007, 2024. 3

    2024

  15. [22]

    Mbq: Modality-balanced quantization for large vision-language models

    Shiyao Li, Yingchun Hu, Xuefei Ning, Xihui Liu, Ke Hong, Xiaotao Jia, Xiuhong Li, Yaqi Yan, Pei Ran, Guohao Dai, et al. Mbq: Modality-balanced quantization for large vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference , pages 4167–4...

    2025

  16. [23]

    Evaluating quantized large language models

    Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Evaluating quantized large language models. In Proceedings of the 41st International Conference on Machine Learning , pages 28480–28524, 2024. 3

    2024

  17. [24]

    Q-diffusion: Quantizing diffusion models

    Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17535–17545, 2023. 3

    2023

  18. [25]

    Toklip: Marry visual tokens to clip for multimodal comprehension and generation

    Haokun Lin, Teng Wang, Yixiao Ge, Yuying Ge, Zhichao Lu, Ying Wei, Qingfu Zhang, Zhenan Sun, and Ying Shan. Toklip: Marry visual tokens to clip for multimodal comprehension and generation. arXiv preprint arXiv:2505.05422, 2025. 21

    2025 arXiv

  19. [26]

    Duquant: Distributing outliers via dual transformation makes stronger quantized llms

    Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized llms. Advances in Neural Information Processing Systems , 37:87766–87800, 2024. 3

    2024

  20. [27]

    Awq: Activation-aware weight quantization for llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023. 6

    2023 arXiv

  21. [28]

    Lawrence Zitnick, and Piotr Dollár

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015. 7

    2015

  22. [29]

    Fpsattention: Training-aware fp8 and sparsity co-design for fast video diffusion

    Akide Liu, Zeyu Zhang, Zhexin Li, Xuehai Bai, Yizeng Han, Jiasheng Tang, Yuanjie Xing, Jichao Wu, Mingyang Yang, Weihua Chen, et al. Fpsattention: Training-aware fp8 and sparsity co-design for fast video diffusion. arXiv preprint arXiv:2506.04648, 2025. 3

    2025 arXiv

  23. [30]

    From reusing to forecasting: Accelerating diffusion models with taylorseers

    Jiacheng Liu, Chang Zou, Yuanhuiyi Lyu, Junjie Chen, and Linfeng Zhang. From reusing to forecasting: Accelerating diffusion models with taylorseers. arXiv preprint arXiv:2503.06923, 2025. 21

    2025 arXiv

  24. [31]

    Pm-kvq: Progressive mixed-precision kv cache quantization for long-cot llms

    Tengxuan Liu, Shiyao Li, Jiayi Yang, Tianchen Zhao, Feng Zhou, Xiaohui Song, Guohao Dai, Shengen Yan, Huazhong Yang, and Yu Wang. Pm-kvq: Progressive mixed-precision kv cache quantization for long-cot llms. arXiv preprint arXiv:2505.18610, 2025. 3

    2025 arXiv

  25. [32]

    Evalcrafter: Benchmarking and evaluating large video generation models

    Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. arXiv preprint arXiv:2310.11440, 2023. 7

    arXiv

  26. [33]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021. 2

    2021

  27. [34]

    A white paper on neural network quantization

    Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295, 2021. 8

    2021 arXiv

  28. [35]

    Scalable diffusion models with transformers, 2023

    William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. 2, 3 11

    2023

  29. [36]

    Post-training quantization on diffusion models

    Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1972–1981, 2023. 3

    1972

  30. [37]

    Plug-and-play 1.x-bit kv cache quantization for video large language models

    Keda Tao, Haoxuan You, Yang Sui, Can Qin, and Huan Wang. Plug-and-play 1.x-bit kv cache quantization for video large language models. arXiv preprint arXiv:2503.16257, 2025. 3

    2025

  31. [38]

    Cogvideox-5b

    THUDM. Cogvideox-5b. https://huggingface.co/THUDM/CogVideoX-5b, 2025. 7

    2025

  32. [39]

    Fvd: A new metric for video generation

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019. 8

    2019

  33. [40]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. 3

    2017

  34. [41]

    Wan: Open and advanced large-scale video generative models

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pande...

    2025 arXiv

  35. [42]

    Image quality assessment: From error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 8

    2004

  36. [43]

    Godiva: Generating open-domain videos from natural descriptions

    Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.14806 ,

    arXiv

  37. [44]

    Exploring video quality assessment on user generated contents from aesthetic and technical perspectives

    Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. In Proceedings of the IEEE/CVF International Conference on...

    2023

  38. [45]

    Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity, 2025

    Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, Jianfei Chen, Ion Stoica, Kurt Keutzer, and Song Han. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity, 2025. 2, 7, 8

    2025

  39. [46]

    Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity

    Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, et al. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity. arXiv preprint arXiv:2502.01776, 2025. 3, 17

    2025 arXiv

  40. [47]

    Efficient multi-task llm quantization and serving for multiple lora adapters

    Yifei Xia, Fangcheng Fu, Wentao Zhang, Jiawei Jiang, and Bin Cui. Efficient multi-task llm quantization and serving for multiple lora adapters. Advances in Neural Information Processing Systems , 37:63686– 63714, 2024. 21

    2024

  41. [48]

    Training-free and adaptive sparse attention for efficient long video generation, 2025

    Yifei Xia, Suhan Ling, Fangcheng Fu, Yujie Wang, Huixia Li, Xuefeng Xiao, and Bin Cui. Training-free and adaptive sparse attention for efficient long video generation, 2025. 3

    2025

  42. [49]

    Infllm: Training-free long-context extrapolation for llms with an efficient context memory,

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training-free long-context extrapolation for llms with an efficient context memory,

  43. [50]

    Duoattention: Efficient long-context llm inference with retrieval and streaming heads

    Guang Xuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context llm inference with retrieval and streaming heads. arXiv preprint arXiv:2410.10819, 2024. 3

    arXiv

  44. [51]

    Efficient streaming language models with attention sinks, 2024

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024. 3

    2024

  45. [52]

    Efficientllm: Scalable pruning-aware pretraining for architecture-agnostic edge language models

    Xingrun Xing, Zheng Liu, Shitao Xiao, Boyan Gao, Yiming Liang, Wanpeng Zhang, Haokun Lin, Guoqi Li, and Jiajun Zhang. Efficientllm: Scalable pruning-aware pretraining for architecture-agnostic edge language models. arXiv preprint arXiv:2502.06663, 2025. 21 12

    2025 arXiv

  46. [53]

    Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023. 7

    2023

  47. [55]

    Seerattention: Learning intrinsic sparse attention in your llms

    Zhuoyi Yang et al. Seerattention: Learning intrinsic sparse attention in your llms. arXiv preprint arXiv:2410.13276, 2024. 2

    2024 arXiv

  48. [56]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 2, 3, 7

    2024 arXiv

  49. [57]

    DiTFastattn: Attention compression for diffusion transformer models

    Zhihang Yuan, Hanling Zhang, Lu Pu, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. DiTFastattn: Attention compression for diffusion transformer models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. 2, 3, 8, 17

    2024

  50. [58]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Avinava Dubey, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020. 2

    2007 arXiv

  51. [59]

    Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization, 2024

    Jintao Zhang, Haofeng Huang, Pengle Zhang, Jia Wei, Jun Zhu, and Jianfei Chen. Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization, 2024. 2, 3, 8

    2024

  52. [60]

    Sageattention3: Microscaling fp4 attention for inference and an exploration of 8-bit training

    Jintao Zhang, Jia Wei, Pengle Zhang, Xiaoming Xu, Haofeng Huang, Haoxu Wang, Kai Jiang, Jun Zhu, and Jianfei Chen. Sageattention3: Microscaling fp4 attention for inference and an exploration of 8-bit training. arXiv preprint arXiv:2505.11594, 2025. 2

    2025

  53. [61]

    Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration

    Jintao Zhang, Jia Wei, Pengle Zhang, Jun Zhu, and Jianfei Chen. Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration. In International Conference on Learning Representations (ICLR) ,

  54. [62]

    Spargeattn: Accurate sparse attention accelerating any model inference

    Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattn: Accurate sparse attention accelerating any model inference. arXiv preprint arXiv:2502.18137,

  55. [63]

    Spargeattn: Accurate sparse attention accelerating any model inference, 2025

    Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattn: Accurate sparse attention accelerating any model inference, 2025. 14, 17, 21

    2025

  56. [64]

    Fast video generation with sliding tile attention

    Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhenghong Liu, and Hao Zhang. Fast video generation with sliding tile attention. arXiv preprint arXiv:2502.04507, 2025. 2, 3, 7

    2025 arXiv

  57. [65]

    Faster video diffusion with trainable sparse attention

    Peiyuan Zhang, Haofeng Huang, Yongqi Chen, Will Lin, Zhengzhong Liu, Ion Stoica, Eric P Xing, and Hao Zhang. Faster video diffusion with trainable sparse attention. arXiv preprint arXiv:2505.13389, 2025. 3

    2025

  58. [66]

    Accurate int8 training through dynamic block-level fallback

    Pengle Zhang, Jia Wei, Jintao Zhang, Jun Zhu, and Jianfei Chen. Accurate int8 training through dynamic block-level fallback. arXiv preprint arXiv:2503.08040, 2025. 3

    2025 arXiv

  59. [67]

    H 2o: Heavy-hitter oracle for efficient generative inference of large language models, 2023

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H 2o: Heavy-hitter oracle for efficient generative inference of large language models, 2023. 3

    2023

  60. [68]

    Flasheval: Towards fast and accurate evaluation of text-to-image diffusion generative models

    Lin Zhao, Tianchen Zhao, Zinan Lin, Xuefei Ning, Guohao Dai, Huazhong Yang, and Yu Wang. Flasheval: Towards fast and accurate evaluation of text-to-image diffusion generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16...

    2024

  61. [69]

    Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation, 2025

    Tianchen Zhao, Tongcheng Fang, Haofeng Huang, Enshu Liu, Rui Wan, Widyadewi Soedarmadji, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, Huazhong Yang, Xuefei Ning, and Yu Wang. Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generatio...

    2025

  62. [70]

    skipping

    Tianchen Zhao, Xuefei Ning, Tongcheng Fang, Enshu Liu, Guyue Huang, Zinan Lin, Shengen Yan, Guohao Dai, and Yu Wang. Mixdq: Memory-efficient few-step text-to-image diffusion models with metric-decoupled mixed precision quantization. In European Conference on Computer Vision , ...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.