Pith. sign in

REVIEW 4 major objections 6 minor 72 references

ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ALTER trains one diffusion UNet into ten timestep-specialized pruned experts, matching full-model image quality at about a quarter of the compute.

desk verdict A genuine new combination of timestep routing and layer pruning, with competitive quality numbers, but the headline speedup is unverified and the single-stage claim needs cleanup. read the letter →

arxiv 2505.21817 v1 pith:MMDQ64ZT submitted 2025-05-27 cs.CV

Xiaomeng Yang , Lei Lu , Qihui Fan , Changdi Yang , Juyi Lin , Yanzhi Wang , Xuan Zhang , Shangqian Gao This is my paper · ORCID
classification cs.CV
keywords diffusionmodelslayerpruningtemporalexpertroutinghypernetworkmixtureofexpertsinferenceaccelerationStable
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ALTER sets out to show that a diffusion model can be made much cheaper without training a separate pruned model: the same UNet is transformed into a mixture of "temporal experts," each a layer-pruned subnetwork specialized to a range of denoising timesteps. A trainable hypernetwork learns both which layers each expert keeps and which expert handles each timestep, and these decisions are refined jointly with the UNet's fine-tuning in a single stage rather than pruning first and fine-tuning later. On Stable Diffusion v2.1, the paper reports that the resulting 20-step model matches the visual fidelity of the original 50-step model while using 25.9% of its total multiply-accumulate operations, giving a 3.64x speedup at 35% sparsity. The motivation is that different phases of denoising genuinely need different network capacity, so pruning decisions should be timestep-aware rather than static or prompt-only.

What carries the argument

The load-bearing object is the trainable hypernetwork $H_\Phi$, which contains an Expert Generator $G$ that maps frozen orthogonal embeddings to $N_e \times N_L$ layer-mask logits and a Temporal Router $R$ that maps timestep embeddings to routing logits over experts. Binary masks and one-hot expert selections are obtained with Gumbel-Sigmoid and Gumbel-Softmax combined with the straight-through estimator, so discrete architecture decisions remain differentiable. Layer skipping is simulated by $x_{\mathrm{out}} = (1 - (m_t)_l)x_{\mathrm{in}} + (m_t)_l f_l(x_{\mathrm{in}})$, which either passes input through or executes layer $f_l$; at inference the zero-mask layers are truly skipped to save MACs. The objective couples a denoising-plus-distillation loss for the UNet with a hypernetwork loss that adds a log-ratio sparsity term and a router-balance term, in an alternating bi-level update.

What would settle it

Measure end-to-end wall-clock latency for ALTER's 20-step model and the original 50-step SDv2.1 on the same GPU with batch size 1, including all routing and layer-skipping overhead, and compare the ratio to 3.64; a substantially smaller ratio would show the MAC-based speedup overstates real acceleration.

Watch

Extended reading notes

Core claim

The paper's central claim is that temporal asymmetry in diffusion can be turned into a structural asset: instead of one pruned model for all steps, ALTER learns ten pruned expert subnetworks of a shared UNet and a router that assigns each denoising timestep to an expert. The pruning masks and routing are produced by a hypernetwork with Gumbel-Sigmoid and Gumbel-Softmax straight-through sampling, and the forward pass simulates pruning with the residual update $x_{\mathrm{out}} = (1 - m_l)x_{\mathrm{in}} + m_l f_l(x_{\mathrm{in}})$, which skips a layer when its mask is zero while keeping gradients flowing. An alternating optimization updates the UNet to perform well under the current expert structures and then updates the hypernetwork to find structures that preserve denoising performance, hit a target sparsity ratio, and keep router use balanced. The paper reports that this single-stage co-adaptation beats static pruning, fixed manual timestep intervals, and two-stage training, and that at 20 inference steps ALTER matches or beats the unpruned 50-step SDv2.1 on FID, CLIP, and CMMD on CC3M and MS-COCO.

Load-bearing premise

The 3.64x speedup is reported as a ratio of total multiply-accumulate counts rather than a measured wall-clock latency for the 50-step baseline, so the claim assumes MAC reductions translate directly into runtime savings without significant overhead from the router or from irregular layer skipping.

Editorial extensions

If this is right

  • At 20 inference steps, ALTER uses 9.89T MACs versus 38.04T for the 50-step SDv2.1 baseline, a 3.64x reduction by the paper's MAC-based measure, with FID-5K of 25.25 versus 27.29.
  • Because the router keys on timestep embeddings, the same trained ALTER model can be run at 15, 20, or 25 steps without retraining or adjusting cache rates, unlike cache-based methods that need parameter tuning per schedule.
  • Timestep-wise routing uses every expert subnetwork somewhere along the denoising trajectory, so it avoids the capacity under-utilization of sample-wise dynamic pruning, where one prompt commits to one subnetwork for the whole trajectory.
  • The single-stage joint optimization is necessary: the paper's ablation shows the two-stage variant (hypernetwork first, then UNet fine-tuning) is worse on FID, CLIP, and CMMD on both datasets.
  • The paper's own appendix states that a final fine-tuning stage for the dynamically pruned UNet is still needed after the hypernetwork training ends, so the joint optimization covers the pruning and routing search rather than the complete training pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported 3.64x speedup is a MAC-count ratio; actual runtime gains may be lower on real hardware because the router adds a per-step forward pass and skipped layers create irregular memory access patterns, so wall-clock latency should be measured before deployment decisions are made.
  • The same hypernetwork-plus-timestep-router recipe could in principle be applied to transformer-based diffusion backbones, where prunable "layers" are attention blocks, though the paper only demonstrates it on a UNet.
  • One testable extension is combining ALTER's layer skipping with feature caching: the two target different sources of redundancy and could compound, but the paper does not measure that combination.
  • Because routing depends only on the timestep embedding, the learned experts might transfer across samplers with different step schedules, but guidance scale and text-conditioning changes could shift what each expert should do; that transfer is not tested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. ALTER proposes a unified framework for accelerating Stable Diffusion v2.1 by combining layer-wise pruning with timestep-conditioned expert routing. A hypernetwork generates binary pruning masks for multiple expert subnetworks and a temporal router assigns each denoising timestep to an expert; the UNet and hypernetwork are trained in an alternating scheme. The authors report that ALTER achieves FID/CLIP comparable to the original 50-step SDv2.1 while using about 25.9% of the MACs with 20 inference steps, claiming a 3.64x speedup at 35% sparsity. Experiments on CC3M and MS-COCO compare against BK-SDM, APTP, and DiP-GO, with ablations on the number of experts, sparsity ratio, and the joint-training strategy.

Significance. The core idea of temporal expert routing for pruned diffusion models is timely and has clear practical motivation: it addresses the inflexibility of static pruning and the low parameter utilization of sample-wise dynamic pruning. The ablation results in Table 3 support the contribution of each design component, and the reported quality at reduced MACs is encouraging if confirmed. However, the paper's main quantitative claims—the speedup factor and the single-stage characterization—are not currently backed by the reported data, which is an issue that must be resolved before the significance can be assessed. The paper does not provide code or error bars, so the robustness of the empirical results remains unverified.

major comments (4)
  1. [Abstract and Table 2] The headline 3.64x speedup is not consistent with the reported MACs. In Table 2, SDv2.1 (50 steps) has 38.04T MACs and ALTER (20 steps) has 9.89T MACs; 38.04/9.89 = 3.85, not 3.64. The same table shows similar mismatches for the DiP-GO baselines (38.04/11.42 = 3.33 vs. reported 3.02; 38.04/7.61 = 5.00 vs. reported 3.81). Since no 50-step wall-clock latency is reported anywhere, the speedup column cannot be audited. The authors should either (a) report measured wall-clock latencies for the exact 50-step baseline in Table 2 and recompute speedups, or (b) clearly label the column as a MACs ratio and correct the inconsistent numbers. The abstract's '3.64x speedup' should be revised accordingly.
  2. [Section 3.3 and Appendix A] The paper repeatedly claims 'single-stage optimization' (Abstract, Section 1, Section 3.3) and uses Table 3 to argue that joint training outperforms a two-stage approach. However, Appendix A explicitly states that 'a final fine-tuning stage for the dynamically pruned U-Net is still found to be a necessary step after T_end is reached.' This contradicts the single-stage claim and leaves the meaning of 'Joint Training' in Table 3 ambiguous. The authors must clarify whether this final fine-tuning is applied in all reported experiments, and if so, how the ablation distinguishes the alternating co-optimization from a two-stage procedure. Without this clarification, the central methodological contribution is not well-defined.
  3. [Tables 1 and 2] The MACs and latency figures are internally inconsistent across tables. Table 1 reports SDv2.1 with 25 steps as 1384.2G MACs (implying ~55G per step), whereas Table 2 reports SDv2.1 with 50 steps as 38.04T MACs (implying ~761G per step), a factor of about 14 difference in per-step MACs. Additionally, in Table 1 the latency for SDv2.1 is 4.0s for both 25 and 20 steps, which is implausible and suggests the measurement protocol or the definition of 'MACs' differs between tables. The authors should specify the measurement setup (resolution, batch size, whether MACs include the autoencoder/text encoder, and how latency is averaged) and ensure consistency across all tables.
  4. [Tables 1-3 and Section 4.2] No error bars, confidence intervals, or multiple-seed results are reported for any of the evaluation metrics. Many of the headline comparisons involve differences of 0.1-0.3 in FID or CLIP (e.g., ALTER 20-step vs. SDv2.1 20-step in Table 1, and ALTER vs. the 'Manual' variant in Table 3), which may be within noise. The authors should run at least three independent training runs with different random seeds and report mean and standard error for the main quantitative claims, or at minimum for the ablation study.
minor comments (6)
  1. [Section 4.1] The phrase 'We prune the models as 65%' is ambiguous: it could mean a target sparsity of 65% (i.e., 35% of weights retained) or a retained-MAC ratio of 65%. Please clarify the definition of p and the sparsity terminology used throughout the paper.
  2. [Conclusion and Abstract] The conclusion mentions '3.64× speedup ... with 35% sparsity,' while the abstract says 'utilizing only 25.9% of its total MACs.' These two numbers are not obviously consistent (1/0.259 ≈ 3.86), so please state explicitly how the speedup is derived from the sparsity and step-count information.
  3. [Figure 2] The notation '$x_T^{total:1}$' and '$x_T^{total-1:0}$' in the figure is unclear and appears to contain typos. Please replace with a clear description of the timestep routing variable.
  4. [Section 4.3 heading] The word 'Abalation' is misspelled; it should be 'Ablation'.
  5. [Appendix C] 'depedent' is a typo for 'dependent' in 'the Demb is depedent on the pre-trained official SDv2.1 model.'
  6. [Algorithm 1] The algorithm continues updating the UNet after T_end but stops updating the hypernetwork; this is effectively a second fine-tuning phase. Please make this explicit in the main text or revise the description to align with the 'single-stage' terminology.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: ALTER's efficiency and quality claims are empirical results from training and external benchmarks, not consequences of its own definitions.

full rationale

The paper's derivation chain is empirical rather than self-referential. The hypernetwork H_Phi is trained with a denoising objective, a sparsity regularization target, and a router balance loss (Eqs. 4, 5, 7); the UNet is then fine-tuned with the resulting masks, and quality is measured on held-out benchmarks (FID, CLIP, CMMD) on CC3M and MS-COCO. No predicted quantity is defined in terms of a fitted input: the 65% sparsity target is a training constraint, and the reported MACs and latency are measurements of the optimized model. The self-citations (APTP [18], To-MoE [55]) are used as related work and baselines and are not load-bearing for ALTER's central claim. Appendix A explicitly discloses that a final fine-tuning stage is necessary after T_end; this qualifies the 'single-stage' wording but does not create a logical loop. The headline 3.64x speedup lacks full support because Table 2 reports no 50-step wall-clock latency and its speedup column is inconsistent with the table's own MAC ratios (38.04T / 9.89T = 3.85), but this is a measurement/reporting concern, not circularity. Overall, the paper is self-contained against external benchmarks, so the circularity score is low.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim depends on standard assumptions about layer-wise pruning for diffusion models, the transferability of the trained router to inference, and the fidelity of MACs as a latency proxy. Key hyperparameters (sparsity target p=0.65, expert count Ne=10, loss weights) are chosen by hand rather than fitted to data.

free parameters (5)
  • target sparsity p = 0.65
    User-defined compression target, not fitted to data; the paper also reports p=0.60, 0.55, 0.50 in ablations.
  • number of experts Ne = 10
    Chosen based on ablation study (Fig. 4c); Ne=15 gives no further improvement.
  • loss weights = lambda_ratio=5.0, lambda_balance=1.0, lambda_outKD=1.0, lambda_featKD=1.0, lambda_denoise=1e-4
    Set by hand in Appendix C, not systematically tuned.
  • Gumbel temperatures and offsets = tau_g=0.4, b_g=4.0, tau_r=0.4, b_r=0.0
    Chosen for the Gumbel-Sigmoid and Gumbel-Softmax sampling; not ablated.
  • learning rates = 1e-5 (UNet), 7e-5 (hypernetwork)
    Standard AdamW settings, reported in Appendix C.
assumptions (4)
  • standard math Backpropagation through Gumbel-Sigmoid and Gumbel-Softmax with straight-through estimation provides unbiased or useful gradients for discrete architectural selection.
    The method relies on STE (Appendix B.3) for differentiable mask and routing sampling; this is a standard but unproven approximation.
  • domain assumption Layer-wise pruning of residual and transformer blocks is an effective way to compress diffusion UNets without catastrophic quality loss.
    The entire method is built on this premise, which is supported by prior work (BK-SDM, LAPTOP-Diff) but not re-derived here.
  • domain assumption Timestep embeddings carry enough signal to route to specialized experts; the router can learn a mapping from timestep to expert that generalizes to unseen prompts.
    The temporal router takes only the timestep embedding as input (Section 3.2); its generalization across prompts is assumed and tested on CC3M/MS-COCO.
  • ad hoc to paper The final fine-tuning stage does not invalidate the hypernetwork's decisions; the masks and routing learned during the alternating phase remain appropriate after the UNet is fine-tuned without updating the hypernetwork.
    Appendix A discloses a final fine-tuning stage after Tend; the paper does not analyze whether the hypernetwork's decisions become stale.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation." pith.science (2026). https://pith.science/paper/MMDQ64ZT

@misc{pith2026250521817,
  author       = {Pith},
  title        = {Pith review of: ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MMDQ64ZT}},
  note         = {Machine review of arXiv:2505.21817}
}
read the original abstract

Diffusion models have demonstrated exceptional capabilities in generating high-fidelity images. However, their iterative denoising process results in significant computational overhead during inference, limiting their practical deployment in resource-constrained environments. Existing acceleration methods often adopt uniform strategies that fail to capture the temporal variations during diffusion generation, while the commonly adopted sequential pruning-then-fine-tuning strategy suffers from sub-optimality due to the misalignment between pruning decisions made on pretrained weights and the model's final parameters. To address these limitations, we introduce ALTER: All-in-One Layer Pruning and Temporal Expert Routing, a unified framework that transforms diffusion models into a mixture of efficient temporal experts. ALTER achieves a single-stage optimization that unifies layer pruning, expert routing, and model fine-tuning by employing a trainable hypernetwork, which dynamically generates layer pruning decisions and manages timestep routing to specialized, pruned expert sub-networks throughout the ongoing fine-tuning of the UNet. This unified co-optimization strategy enables significant efficiency gains while preserving high generative quality. Specifically, ALTER achieves same-level visual fidelity to the original 50-step Stable Diffusion v2.1 model while utilizing only 25.9% of its total MACs with just 20 inference steps and delivering a 3.64x speedup through 35% sparsity.

Figures

Figures reproduced from arXiv: 2505.21817 by the authors.

Figure 1
Figure 1. Comparison of model utilization in dynamic pruning. Sample-wise pruning can only use [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the ALTER framework. ALTER is a temporal-adaptive-pruning framework [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. A qualitative comparison with original SDv2.1 and BK-SDM-Small. SDv2.1 and BK [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: ALTER (0.65)’s temporal experts and router behavior. (a) Visualization of pruning patterns [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: The training dynamics given different ratios [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: More qualitative results compared with original SDv2.1 and BK-SDM-Small. SDv2.1 and [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 28 canonical work pages

  1. [1]

    Generative modeling by estimating gradients of the data distribution.Advances in neural information processing systems, 32, 2019

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution.Advances in neural information processing systems, 32, 2019

    2019

  2. [2]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  3. [3]

    Score-based generative modeling through stochastic evolution equations in hilbert spaces

    Sungbin Lim, Eunbi Yoon, Taehyun Byun, Taewon Kang, Seungwoo Kim, Kyungjae Lee, and Sungjoon Choi. Score-based generative modeling through stochastic evolution equations in hilbert spaces. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    work page 2023

  4. [4]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  5. [5]

    SDXL: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. InThe Twelfth International Conference on Learning Representations, 2024

    work page 2024

  6. [6]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    arXiv 2022

  7. [7]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  8. [8]

    Blended diffusion for text-driven editing of natural images

    Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18208–18218, 2022

    2022

Show all 72 references
  1. [9]

    Imagic: Text-based real image editing with diffusion models

    Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6007–6017, 2023

    2023

  2. [10]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. InProceedings of the IEEE/CVF international conference on computer vision, pages 3836–3847, 2023

    2023

  3. [11]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500...

    2023

  4. [12]

    Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  5. [13]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.Advances in Neural Information Processing Systems, 35:5775–5787, 2022

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.Advances in Neural Information Processing Systems, 35:5775–5787, 2022. 10

    2022

  6. [14]

    Consistency models, 2023

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023

    2023

  7. [15]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022

    2022 arXiv

  8. [16]

    Deepcache: Accelerating diffusion models for free

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15762–15772, 2024

    2024

  9. [17]

    Headrouter: A training-free image editing framework for mm-dits by adaptively routing attention heads.arXiv preprint arXiv:2411.15034, 2024

    Yu Xu, Fan Tang, Juan Cao, Yuxin Zhang, Xiaoyu Kong, Jintao Li, Oliver Deussen, and Tong-Yee Lee. Headrouter: A training-free image editing framework for mm-dits by adaptively routing attention heads.arXiv preprint arXiv:2411.15034, 2024

    2024 arXiv

  10. [18]

    Not all prompts are made equal: Prompt-based pruning of text-to-image diffusion models

    Alireza Ganjdanesh, Reza Shirkavand, Shangqian Gao, and Heng Huang. Not all prompts are made equal: Prompt-based pruning of text-to-image diffusion models. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  11. [19]

    Laptop-diff: Layer pruning and normalized distillation for compressing diffusion models.arXiv preprint arXiv:2404.11098, 2024

    Dingkun Zhang, Sijia Li, Chen Chen, Qingsong Xie, and Haonan Lu. Laptop-diff: Layer pruning and normalized distillation for compressing diffusion models.arXiv preprint arXiv:2404.11098, 2024

    2024 arXiv

  12. [20]

    Ld-pruner: Effi- cient pruning of latent diffusion models using task-agnostic insights, 2024

    Thibault Castells, Hyoung-Kyu Song, Bo-Kyeong Kim, and Shinkook Choi. Ld-pruner: Effi- cient pruning of latent diffusion models using task-agnostic insights, 2024

    2024

  13. [21]

    Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models, 2022

    Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models, 2022

    2022

  14. [22]

    Pseudo numerical methods for diffusion models on manifolds

    Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. InInternational Conference on Learning Representations, 2022

    2022

  15. [23]

    Fast sampling of diffusion models with exponential integrator, 2023

    Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator, 2023

    2023

  16. [24]

    Parallel sampling of diffusion models, 2023

    Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models, 2023

    2023

  17. [25]

    Latent consistency models: Synthesizing high-resolution images with few-step inference, 2023

    Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference, 2023

    2023

  18. [26]

    Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans

    Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models, 2023

    2023

  19. [27]

    Adversarial diffusion distillation, 2023

    Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation, 2023

    2023

  20. [28]

    Tract: Denoising diffusion models with transitive closure time-distillation, 2023

    David Berthelot, Arnaud Autef, Jierui Lin, Dian Ang Yap, Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbott, and Eric Gu. Tract: Denoising diffusion models with transitive closure time-distillation, 2023

    2023

  21. [29]

    Accelerating diffusion models via early stop of the diffusion process, 2022

    Zhaoyang Lyu, Xudong XU, Ceyuan Yang, Dahua Lin, and Bo Dai. Accelerating diffusion models via early stop of the diffusion process, 2022

    2022

  22. [30]

    Dip-go: A diffusion pruner via few-step gradient optimization

    Haowei Zhu, Dehua Tang, Ji Liu, Mingjie Lu, Jintu Zheng, Jinzhang Peng, Dong Li, Yu Wang, Fan Jiang, Lu Tian, et al. Dip-go: A diffusion pruner via few-step gradient optimization. Advances in Neural Information Processing Systems, 37:92581–92604, 2024

    2024

  23. [31]

    Faster diffusion: Rethinking the role of the encoder for diffusion model inference

    Senmao Li, taihang Hu, Joost van de Weijer, Fahad Khan, Tao Liu, Linxuan Li, Shiqi Yang, Yaxing Wang, Ming-Ming Cheng, and jian Yang. Faster diffusion: Rethinking the role of the encoder for diffusion model inference. InThe Thirty-eighth Annual Conference on Neural Information...

    2024

  24. [32]

    Learning-to-cache: Accelerating diffusion transformer via layer caching

    Xinyin Ma, Gongfan Fang, Michael Bi Mi, and Xinchao Wang. Learning-to-cache: Accelerating diffusion transformer via layer caching. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 11

    2024

  25. [33]

    Cache me if you can: Accelerating diffusion models through block caching

    Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, et al. Cache me if you can: Accelerating diffusion models through block caching. InProceedings of the IEEE/CVF Conference on Computer Vision...

    2024

  26. [34]

    Post-training quantization on diffusion models

    Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1972–1981, 2023

    1972

  27. [35]

    Temporal dynamic quantization for diffusion models.Advances in neural information processing systems, 36:48686–48698, 2023

    Junhyuk So, Jungwon Lee, Daehyun Ahn, Hyungjun Kim, and Eunhyeok Park. Temporal dynamic quantization for diffusion models.Advances in neural information processing systems, 36:48686–48698, 2023

    2023

  28. [36]

    Sana: Efficient high-resolution image synthesis with linear diffusion transformers.arXiv preprint arXiv:2410.10629, 2024

    Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffusion transformers.arXiv preprint arXiv:2410.10629, 2024

    arXiv

  29. [37]

    Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding.arXiv preprint arXiv:1510.00149, 2015

    Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding.arXiv preprint arXiv:1510.00149, 2015

    2015 arXiv

  30. [38]

    The state of sparsity in deep neural networks.arXiv preprint arXiv:1902.09574, 2019

    Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks.arXiv preprint arXiv:1902.09574, 2019

    1902 arXiv

  31. [39]

    Otov2: Automatic, generic, user-friendly.arXiv preprint arXiv:2303.06862, 2023

    Tianyi Chen, Luming Liang, Tianyu Ding, Zhihui Zhu, and Ilya Zharkov. Otov2: Automatic, generic, user-friendly.arXiv preprint arXiv:2303.06862, 2023

    2023 arXiv

  32. [40]

    An accelerated doubly stochastic gradient method with faster explicit model identification

    Runxue Bao, Bin Gu, and Heng Huang. An accelerated doubly stochastic gradient method with faster explicit model identification. InProceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 57–66, 2022

    2022

  33. [41]

    Doubly sparse asynchronous learning for stochastic composite optimization

    Runxue Bao, Xidong Wu, Wenhan Xian, and Heng Huang. Doubly sparse asynchronous learning for stochastic composite optimization. InThirty-First International Joint Conference on Artificial Intelligence (IJCAI), pages 1916–1922, 2022

    1916

  34. [42]

    Learning to prune deep neural networks via layer-wise optimal brain surgeon, 2017

    Xin Dong, Shangyu Chen, and Sinno Jialin Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon, 2017

    2017

  35. [43]

    Lookahead: A far-sighted alternative of magnitude-based pruning, 2020

    Sejun Park, Jaeho Lee, Sangwoo Mo, and Jinwoo Shin. Lookahead: A far-sighted alternative of magnitude-based pruning, 2020

    2020

  36. [44]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. InInternational Conference on Machine Learning, pages 10323–10337. PMLR, 2023

    2023

  37. [45]

    Loraprune: Pruning meets low-rank parameter-efficient fine-tuning.arXiv preprint arXiv:2305.18403, 2023

    Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. Loraprune: Pruning meets low-rank parameter-efficient fine-tuning.arXiv preprint arXiv:2305.18403, 2023

    2023 arXiv

  38. [46]

    Slicegpt: Compress large language models by deleting rows and columns

    Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, 2024

    2024

  39. [47]

    Structural pruning for diffusion models

    Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  40. [48]

    Snapfusion: Text-to-image diffusion model on mobile devices within two seconds.Advances in Neural Information Processing Systems, 36:20662–20678, 2023

    Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds.Advances in Neural Information Processing Systems, 36:20662–20678, 2023

    2023

  41. [49]

    Diffusion probabilistic model made slim, 2022

    Xingyi Yang, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffusion probabilistic model made slim, 2022

    2022

  42. [50]

    Bk-sdm: A lightweight, fast, and cheap version of stable diffusion.arXiv preprint arXiv:2305.15798, 2023

    Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells, and Shinkook Choi. Bk-sdm: A lightweight, fast, and cheap version of stable diffusion.arXiv preprint arXiv:2305.15798, 2023. 12

    2023 arXiv

  43. [51]

    Dynamic diffusion transformer

    Wangbo Zhao, Yizeng Han, Jiasheng Tang, Kai Wang, Yibing Song, Gao Huang, Fan Wang, and Yang You. Dynamic diffusion transformer. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  44. [52]

    Glam: Efficient scaling of language models with mixture-of-experts

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. InInternational Conference on Machine Learning, pages 5547–5569. PMLR, 2022

    2022

  45. [53]

    Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  46. [54]

    Self-moe: Towards compositional large language models with self-specialized experts, 2024

    Junmo Kang, Leonid Karlinsky, Hongyin Luo, Zhen Wang, Jacob Hansen, James Glass, David Cox, Rameswar Panda, Rogerio Feris, and Alan Ritter. Self-moe: Towards compositional large language models with self-specialized experts, 2024

    2024

  47. [55]

    To- moe: Converting dense large language models to mixture-of-experts through dynamic structural pruning, 2025

    Shangqian Gao, Ting Hua, Reza Shirkavand, Chi-Heng Lin, Zhen Tang, Zhengao Li, Longge Yuan, Fangyi Li, Zeyu Zhang, Alireza Ganjdanesh, Lou Qian, Xu Jie, and Yen-Chang Hsu. To- moe: Converting dense large language models to mixture-of-experts through dynamic structural pruning, 2025

    2025

  48. [56]

    Mixture of efficient diffusion experts through automatic interval and sub-network selection

    Alireza Ganjdanesh, Yan Kang, Yuchen Liu, Richard Zhang, Zhe Lin, and Heng Huang. Mixture of efficient diffusion experts through automatic interval and sub-network selection. InEuropean Conference on Computer Vision, pages 54–71. Springer, 2024

    2024

  49. [57]

    EC-DIT: Scaling diffusion transformers with adaptive expert-choice routing

    Haotian Sun, Tao Lei, Bowen Zhang, Yanghao Li, Haoshuo Huang, Ruoming Pang, Bo Dai, and Nan Du. EC-DIT: Scaling diffusion transformers with adaptive expert-choice routing. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  50. [58]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  51. [59]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. InMedical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part I...

    2015

  52. [60]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021

    2021

  53. [61]

    Categorical reparameterization with gumbel-softmax

    Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016

    2016 arXiv

  54. [62]

    {GS}hard: Scaling giant models with conditional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. {GS}hard: Scaling giant models with conditional computation and automatic sharding. InInternational Conference on Learning Representations, 2021

    2021

  55. [63]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    2022

  56. [64]

    Laion- 5b: An open large-scale dataset for training next generation image-text models.Advances in neural information processing systems, 35:25278–25294, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models.Advances in neural informa...

    2022

  57. [65]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p...

    2018

  58. [66]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InComputer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings,...

    2014

  59. [67]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017

    2017

  60. [68]

    Rethinking fid: Towards a better evaluation metric for image generation

    Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9307–9...

    2024

  61. [69]

    Clipscore: A reference-free evaluation metric for image captioning.arXiv preprint arXiv:2104.08718, 2021

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning.arXiv preprint arXiv:2104.08718, 2021

    2021 arXiv

  62. [70]

    Estimating or propagating gradients through stochastic neurons for conditional computation.arXiv preprint arXiv:1308.3432, 2013

    Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation.arXiv preprint arXiv:1308.3432, 2013

    2013 arXiv

  63. [71]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations, 2019

    2019

  64. [72]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022. 14 A Limitations While ALTER employs joint optimization for the co-adaptation of the U-Net and hypernetwork, a final fine-tuning stage for the dynamically pruned U-Net is st...

    2022 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.