Pith. sign in

REVIEW 4 major objections 6 minor 66 references

SD-Acc: Accelerating Stable Diffusion through Phase-aware Sampling and Hardware Co-Optimizations

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Phase-aware sampling lets Stable Diffusion skip most of the U-Net in the later denoising phase, cutting compute up to 5.7x without retraining while a co-designed FPGA raises energy efficiency up to 6.0x over a V100 GPU.

desk verdict A credible co-design paper with a genuinely useful phase-aware sampling scheme, but the 'no quality compromise' headline is only proven in-sample; deserves peer review with revisions. read the letter →

arxiv 2507.01309 v1 pith:BBEA3YBA submitted 2025-07-02 cs.AR

Zhican Wang , Guanghui He , Hongxiang Fan This is my paper · ORCID
classification cs.AR
keywords StableDiffusionmodelaccelerationphase-awaresamplingFPGAacceleratorsystolicarraydataflowalgorithm-hardwareco-designU-Netblockskippingtext-to-imagegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SD-Acc claims that Stable Diffusion's 50-step denoising process splits into a sketching phase, which fixes the image's global structure, and a refinement phase, which adjusts local texture. Analyzing how much each U-Net block's activations change between adjacent timesteps, the paper argues that in the refinement phase only the first two U-Net blocks need to compute, reusing activations from the nearest fully computed timestep. This phase-aware sampling (PAS) schedule needs no retraining and, across StableDiff v1.4, v2.1, and XL, removes a factor of 2.4x to 5.7x of multiply-accumulate work while keeping image-quality metrics close to the original. The paper's hardware half is a co-designed FPGA that maps both convolution and attention onto a simple systolic array, hides softmax/layernorm/GELU latency with two-stage streaming, and adapts data reuse per layer, which together give 2.7x to 6.0x lower energy per generated image than an Nvidia V100 GPU. If correct, SD-Acc offers a retraining-free path to cheaper, faster deployment of text-to-image models on energy-limited devices.

What carries the argument

The load-bearing object is the normalized shift score $\bar{S}_t$ derived from Eq. (1), averaged over calibration prompts and min-max scaled per U-Net block; it is the evidence that high-level features stabilize while the top blocks remain outliers, and Eq. (2) converts it into the phase boundary $D^*$. Phase-aware sampling is the schedule built on that boundary: the full U-Net runs at the start and periodically in the sketching phase, and in the refinement phase only the first $L_{\mathrm{refine}} = 2$ blocks run, fed by activations reused from the nearest complete timestep. On the hardware side, the carry mechanism is the address-centric dataflow, which decomposes each convolution into $1\times1$ kernels that behave as matrix multiplications and routes partial sums to output addresses through fixed offsets such as $l \to l+1$, removing im2col's irregular memory traffic and letting one systolic array handle convolution and attention together. The two-stage streaming unit then hides nonlinear operations by splitting softmax and layernorm into numeric-characteristic acquisition and normalization and scheduling those stages inside the systolic array's streaming writes and reads.

What would settle it

Recompute the phase boundary of Eq. (2) on many random 5% subsets of the MS-COCO prompt set for StableDiff v1.4: if the argmin $D^*$ varies by more than a few timesteps across subsets, or if the top-two outlier blocks differ for v2.1 or XL, the fixed PAS schedule is an artifact of one calibration run. A second decisive experiment is to run the most aggressive tested schedule on a prompt set dominated by fine-grained text and measure CLIP and FID; a sharp drop would show that the refinement phase needs more than the top two blocks when local texture carries semantic content.

Watch

Extended reading notes

Core claim

Phase-aware sampling is the paper's answer to the observation that Stable Diffusion's denoising is not uniformly redundant. The authors define a shift score $S_i^t = \frac{\|A_i^t - A_i^{t-1}\|_2}{\|A_i^{t-1}\|_2}$ on the main-branch activations of each upsampling block, min-max normalize it per block, average over 100 PartiPrompts prompts using the 50-timestep scheduler, and find that most blocks settle into low variation after roughly the first half of denoising while the top blocks stay active. They read this as a sketching phase followed by a refinement phase, locate the boundary $D^*$ with a k-means-style sweep, and then execute only the first $L_{\mathrm{sketch}} = L_{\mathrm{refine}} = 2$ blocks during most of the refinement timesteps, using activations from the nearest complete timestep as the entry point. Across StableDiff v1.4, v2.1, and XL this removes a factor of 2.4x to 5.7x of the MAC work at nearly unchanged CLIP score and no retraining. The hardware half of the paper shows that the same saving can be realized: an address-centric dataflow maps convolution onto a plain systolic array as decomposed 1x1 matrix products, the two-stage streaming unit hides softmax/layernorm/GELU latency in the data stream, and adaptive reuse and fusion cut off-chip traffic, yielding 2.7x to 6.0x lower energy per image than an Nvidia V100 GPU.

Load-bearing premise

The load-bearing premise is that the two-phase pattern seen in activation changes on 100 calibration prompts with one scheduler is stable enough that the phase boundary and the choice to keep only the top two U-Net blocks transfer to the target prompts and to all three StableDiff versions.

Editorial extensions

If this is right

  • Across StableDiff v1.4, v2.1, and XL, the PAS schedules of Table II remove a factor of 2.4x to 5.7x of multiply-accumulate work while keeping CLIP scores within about 0.005 of the original, with FID slightly improved on v1.4 and v2.1 and slightly worse on XL.
  • Because the schedule hyperparameters are searched against user-specified quality and performance targets, the same framework can tune StableDiff for a desired speed/quality trade-off instead of applying one fixed pruning recipe.
  • The co-designed FPGA converts the algorithm savings into 2.7x to 6.0x lower energy per image than an Nvidia V100 GPU and 14.7x to 44.9x lower than the tested CPUs.
  • The address-centric dataflow lets one static systolic array execute both convolutions and attention with high PE utilization, and the two-stage streaming unit hides most nonlinear-operator latency, so the hardware gains are largely orthogonal to the phase-aware sampling gains.
  • Because the retained top blocks run at the largest resolutions, the pruned workload has higher operational intensity, moving StableDiff toward the compute-bound region where the systolic array is most effective.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The shift-score analysis is reported for one scheduler and one calibration distribution; if the same two-phase boundary appears under faster solvers such as DDIM or DPM-Solver, PAS could be combined with step distillation to compound the savings, but the paper does not test this.
  • The fixed choice of the top two blocks as the only refinement-phase survivors is a statistical choice; a stress test with text-heavy prompts, where local texture carries semantic information, would reveal whether the phase division is a general property of denoising or an artifact of the MS-COCO distribution.
  • Because the FPGA is estimated to consume over five times the power of an ASIC at the same node, the reported energy savings are a lower bound for a production ASIC implementation; the paper notes this but does not quantify it.
  • The address-centric convolution decomposition is a generic way to make one systolic array handle mixed CNN-Transformer workloads, so the hardware mechanism could transfer to other U-Net-like backbones such as video diffusion models, though the paper evaluates only text-to-image generation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SD-Acc, an algorithm-hardware co-design for accelerating Stable Diffusion U-Net inference. At the algorithm level, it analyzes shift scores of U-Net activations across denoising timesteps (Eq. 1) and identifies a two-phase structure: a sketching phase and a refinement phase, separated by a transition timestep D* computed via a K-means-style objective (Eq. 2). During refinement, the method executes only the top Lrefine blocks of the U-Net, reusing activations from the nearest complete timestep, which yields reported MAC reductions of 2.39x to 5.68x across StableDiff v1.4, v2.1, and XL (Table II). A general framework (Sec. III-C) searches hyperparameters {Tsketch, Tcomplete, Tsparse, Lsketch, Lrefine} under quality and performance constraints. At the hardware level, the paper introduces an address-centric dataflow that maps convolutions onto a simple systolic array, a 2-stage streaming scheme for nonlinear operations, a reconfigurable vector processing unit, and adaptive reuse/fusion strategies (Secs. IV and V). The accelerator is synthesized on a VCU118 FPGA, and the paper reports up to 2.7x-6.0x energy savings over an NVIDIA V100 GPU (Fig. 19) and projected speedups of 2.2x-4.7x over V100 after scaling to 1 GHz and 4096 MACs (Fig. 20). The hardware claims are supported by per-optimization ablations and a roofline analysis, while the algorithm-level quality claim is the main load-bearing concern.

Significance. If the central claims hold, this is a useful retraining-free approximate-inference scheme for a widely deployed generative model, combined with a working FPGA implementation. The hardware work is substantial: the design is synthesized on VCU118, the ablations isolate each hardware optimization (address-centric dataflow, adaptive dataflow, 2-stage streaming), and the roofline accounting is internally consistent, with measured speedup reaching about 95% of the theoretical value for PAS-25/4. The phase-aware sampling idea is interesting and plausibly generalizable. However, the significance of the algorithm-level contribution depends on whether the 'without compromising image quality' claim survives evaluation on held-out prompt distributions and with proper statistical treatment, which the current manuscript does not provide. The comparison with prior accelerators and with CPU/GPU baselines is generally fair in its energy metrics, though the speed comparison in Fig. 20 is explicitly a projection rather than a measured result.

major comments (4)
  1. [Sec. III-C and Table II] The headline claim of preserving image quality is evaluated in-sample. The hyperparameters are calibrated on 5% of the MS-COCO prompt set (Sec. III-C), and Table II reports CLIP/FID/IS on the same MS-COCO validation split, with no held-out prompt distribution, no variance across calibration seeds, and no variance across image generations. Since every schedule parameter (Tsketch, Tcomplete, Tsparse, Lsketch, Lrefine, D*) is derived from this calibration, the absence of a transfer experiment leaves the 'without compromising image quality' claim unsupported for new prompts, schedulers, or model variants. Please add an evaluation on a disjoint prompt dataset (e.g., PartiPrompts or DrawBench) and report confidence intervals over multiple calibration seeds and generated images.
  2. [Table II and Sec. VI-A] The quality metrics in Table II are mutually inconsistent with the 'no quality compromise' claim. For every model, CLIP score monotonically decreases with MAC reduction (e.g., StableDiff v1.4: 0.3004 -> 0.2966 for PAS-25/5), while FID improves. The paper does not state a pre-specified acceptance criterion for quality preservation, and the monotone CLIP decline is evidence of quality loss under any fixed tolerance. The FID improvement is explained only post hoc as a regularization effect, which is not substantiated. Please specify a quality tolerance (e.g., ΔCLIP within measurement noise, FID not statistically worse) and report per-configuration error bars or a paired statistical test.
  3. [Sec. III-A, Eq. (2), Table II] The phase boundary D* is a single point estimate derived from averaged min-max-normalized shift scores over 100 PartiPrompts prompts with one scheduler (Sec. III-A), yet Tsketch=25 is applied uniformly to StableDiff v1.4, v2.1, and XL in Table II. No sensitivity analysis around D* or Tsketch is reported for any model. This is load-bearing because if the true transition timestep for a given model or prompt distribution differs from 25, the quality/complexity trade-off reported in Table II is not the one that a user would obtain in practice. Please report the computed D* for each model and a sweep over nearby Tsketch values for at least one model, together with the resulting quality metrics.
  4. [Sec. VI-F and Fig. 20] The speedups over the NVIDIA V100 reported in Fig. 20 (2.2x-4.7x) are projections obtained by scaling the accelerator from 200 MHz and 1024 MACs to 1 GHz and 4096 MACs, not measured results. The DRAM bandwidth (38.4 GB/s) is not scaled correspondingly, which is important because the roofline position of the projected design changes with the compute-to-memory ratio. Please label Fig. 20 as a projected speedup, state whether the memory system is also scaled, and discuss the sensitivity of the reported speedup to this assumption.
minor comments (6)
  1. [Sec. III-A vs. Sec. VI-A] Eq. (1) cites [21] for the 50-timestep scheduler used in the shift-score analysis, but Sec. VI-A states that the PNDM scheduler [33] is used for sampling; please reconcile this citation inconsistency.
  2. [Sec. III-C] The statement that 'D* is quite robust to the randomness of the prompt' is currently unsupported; please provide the supporting data, for example the distribution of D* across multiple calibration subsets.
  3. [Fig. 4 caption] The caption states '100 x 100 image generations'; please clarify whether this means 100 prompts with 100 images per prompt, and report the exact number of images used in the shift-score averaging.
  4. [Fig. 16 (right)] The y-axis label 'Layer Buffer' appears to refer to the global buffer size; please correct the label and state the normalization point explicitly.
  5. [Table II] The table would be easier to interpret if the hyperparameters for each model were included as columns or in the row labels, since Tcomplete differs between v1.4 (4) and v2.1/XL (3).
  6. [Conclusion] There are several typos, e.g., 'approiate' for 'approximate' and 'Deepcahe' for 'Deepcache' in Sec. III-B; a final proofread is recommended.

Circularity Check

1 steps flagged · score 4.0 of 10

PAS quality claim is partly in-sample: hyperparameters are fitted on a 5% slice of the same MS-COCO benchmark that later reports the quality-preservation numbers, but the phase-division observation and hardware design retain independent content.

  1. fitted input called prediction [Sec. III-C (General Optimization Framework) and Sec. VI-B (Evaluation, Table II)]
    "we also employ a calibration prompt dataset to optimize the hyperparameters offline. The calibration dataset is generated by randomly selecting 5% of the targeted prompt dataset. ... We benchmark our method using the MS-COCO dataset validation split [32]. ... We randomly sample 5000 captions from it as the prompt input for StableDiff, thus generating 5000 images with one prompt per image."

    The framework's solution search (Fig. 7) uses a quality checker with thresholds (delta-FID < 1.0, delta-CLIP < 0.01) on a calibration subset of the target prompt dataset, and the example output is the exact schedule {Tsketch=25, Tcomplete=4, Tsparse=5, Lsketch=2, Lrefine=2} evaluated in Table II. That table then reports 'CLIP scores similar to those of the original model' on the full MS-COCO validation split, which contains the same calibration distribution. The quality metric used to select the schedule is therefore the same quality metric used to claim success, with no held-out split or variance analysis; the 'preserves image quality' result is at least partly fitted rather than predicted. This is not a pure construction, but it is an in-sample evaluation of a fitted schedule.

full rationale

The derivation of PAS itself (shift-score Eq. (1), phase boundary Eq. (2), outlier-block selection) is an empirical observation about trained-model activations on PartiPrompts and is not logically identical to the quality claim. The hardware contributions (address-centric dataflow, 2-stage streaming, adaptive reuse/fusion) are compared against independent baselines and do not rely on author self-citations or imported uniqueness theorems. The only circular element is the quality-gated hyperparameter search: 5% of the MS-COCO target set is used to fit {Tsketch, Tcomplete, Tsparse, Lsketch, Lrefine}, and Table II reports quality on the same MS-COCO validation distribution with no held-out transfer or error bars. This weakens the 'without compromising image quality' claim from a prediction to an in-sample result, but it does not reduce the entire co-design to its inputs, hence the moderate score 4.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim rests on six PAS hyperparameters selected via a quality-gated calibration search on 5% of the target prompt dataset; on the empirical two-phase (sketch/refine) model of denoising, asserted to be stable across prompts and models without a reported sensitivity analysis; on the block-activation reuse approximation; on the untested equivalence between fp16 hardware arithmetic and the software-measured quality; and on self-built simulators for the comparison with Cambricon-D and SDP. No new physical entities are introduced; the sketching/refinement division is an analytic construct grounded in activation statistics, not an independent postulated entity.

free parameters (6)
  • Tsketch (sketching phase duration) = 25 (all reported configs)
    Hyperparameter of phase-aware sampling selected via the calibration framework of Sec. III-C; it bounds the sketching phase and must be no smaller than D*.
  • Tcomplete (initial full-U-Net timesteps) = 4 (SD1.4), 3 (SD2.1, SDXL)
    Chosen by the solution search in Fig. 7 subject to the quality constraint; reported in Table II.
  • Tsparse (period of full U-Net runs within sketching) = swept over 2, 3, 4, 5 (PAS-25/2 through PAS-25/5)
    Trade-off knob between MAC reduction and quality; the reported configs are the ones that passed the framework's quality checker (Sec. III-C, Table II).
  • Lsketch (top blocks kept on sparse sketching steps) = 2
    Number of retained top blocks during sketching; constrained to be at least Lrefine (Sec. III-B).
  • Lrefine (top blocks kept in refinement phase) = 2
    Set from the number of shift-score outlier blocks, determined by a 'significance level' that is not precisely specified (Sec. III-B).
  • D* (optimal phase transition timestep) = model-dependent, not reported numerically
    Meta-parameter from the K-means-style sweep of Eq. (2) on calibration shift scores; it fixes Tsketch's lower bound and the refinement-phase start (Sec. III-B, III-C).
assumptions (6)
  • ad hoc to paper K-means-style objective in Eq. (2) with K=2 correctly splits the 50-step schedule into sketching and refinement phases at the optimal transition timestep D*.
    The two-phase division is the paper's own construct; its optimality is asserted rather than derived, and it drives all downstream hyperparameter choices (Sec. III-B).
  • domain assumption Min-max-normalized, image-averaged shift scores measured with the 50-step PNDM scheduler on 100 PartiPrompts prompts are representative of the target prompts, schedulers, and model variants used in evaluation.
    The entire PAS schedule is calibrated from these statistics; the paper asserts robustness of D* without a reported sensitivity analysis (Sec. III-A, III-C).
  • domain assumption Reusing the activation from the nearest complete timestep as the entry point for retained blocks yields outputs that approximately match the true values.
    This is the core approximation of phase-aware sampling (Sec. III-B, Fig. 5, 'the output approximately matches the real value'); it is verified only indirectly through aggregate CLIP/FID/IS metrics.
  • domain assumption fp16 arithmetic on the FPGA datapath preserves the image quality measured in the algorithm-level evaluation.
    Hardware arithmetic is fp16 (Sec. VI-A), while Tables II and III report algorithm-level quality; no end-to-end quality pass through the synthesized datapath is shown.
  • domain assumption The self-built simulators of Cambricon-D and SDP faithfully reproduce the published accelerators' performance.
    Sec. VI-E states simulators are built from paper details and scaled to equal peak throughput, with no validation against numbers reported by those papers.
  • standard math The online softmax normalizer (Eqs. 5-6, from [40]) and the streaming variance formula (Eq. 4, E[x^2] minus E[x]^2) are valid and numerically acceptable in fp16.
    The softmax trick is prior published work; the streaming variance form is subject to catastrophic cancellation, and the paper does not analyze fp16 precision effects (Sec. IV-C, IV-D).

how reviews work

0 comments
Cite this review

Pith. "Pith review of SD-Acc: Accelerating Stable Diffusion through Phase-aware Sampling and Hardware Co-Optimizations." pith.science (2026). https://pith.science/paper/BBEA3YBA

@misc{pith2026250701309,
  author       = {Pith},
  title        = {Pith review of: SD-Acc: Accelerating Stable Diffusion through Phase-aware Sampling and Hardware Co-Optimizations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BBEA3YBA}},
  note         = {Machine review of arXiv:2507.01309}
}
read the original abstract

The emergence of diffusion models has significantly advanced generative AI, improving the quality, realism, and creativity of image and video generation. Among them, Stable Diffusion (StableDiff) stands out as a key model for text-to-image generation and a foundation for next-generation multi-modal algorithms. However, its high computational and memory demands hinder inference speed and energy efficiency. To address these challenges, we identify three core issues: (1) intensive and often redundant computations, (2) heterogeneous operations involving convolutions and attention mechanisms, and (3) diverse weight and activation sizes. We present SD-Acc, a novel algorithm and hardware co-optimization framework. At the algorithm level, we observe that high-level features in certain denoising phases show significant similarity, enabling approximate computation. Leveraging this, we propose an adaptive, phase-aware sampling strategy that reduces compute and memory loads. This framework automatically balances image quality and complexity based on the StableDiff model and user requirements. At the hardware level, we design an address-centric dataflow to efficiently handle heterogeneous operations within a simple systolic array. We address the bottleneck of nonlinear functions via a two-stage streaming architecture and a reconfigurable vector processing unit. Additionally, we implement adaptive dataflow optimizations by combining dynamic reuse and operator fusion tailored to StableDiff workloads, significantly reducing memory access. Across multiple StableDiff models, our method achieves up to a 3x reduction in computational demand without compromising image quality. Combined with our optimized hardware accelerator, SD-Acc delivers higher speed and energy efficiency than traditional CPU and GPU implementations.

Figures

Figures reproduced from arXiv: 2507.01309 by the authors.

Figure 1
Figure 1. Software and hardware challenges of the StableDiff acceleration. undergoing active variation. The denoising process illustrated in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Profiling results of StableDiff v 1.4 with 50 timesteps using thop tool [66] (left). One addition and one multiplication count as one MAC operation. The latency breakdown (right) is measured on the single-precision model. To analyze the performance bottleneck of StableDiff, we perform profiling of three main components with respect to the number of parameters, operation counts, and latency performance on both CPU an… view at source ↗
Figure 3
Figure 3. The U-Net architecture of StableDiff (left) with acti￾vation variations and information flow (right). convolution layer, the 4th, 7th, 10th downsampling block con￾sists solely of a downsampling operation, and the 4th, 7th, 10th upsampling block includes an additional upsampling opera￾tion. Downsampling and upsampling operations realize reso￾lution variation, while the others only vary the channel sizes. The downsamp… view at source ↗
Figures from the paper (15 more)
Figure 5
Figure 5. Figure 5: An example of phase-aware sampling with layer [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: The MAC breakdown of the downsampling and up [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: General framework for Phase-aware sampling (Exam [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: Convolution decomposition (left). Address mapping scheme of the partial sum from input activation to output (right). [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Illustration of convolution (left), conventional loop nest [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 10
Figure 10. Figure 10: Hardware architecture (left), mapping (center), and address-centric dataflow (right). [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 11
Figure 11. Figure 11: (a) compares the latency of ours and the original, (b)-(d) shows three optimizations for dataflow scheduling. Notice that SA here contains the accumulation unit, because nonlinear operations are only performed on final results, and VPU specifically refers to arithmeti…
Figure 12
Figure 12. Figure 12: Configured datapath for softmax, layernorm, and GELU by reconfigurable and reusable arithmetic and exponential arrays. multiple specialized engines for various operations can reduce overall efficiency. Overview. To uniformly support various nonlinear oper￾ations, we d…
Figure 13
Figure 13. Figure 13: The variation of sizes across different convolutions in [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]
Figure 14
Figure 14. Figure 14: Adaptive input, weight reuse, with corresponding illustration (a), (b), and the adaptive fusion (c). [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]
Figure 15
Figure 15. Figure 15: Latency reduction by streaming computing. 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 Saving by Adaptive Reuse Saving by Adaptive Reuse and Fusion Acess Saving O…
Figure 16
Figure 16. Figure 16: The performance gain brought by adaptive fusion [PITH_FULL_IMAGE:figures/full_fig_p011_16.png]
Figure 17
Figure 17. Figure 17: Algorithmic and hardware optimizations breakdown analysis. other hand, absorbs computation into the data streaming of the systolic array’s read and write operations. Additionally, it decouples data dependencies and eliminates systolic array idleness caused by bottlene…
Figure 19
Figure 19. Figure 19: Energy saving of our approach over baselines. [PITH_FULL_IMAGE:figures/full_fig_p013_19.png]
Figure 20
Figure 20. Figure 20: Scaled speedup of our approach over baselines. [PITH_FULL_IMAGE:figures/full_fig_p013_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 38 canonical work pages

  1. [1]

    Fused-layer cnn accelerators,

    M. Alwani, H. Chen, M. Ferdman, and P. Milder, “Fused-layer cnn accelerators,” in 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture. IEEE, 2016, pp. 1–12

    work page 2016

  2. [2]

    Diannao: A small-footprint high-throughput accelerator for ubiqui- tous machine-learning,

    T. Chen, Z. Du, N. Sun, J. Wang, C. Wu, Y . Chen, and O. Temam, “Diannao: A small-footprint high-throughput accelerator for ubiqui- tous machine-learning,” ACM SIGARCH Computer Architecture News , vol. 42, no. 1, pp. 269–284, 2014

    work page 2014

  3. [3]

    Eyeriss: An energy- efficient reconfigurable accelerator for deep convolutional neural net- works,

    Y .-H. Chen, T. Krishna, J. S. Emer, and V . Sze, “Eyeriss: An energy- efficient reconfigurable accelerator for deep convolutional neural net- works,” IEEE Journal of Solid-State Circuits , vol. 52, no. 1, pp. 127– 138, 2016

    work page 2016

  4. [4]

    Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices,

    Y .-H. Chen, T.-J. Yang, J. Emer, and V . Sze, “Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 9, no. 2, pp. 292–308, 2019

    2019

  5. [5]

    A 28.6 mj/iter stable diffusion processor for text-to-image generation with patch similarity-based sparsity augmentation and text-based mixed-precision,

    J. Choi, W. Jo, S. Hong, B. Kwon, W. Park, and H.-J. Yoo, “A 28.6 mj/iter stable diffusion processor for text-to-image generation with patch similarity-based sparsity augmentation and text-based mixed-precision,” in 2024 IEEE International Symposium on Circuits and Systems (IS- CAS), 2024, pp. 1–5

    work page 2024

  6. [6]

    Flashattention-2: Faster attention with better parallelism and work partitioning,

    T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning,” 2023. [Online]. Available: https://arxiv.org/abs/2307 .08691

    work page 2023

  7. [7]

    Flashattention: Fast and memory-efficient exact attention with io-awareness,

    T. Dao, D. Y . Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” 2022. [Online]. Available: https://arxiv.org/abs/2205.14135

    arXiv 2022

  8. [8]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019. [Online]. Available: https://arxiv.org/abs/1810.04805

    arXiv 2019

Show all 66 references
  1. [9]

    Adaptable butterfly accelerator for attention-based nns via hardware and algorithm co-design,

    H. Fan, T. Chau, S. I. Venieris, R. Lee, A. Kouris, W. Luk, N. D. Lane, and M. S. Abdelfattah, “Adaptable butterfly accelerator for attention-based nns via hardware and algorithm co-design,” in 2022 55th IEEE/ACM International Symposium on Microarchitecture. IEEE, 2022, pp. 599–615

    2022

  2. [10]

    Structural pruning for diffusion models,

    G. Fang, X. Ma, and X. Wang, “Structural pruning for diffusion models,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  3. [11]

    Gemmini: Enabling systematic deep- learning architecture evaluation via full-stack integration,

    H. Genc, S. Kim, A. Amid, A. Haj-Ali, V . Iyer, P. Prakash, J. Zhao, D. Grubb, H. Liew, H. Mao et al., “Gemmini: Enabling systematic deep- learning architecture evaluation via full-stack integration,” in 2021 58th ACM/IEEE Design Automation Conference. IEEE, 2021, pp. 769–774

    2021

  4. [12]

    Aˆ 3: Accelerating attention mechanisms in neural networks with approximation,

    T. J. Ham, S. J. Jung, S. Kim, Y . H. Oh, Y . Park, Y . Song, J.-H. Park, S. Lee, K. Park, J. W. Lee et al. , “Aˆ 3: Accelerating attention mechanisms in neural networks with approximation,” in 2020 IEEE International Symposium on High Performance Computer Architecture . IEEE,...

    2020

  5. [13]

    A k-means clustering algorithm,

    J. A. Hartigan, M. A. Wong et al. , “A k-means clustering algorithm,” Applied statistics, vol. 28, no. 1, pp. 100–108, 1979

    1979

  6. [14]

    Ptqd: Accurate post-training quantization for diffusion models,

    Y . He, L. Liu, J. Liu, W. Wu, H. Zhou, and B. Zhuang, “Ptqd: Accurate post-training quantization for diffusion models,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  7. [15]

    Gaussian error linear units (gelus),

    D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  8. [16]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in Neural Information Processing Systems , vol. 33, pp. 6840– 6851, 2020

    2020

  9. [17]

    Flashdecoding++: Faster large language model inference on gpus,

    K. Hong, G. Dai, J. Xu, Q. Mao, X. Li, J. Liu, K. Chen, Y . Dong, and Y . Wang, “Flashdecoding++: Faster large language model inference on gpus,” 2024. [Online]. Available: https://arxiv.org/abs/2311.01282

    2024 arXiv

  10. [18]

    In-datacenter performance analysis of a tensor processing unit,

    N. P. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers et al. , “In-datacenter performance analysis of a tensor processing unit,” in Proceedings of the 44th annual international symposium on computer architecture , 2017...

    2017

  11. [19]

    Stripes: Bit-serial deep neural network computing,

    P. Judd, J. Albericio, T. Hetherington, T. M. Aamodt, and A. Moshovos, “Stripes: Bit-serial deep neural network computing,” in 2016 49th An- nual IEEE/ACM International Symposium on Microarchitecture. IEEE, 2016, pp. 1–12

    2016

  12. [20]

    Flat: An optimized dataflow for mitigating attention bottlenecks,

    S.-C. Kao, S. Subramanian, G. Agrawal, A. Yazdanbakhsh, and T. Kr- ishna, “Flat: An optimized dataflow for mitigating attention bottlenecks,” in Proceedings of the 28th ACM International Conference on Archi- tectural Support for Programming Languages and Operating Systems, Vol...

    2023

  13. [21]

    Elucidating the design space of diffusion-based generative models,

    T. Karras, M. Aittala, T. Aila, and S. Laine, “Elucidating the design space of diffusion-based generative models,” Advances in Neural Infor- mation Processing Systems , vol. 35, pp. 26 565–26 577, 2022

    2022

  14. [22]

    Bk-sdm: Archi- tecturally compressed stable diffusion for efficient text-to-image gen- eration,

    B.-K. Kim, H.-K. Song, T. Castells, and S. Choi, “Bk-sdm: Archi- tecturally compressed stable diffusion for efficient text-to-image gen- eration,” in Workshop on Efficient Systems for Foundation Models@ ICML2023, 2023

    2023

  15. [23]

    I-bert: Integer-only bert quantization,

    S. Kim, A. Gholami, Z. Yao, M. W. Mahoney, and K. Keutzer, “I-bert: Integer-only bert quantization,” 2021. [Online]. Available: https://arxiv.org/abs/2101.01321

    2021 arXiv

  16. [24]

    Full stack optimization of transformer inference: a survey,

    S. Kim, C. Hooper, T. Wattanawong, M. Kang, R. Yan, H. Genc, G. Dinh, Q. Huang, K. Keutzer, M. W. Mahoney, Y . S. Shao, and A. Gholami, “Full stack optimization of transformer inference: a survey,” 2023. [Online]. Available: https://arxiv.org/abs/2302.14017

    2023 arXiv

  17. [25]

    Cambricon-D: Full-network differential acceleration for diffusion models,

    W. Kong et al. , “Cambricon-D: Full-network differential acceleration for diffusion models,” in 2024 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA) . IEEE, 2024

    2024

  18. [26]

    Packing sparse convolutional neural networks for efficient systolic array implementations: Column combining under joint optimization,

    H. Kung, B. McDanel, and S. Q. Zhang, “Packing sparse convolutional neural networks for efficient systolic array implementations: Column combining under joint optimization,” in Proceedings of the Twenty- Fourth International Conference on Architectural Support for Program- min...

    2019

  19. [27]

    Measuring the gap between fpgas and asics,

    I. Kuon and J. Rose, “Measuring the gap between fpgas and asics,” in Proceedings of the 2006 ACM/SIGDA 14th international symposium on Field programmable gate arrays , 2006, pp. 21–30

    2006

  20. [28]

    Maeri: Enabling flexible dataflow mapping over dnn accelerators via reconfigurable intercon- nects,

    H. Kwon, A. Samajdar, and T. Krishna, “Maeri: Enabling flexible dataflow mapping over dnn accelerators via reconfigurable intercon- nects,” ACM SIGPLAN Notices , vol. 53, no. 2, pp. 461–475, 2018

    2018

  21. [29]

    Q-diffusion: Quantizing diffusion models,

    X. Li, Y . Liu, L. Lian, H. Yang, Z. Dong, D. Kang, S. Zhang, and K. Keutzer, “Q-diffusion: Quantizing diffusion models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 17 535–17 545

    2023

  22. [30]

    Snapfusion: Text-to-image diffusion model on mobile devices within two seconds,

    Y . Li, H. Wang, Q. Jin, J. Hu, P. Chemerys, Y . Fu, Y . Wang, S. Tulyakov, and J. Ren, “Snapfusion: Text-to-image diffusion model on mobile devices within two seconds,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  23. [31]

    Davinci: A scalable architecture for neural network computing

    H. Liao, J. Tu, J. Xia, and X. Zhou, “Davinci: A scalable architecture for neural network computing.” in Hot Chips Symposium , 2019, pp. 1–44

    2019

  24. [32]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 . Springer,...

    2014

  25. [33]

    Pseudo numerical methods for diffusion models on manifolds,

    L. Liu, Y . Ren, Z. Lin, and Z. Zhao, “Pseudo numerical methods for diffusion models on manifolds,” arXiv preprint arXiv:2202.09778, 2022

    2022 arXiv

  26. [34]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps,

    C. Lu, Y . Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, “Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps,” Advances in Neural Information Processing Systems , vol. 35, pp. 5775–5787, 2022

    2022

  27. [35]

    Sanger: A co-design framework for enabling sparse attention using reconfigurable architecture,

    L. Lu, Y . Jin, H. Bi, Z. Luo, P. Li, T. Wang, and Y . Liang, “Sanger: A co-design framework for enabling sparse attention using reconfigurable architecture,” in MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture, 2021, pp. 977–991

    2021

  28. [36]

    Flexflow: A flexible dataflow accelerator architecture for convolutional neural networks,

    W. Lu, G. Yan, J. Li, S. Gong, Y . Han, and X. Li, “Flexflow: A flexible dataflow accelerator architecture for convolutional neural networks,” in 2017 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2017, pp. 553–564

    2017

  29. [37]

    Accelerating diffu- sion models via early stop of the diffusion process,

    Z. Lyu, X. Xu, C. Yang, D. Lin, and B. Dai, “Accelerating diffu- sion models via early stop of the diffusion process,” arXiv preprint arXiv:2205.12524, 2022

    2022 arXiv

  30. [38]

    Deepcache: Accelerating diffusion models for free,

    X. Ma, G. Fang, and X. Wang, “Deepcache: Accelerating diffusion models for free,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    2024

  31. [39]

    On distillation of guided diffusion models,

    C. Meng, R. Rombach, R. Gao, D. Kingma, S. Ermon, J. Ho, and T. Salimans, “On distillation of guided diffusion models,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 14 297–14 306

    2023

  32. [40]

    Online normalizer calculation for softmax,

    M. Milakov and N. Gimelshein, “Online normalizer calculation for softmax,” 2018. [Online]. Available: https://arxiv.org/abs/1805.02867

    2018 arXiv

  33. [41]

    A white paper on neural network quantization,

    M. Nagel, M. Fournarakis, R. A. Amjad, Y . Bondarenko, M. Van Baalen, and T. Blankevoort, “A white paper on neural network quantization,” arXiv preprint arXiv:2106.08295 , 2021. 16

    2021 arXiv

  34. [42]

    Vitality: Pro- moting serendipitous discovery of academic literature with transformers & visual analytics,

    A. Narechania, A. Karduni, R. Wesslen, and E. Wall, “Vitality: Pro- moting serendipitous discovery of academic literature with transformers & visual analytics,” IEEE Transactions on Visualization and Computer Graphics, vol. 28, no. 1, pp. 486–496, 2021

    2021

  35. [43]

    Nvidia deep learning accelerator

    Nvidia., “Nvidia deep learning accelerator.” http://nvdla.org/, 2018

    2018

  36. [44]

    T-stitch: Accelerating sampling in pre-trained diffu- sion models with trajectory stitching,

    Z. Pan, B. Zhuang, D.-A. Huang, W. Nie, Z. Yu, C. Xiao, J. Cai, and A. Anandkumar, “T-stitch: Accelerating sampling in pre-trained diffu- sion models with trajectory stitching,” arXiv preprint arXiv:2402.14167, 2024

    2024 arXiv

  37. [45]

    Hybrid memory cube (hmc),

    J. T. Pawlowski, “Hybrid memory cube (hmc),” in 2011 IEEE Hot chips 23 symposium. IEEE, 2011, pp. 1–24

    2011

  38. [46]

    Sigma: A sparse and irregular gemm ac- celerator with flexible interconnects for dnn training,

    E. Qin, A. Samajdar, H. Kwon, V . Nadella, S. Srinivasan, D. Das, B. Kaul, and T. Krishna, “Sigma: A sparse and irregular gemm ac- celerator with flexible interconnects for dnn training,” in 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA). IE...

    2020

  39. [47]

    Dota: detect and omit weak attentions for scalable transformer acceleration,

    Z. Qu, L. Liu, F. Tu, Z. Chen, Y . Ding, and Y . Xie, “Dota: detect and omit weak attentions for scalable transformer acceleration,” in Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems , 2022, pp. 14–26

    2022

  40. [48]

    Zero-shot text-to-image generation,

    A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. V oss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in International Conference on Machine Learning . PMLR, 2021, pp. 8821–8831

    2021

  41. [49]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 10 684–10 695

    2022

  42. [50]

    Fast high-resolution image synthesis with latent adversarial diffusion distillation,

    A. Sauer, F. Boesel, T. Dockhorn, A. Blattmann, P. Esser, and R. Rom- bach, “Fast high-resolution image synthesis with latent adversarial diffusion distillation,” arXiv preprint arXiv:2403.12015 , 2024

    2024 arXiv

  43. [51]

    Adversarial diffusion distillation,

    A. Sauer, D. Lorenz, A. Blattmann, and R. Rombach, “Adversarial diffusion distillation,” arXiv preprint arXiv:2311.17042 , 2023

    2023 arXiv

  44. [52]

    Deep unsupervised learning using nonequilibrium thermodynamics,

    J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in International Conference on Machine Learning , 2015, pp. 2256–2265

    2015

  45. [53]

    An accelerator for sparse convolutional neural networks leveraging systolic general matrix- matrix multiplication,

    M. Soltaniyeh, R. P. Martin, and S. Nagarakatte, “An accelerator for sparse convolutional neural networks leveraging systolic general matrix- matrix multiplication,” ACM Transactions on Architecture and Code Optimization (TACO), vol. 19, no. 3, pp. 1–26, 2022

    2022

  46. [54]

    Drq: dynamic region-based quantization for deep neural network acceleration,

    Z. Song, B. Fu, F. Wu, Z. Jiang, L. Jiang, N. Jing, and X. Liang, “Drq: dynamic region-based quantization for deep neural network acceleration,” in Proceedings of the ACM/IEEE 47th Annual International Symposium on Computer Architecture , ser. ISCA ’20. IEEE Press, 2020, p. 10...

    2020

  47. [55]

    Softermax: Hardware/software co-design of an efficient softmax for transformers,

    J. R. Stevens, R. Venkatesan, S. Dai, B. Khailany, and A. Raghunathan, “Softermax: Hardware/software co-design of an efficient softmax for transformers,” in 2021 58th ACM/IEEE Design Automation Conference (DAC), 2021, pp. 469–474

    2021

  48. [56]

    Sze, Y .-H

    V . Sze, Y .-H. Chen, T.-J. Yang, and J. S. Emer, Efficient processing of deep neural networks . Springer, 2020

    2020

  49. [57]

    Feather: A reconfigurable accelerator with data reordering support for low-cost on-chip dataflow switching,

    J. Tong, A. Itagi, P. Chatarasi, and T. Krishna, “Feather: A reconfigurable accelerator with data reordering support for low-cost on-chip dataflow switching,” arXiv preprint arXiv:2405.13170 , 2024

    2024 arXiv

  50. [58]

    Spatten: Efficient sparse attention architecture with cascade token and head pruning,

    H. Wang, Z. Zhang, and S. Han, “Spatten: Efficient sparse attention architecture with cascade token and head pruning,” in 2021 IEEE International Symposium on High-Performance Computer Architecture . IEEE, 2021, pp. 97–110

    2021

  51. [59]

    Automated systolic array architecture synthesis for high throughput cnn inference on fpgas,

    X. Wei, C. H. Yu, P. Zhang, Y . Chen, Y . Wang, H. Hu, Y . Liang, and J. Cong, “Automated systolic array architecture synthesis for high throughput cnn inference on fpgas,” in Proceedings of the 54th Annual Design Automation Conference 2017 , 2017, pp. 1–6

    2017

  52. [60]

    Cache me if you can: Accelerating diffusion models through block caching,

    F. Wimbauer, B. Wu, E. Schoenfeld, X. Dai, J. Hou, Z. He, A. Sanakoyeu, P. Zhang, S. Tsai, J. Kohler et al., “Cache me if you can: Accelerating diffusion models through block caching,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, ...

    2024

  53. [61]

    Denoising diffusion step-aware models,

    S. Yang, Y . Chen, L. Wang, S. Liu, and Y . Chen, “Denoising diffusion step-aware models,” arXiv preprint arXiv:2310.03337 , 2023

    2023 arXiv

  54. [62]

    Vitcod: Vision transformer acceleration via dedicated algorithm and accelerator co-design,

    H. You, Z. Sun, H. Shi, Z. Yu, Y . Zhao, Y . Zhang, C. Li, B. Li, and Y . Lin, “Vitcod: Vision transformer acceleration via dedicated algorithm and accelerator co-design,” in 2023 IEEE International Symposium on High-Performance Computer Architecture. IEEE, 2023, pp. 273–286

    2023

  55. [63]

    Scaling autoregres- sive models for content-rich text-to-image generation,

    J. Yu, Y . Xu, J. Y . Koh, T. Luong, G. Baid, Z. Wang, V . Va- sudevan, A. Ku, Y . Yang, B. K. Ayan et al. , “Scaling autoregres- sive models for content-rich text-to-image generation,” arXiv preprint arXiv:2206.10789, vol. 2, no. 3, p. 5, 2022

    2022 arXiv

  56. [64]

    Nn-lut: Neural approximation of non-linear operations for efficient transformer inference,

    J. Yu, J. Park, S. Park, M. Kim, S. Lee, D. H. Lee, and J. Choi, “Nn-lut: Neural approximation of non-linear operations for efficient transformer inference,” 2021. [Online]. Available: https://arxiv.org/abs/2112.02191

    2021 arXiv

  57. [65]

    Optimizing fpga-based accelerator design for deep convolutional neural networks,

    C. Zhang, P. Li, G. Sun, Y . Guan, B. Xiao, and J. Cong, “Optimizing fpga-based accelerator design for deep convolutional neural networks,” in Proceedings of the 2015 ACM/SIGDA international symposium on field-programmable gate arrays, 2015, pp. 161–170

    2015

  58. [66]

    Zhu, https://github.com/Lyken17/pytorch-OpCounter, 2018

    L. Zhu, https://github.com/Lyken17/pytorch-OpCounter, 2018

    2018

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.