Pith. sign in

REVIEW 4 major objections 4 minor 31 references

Qua$^2$SeDiMo: Quantifiable Quantization Sensitivity of Diffusion Models

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that quantization sensitivity of individual layers in diffusion image denoisers can be read directly off a graph neural network, and that these readings can be used to build sub-4-bit mixed-precision weight…

desk verdict Measured sub-4-bit configs are the real contribution; the GNN attribution story is the unvalidated soft spot. read the letter →

arxiv 2412.14628 v1 pith:FIU4GN5S submitted 2024-12-19 cs.CV cs.LG

Keith G. Mills , Mohammad Salameh , Ruichen Chen , Negar Hassanpour , Wei Lu , Di Niu This is my paper · ORCID
classification cs.CVcs.LG
keywords post-trainingquantizationmixed-precisiondiffusionmodelsgraphneuralnetworkssensitivityattributionsub-4-bittext-to-imagegenerationFIDoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the quantization sensitivity of every weight layer in a diffusion image denoiser can be read directly off a graph neural network: a GNN trained to predict end-to-end FID from random mixed-precision configurations yields node embeddings whose L1 norms serve as per-layer sensitivity scores. The authors use these scores to assign each layer one of six choices (3/4-bit precision times K-Means channel-wise, K-Means whole-tensor, or uniform affine quantization), producing sub-4-bit weight configurations — 3.4 bits on PixArt-alpha, 3.9 on PixArt-Sigma, 3.65 on Hunyuan-DiT, 3.7 on SDXL — that match or beat the full-precision model's FID and CLIP while using no calibration data. If true, this would make sub-4-bit weight-only post-training quantization practical for both U-Net and DiT denoisers, and would replace hand-crafted heuristics with automatically extracted, architecture-specific sensitivity rankings. The same surrogate also yields qualitative insights: DiT models prefer cluster-based quantization, U-Net ResNet blocks are the fragile components, and time-embedding layers are consistently the most sensitive.

What carries the argument

The load-bearing mechanism is the GNN-attribution identity: after a graph neural network is trained to regress a denoiser's end-to-end quality (negative FID, optionally penalized by average bit precision), the L1 norm of a node's embedding at hop $m$, $\|h^m_v\|_1$, is interpreted as the sensitivity score of the $m$-hop subgraph rooted at weight layer $v$. The objective adds a ranking loss to the regression loss at every GNN layer, applied to the norms of the graph-level embeddings, so the norms are forced to order the sampled quantization configurations by quality. Because message passing spreads a node's features across its neighborhood, the hop-$m$ norm of a root node aggregates the quantization choices of an entire block structure (self-attention, feedforward, ResNet block, time-embedding module). Selecting, for each node or subgraph root, the quantization setting that maximizes this norm converts the exponential search space into per-node decisions and yields the final mixed-precision configuration.

What would settle it

Measure, for one denoiser, the paper's predicted per-layer sensitivity scores for its optimal sub-4-bit configuration, then independently quantize each single layer to 3-bit UAQ while leaving all other weights at full precision and record the FID change. If the Spearman correlation between the predicted scores and these single-layer FID deltas is not clearly positive, the attribution that the entire method rests on is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that quantitative per-layer sensitivity can be derived from a GNN surrogate rather than from Hessians or activation outliers. Encoding each quantizable weight layer as a graph node whose features describe quantization method, precision, operation type, and position, the paper trains an ensemble of GNNs to regress a combined objective $y = -\mathrm{FID} - \lambda\,\overline{\mathrm{Bits}}$ from only 340–447 sampled configurations per denoiser. A ranking loss (differentiable Spearman rho, LambdaRank/NDCG, or their sum) is applied at every message-passing layer to the L1 norm of the graph-level embedding, which compels the embedding norms to rank configurations by quality; the paper then treats $\|h^m_v\|_1$ as the sensitivity score of the $m$-hop subgraph rooted at layer $v$. Enumerating the six quantization options per node and keeping the highest-scoring setting yields mixed-precision configurations at 3.4–3.9 average weight bits that achieve lower FID than the full-precision model on PixArt-$\alpha$, SDXL, Hunyuan-DiT, and DiT-XL/2, and that outperform Q-Diffusion, TFMQ-DM, and ViDiT-Q at W4A8/W4A6. The paper further claims architecture-level insights from the score distributions: time-embedding and output-projection layers are consistently sensitive, DiT blocks favor K-Means clustering while U-Nets favor uniform affine quantization, and U-Net ResNet blocks are more sensitive than Transformer blocks.

Load-bearing premise

The load-bearing premise is that the L1 norm of a GNN node embedding, learned from a few hundred randomly sampled quantization configurations, faithfully measures how much that layer or block actually contributes to end-to-end image quality; if the embedding norms do not track true sensitivity, the constructed configurations and extracted insights would not be reliable.

Editorial extensions

If this is right

  • Sub-4-bit weight-only post-training quantization becomes practical: PixArt-alpha runs at 3.4 average weight bits, PixArt-Sigma at 3.9, Hunyuan-DiT at 3.65, SDXL at 3.7, and DiT-XL/2 at 3.5 bits, with FID and CLIP scores comparable to or better than the W16A16 reference.
  • The extracted sensitivity rankings tell developers where to spend bits: time-embedding layers and output projections are consistently fragile, ResNet blocks in U-Nets are more sensitive than Transformer blocks, and patchify/skip-connection layers matter in DiTs.
  • Architecture-aware quantization-method choice is quantified: DiT-family models prefer K-Means clustering, U-Nets prefer uniform affine quantization (UAQ), and mixing methods by layer yields better configurations than any single method applied uniformly.
  • Pairing these weight configurations with online 6-bit activation quantization (no calibration set) beats calibration-based baselines (Q-Diffusion, TFMQ-DM, ViDiT-Q) on both FID and CLIP for PixArt-alpha/Sigma and Hunyuan-DiT.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable transfer claim the paper leaves implicit: because the GNN encodes only quantization features and architecture position, a predictor trained on one PixArt model may rank configurations for its sibling model without retraining, which would slash the evaluation budget.
  • One could validate the attribution mechanism itself by ablating each layer to a fixed low-bit setting and correlating measured FID deltas with predicted scores; the paper reports ensemble validation accuracy but not this direct check.
  • The same surrogate-search recipe should transfer to video diffusion models or other generative architectures with hundreds of layers, since it needs only a few hundred FID evaluations and the compute is dominated by sampling, not GNN training.
  • The paper briefly notes a sinusoidal pattern of layer-wise sensitivity across transformer depth; an extension would be to use these scores to guide block pruning or layer dropping, analogous to depth-pruning policies in large language models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Qua²SeDiMo, a mixed-precision post-training quantization (PTQ) framework for diffusion-model denoisers. The denoiser is represented as a directed acyclic graph over weight layers; a GNN is trained on a few hundred randomly sampled quantization configurations using a ranking loss on the L1 norm of the averaged graph embedding (Eq. 6). The authors then interpret the L1 norm of each node embedding as a per-layer sensitivity score and greedily select per-layer quantization methods and precisions (K-Means channel-wise, K-Means tensor-wise, UAQ; 3 or 4 bits) to optimize a FID-plus-average-bit objective. The resulting configurations are evaluated on PixArt-α, PixArt-Σ, Hunyuan-DiT, SDXL, SDv1.5 and DiT-XL/2, with sub-4-bit average precisions (e.g., 3.4-bit on PixArt-α) reported and compared against Q-Diffusion, TFMQ-DM, and ViDiT-Q in FID, CLIP, and a small human-preference study.

Significance. If the per-node sensitivity scores are valid, the paper contributes a scalable, calibration-free way to obtain mixed-precision sub-4-bit weight-only PTQ configurations for both U-Net and DiT denoisers. The final reported configurations are measured, not fitted, and the comparisons in Tables 3 to 6 are extensive, covering three activation-precision levels, multiple architectures, and a human-preference study. The claimed architectural insights (time-embedding importance, U-Net ResBlock fragility, DiT preference for K-Means) are also potentially useful. However, the central attribution from a graph-level ranking loss to per-node scores is not established, and the experimental protocol has a serious FID-protocol inconsistency between search and final evaluation, with no error bars or multiple seeds. These issues are fixable with additional validation experiments, but they currently weaken both the interpretability claim and the statistical reliability of the headline numbers.

major comments (4)
  1. [Methodology, Eq. (6)] The central claim that the L1 norm of a node embedding is a valid per-layer sensitivity score is not implied by the loss in Eq. (6). The ranking term supervises the scalar norm of the averaged graph embedding ||h^m_G||_1 (Eq. 5), which can be maximized by encoding global summaries such as average bit precision or method mix, or by concentrating signal in a small number of nodes; nothing in the loss forces per-node norms to be additive, counterfactual, or monotone in the true FID effect of changing that node's quantization. The subsequent greedy selection that maximizes each node's score independently therefore rests on an unvalidated attribution. I request a direct validation: hold out random configurations, measure the change in FID when a single layer is toggled from 4-bit to 3-bit (or its method is changed) while other layers are held fixed, and report the rank correlation between these measured marginal effects and the GNN node scores; also compare the greedy per-node choice against a joint search on a small subnetwork. Without this, the qualitative insights in the 'Extracted Insights' section and the claim that the method 'can learn to assign the optimal configuration to each layer' are not supported.
  2. [Experimental Results and Discussion; Tables 2 vs 3–6] The FID protocol changes between search and final evaluation. The search-time corpus in Table 2 and the Pareto frontiers in Figure 4 use 1k-image FID computed on the COCO 2017 validation split, while Tables 3, 4, 6 and 8 report 10k-image FID on COCO 2014. The gap is not cosmetic: the full-precision PixArt-α FID is 99.67 at 1k images (Table 2) versus 34.05 at 10k images (Table 3). Because the predictor is trained on the 1k scores, λ is tuned for that protocol, and the best configurations are selected from these noisy labels, the reported 10k FID gains may be affected by FID-estimation noise and dataset mismatch. Please report search-time FID using the same 10k COCO-2014 protocol, or at minimum show that the selected configurations are stable when the search objective is re-evaluated on the final protocol, and include multiple seeds or bootstrap intervals.
  3. [Figures 4 and 9, purple-circle selection] The paper evaluates many predictor variants (three ranking losses, Op-level vs Block-level, two objectives) and then selects a small number of configurations for detailed investigation, marked with purple circles in Figures 4 and 9. The selection rule for these circles is not stated. If the choice was made after inspecting test-set FID, the headline comparisons in Tables 3–6 are a form of best-of-many selection and need correction, e.g., report the full distribution of FID across all variants, or pre-register a selection criterion based only on validation performance. This matters because the reported sub-4-bit results are the best among many searched configurations rather than the output of a single prespecified model selection rule.
  4. [Supplementary, Fig. 18] The predictor validation metrics in Fig. 18 are reported as ensemble weights, but there is no analysis of the error of the predicted FID itself or of the sensitivity scores. The search space has size 6^#W with #W up to 803, while only 340–447 samples per model are used. A predictor can rank the sampled configurations well while the per-node score attribution is wrong, because the ranking loss only evaluates graph-level order. Please report the calibration and error of the graph-level predictor on held-out configurations and, more importantly, the per-node validation described in the first major comment.
minor comments (4)
  1. [Table 3] In the W4A6 block of Table 3, the row labeled 'Q-Diffusion OAQ W4A8' (FID 70.96) should presumably be labeled 'Q-Diffusion OAQ W4A6'; as printed, the same precision label appears in two different activation-precision blocks.
  2. [Methodology, 'Operation-Level Sensitivity via Graphs'] The sentence 'K-Means C quantizes each applies output channel cout separately' contains a typo and should read 'K-Means C quantizes each output channel cout separately'.
  3. [Table 1 and Tables 2–3] Table 1 reports FID values for PixArt-α/Σ that are close to but not identical to the corresponding values in Table 3, yet the caption does not specify the prompt set or generation protocol; please reconcile these numbers with the COCO-2017/COCO-2014 protocol distinction.
  4. [Extracted Insights and Figures 8/15] The stacked bar plots in Figures 8 and 15 show quantization-method distributions for a single constructed configuration; statements such as 'DiT blocks have a slight preference for K-Means' would be more convincing if accompanied by variability across configurations, ensemble folds, or random seeds.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the reported sub-4-bit FID numbers are ground-truth evaluations of held-out configurations; the GNN scoring is a heuristic attribution, not a fitted parameter renamed as a prediction.

full rationale

Walking the derivation chain, I find no step where a claimed prediction or first-principles result reduces to its own inputs by construction. The surrogate GNN is trained on sampled configurations with ground-truth FID labels (Eq. 6 and Table 2), and the headline 3.4/3.9/3.65/3.7-bit configurations are subsequently evaluated by generating images and computing FID/CLIP from scratch (Tables 3–6). Those numbers are therefore measured outcomes, not values forced by the fitted surrogate. The per-node L1 norms are an attribution heuristic: the paper asserts that because h^m_G is computed by averaging node embeddings per Eq. 5, 'the GNN is forced to learn which nodes contribute or detract from yG' and that ||h^m_{v_i}||_1 can be treated as a numerical score. This is a genuine inferential leap and is unvalidated against direct single-layer perturbation experiments, but it is not circular—Eq. 6 supervises only the aggregate graph-embedding norm, and the selection procedure maximizing node norms is not algebraically identical to the training target. The self-citations to GENNAPE (Mills et al. 2023) and AutoBuild (Mills et al. 2024) supply graph encoding and ensemble-training machinery; they are not invoked as evidence for the quantization results and are not load-bearing. One extracted insight (t-Embed importance) is independently corroborated against TFMQ-DM (Huang et al. 2024), further showing the insights are not merely self-referential. The correct concern is correctness risk—whether the learned norms track true counterfactual sensitivity—not circularity. Score 2 reflects minor non-load-bearing self-citation plus the unvalidated attribution leap, which does not meet the threshold for a circularity finding.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central method rests on a small number of chosen coefficients (lambda per model, UAQ alpha per layer, predictor hyperparameters) and on the unproven assumption that GNN embedding norms quantify true sensitivity. No new physical or mathematical entities are postulated.

free parameters (4)
  • lambda (objective coefficient) = 150 for PixArt-alpha, PixArt-Sigma, Hunyuan; 200 for SDXL
    Used in y = -FID - lambda * average_bits; chosen per denoiser to trade off quality and size, not derived. See 'Experimental Results', Fig. 4 caption.
  • UAQ alpha per weight layer = grid search over {0,10,...,90} per layer
    Eq. 3 and 4 select alpha to minimize Lp error of the quantized weight tensor; a per-tensor fit, though not to image data.
  • Sampling Bernoulli probability p = p drawn Uniform(0,1) per configuration
    Determines 3-bit versus 4-bit choices when sampling the training corpus; affects coverage of the search space. See supplementary 'Sampling Quantization Configurations'.
  • Predictor hyperparameters = hidden size 64, 4 GNN layers, 10k epochs, batch 128, lr 1e-3, weight decay 1e-6
    GNN architecture and training settings are chosen by the authors, not derived from theory. See supplementary 'Predictor Training Setup'.
assumptions (4)
  • domain assumption FID on a 1000-image set is a valid proxy target for optimizing end-to-end quantization quality
    The surrogate is trained to predict -FID computed from 1000 generated images; final claims use 10k-image FID. The paper does not quantify the rank correlation between 1k and 10k FID.
  • ad hoc to paper The L1 norm of a GNN node embedding after ranking-loss training measures the true marginal sensitivity of that layer or block to its quantization choice
    This attribution property is asserted in 'Methodology' around Eq. 6 and is never validated against one-layer-at-a-time perturbations.
  • domain assumption The sampled 340 to 447 configurations generalize to the full 6^#W search space
    Table 2 lists #W from 201 to 803, making the search spaces astronomically large; validation is only on a 20% holdout of the same sampling distribution, and final configs are chosen by extrapolation.
  • domain assumption Standard GNN message passing on the denoiser DAG preserves the relevant quantization information
    The denoiser is cast as a DAG with node features for bit precision, method, quantization error, size ratio, and positional features; if this encoding omits interactions that matter for FID, the surrogate will be biased.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Qua$^2$SeDiMo: Quantifiable Quantization Sensitivity of Diffusion Models." pith.science (2026). https://pith.science/paper/FIU4GN5S

@misc{pith2026241214628,
  author       = {Pith},
  title        = {Pith review of: Qua$^2$SeDiMo: Quantifiable Quantization Sensitivity of Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FIU4GN5S}},
  note         = {Machine review of arXiv:2412.14628}
}
abstract

Diffusion Models (DM) have democratized AI image generation through an iterative denoising process. Quantization is a major technique to alleviate the inference cost and reduce the size of DM denoiser networks. However, as denoisers evolve from variants of convolutional U-Nets toward newer Transformer architectures, it is of growing importance to understand the quantization sensitivity of different weight layers, operations and architecture types to performance. In this work, we address this challenge with Qua$^2$SeDiMo, a mixed-precision Post-Training Quantization framework that generates explainable insights on the cost-effectiveness of various model weight quantization methods for different denoiser operation types and block structures. We leverage these insights to make high-quality mixed-precision quantization decisions for a myriad of diffusion models ranging from foundational U-Nets to state-of-the-art Transformers. As a result, Qua$^2$SeDiMo can construct 3.4-bit, 3.9-bit, 3.65-bit and 3.7-bit weight quantization on PixArt-${\alpha}$, PixArt-${\Sigma}$, Hunyuan-DiT and SDXL, respectively. We further pair our weight-quantization configurations with 6-bit activation quantization and outperform existing approaches in terms of quantitative metrics and generative image quality.

Figures

Figures reproduced from arXiv: 2412.14628 by the authors.

Figure 1
Figure 1. Example 5122 images generated using PixArt-α. We compare images from the full precision model to those generated by a quantized denoiser using different PTQ techniques. Specifically, we compare Q-Diffusion, TFMQ-DM and ViDiT-Q at W4 precision to three configurations built by Qua2SeDiMo - W4, W3.7 and W3.4 - with and without 6-bit activation quantization. ing PTQ sensitivity of components in various types of DMs to u… view at source ↗
Figure 2
Figure 2. PixArt-α/Σ images at FP16 precision and quan￾tized to W4A16 by K-Means, UAQ and Q-Diffusion. COCO prompt: ‘A jet with smoke pouring from its wings’. Quant. Method α-W4 α-W3 Σ-W4 Σ-W3 K-Means 32.99 80.65 38.13 133.87 UAQ w/Eq. 3 32.74 77.06 143.25 447.11 Q-Diffusion 40.04 264.51 43.14 366.45 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Induced DiT subgraphs. Attention weights (red) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Results on PixArt-α, PixArt-Σ, Hunyuan and SDXL under constrained optimization to minimize FID and Bits Ę. Dashed horizonal line denotes the FID of the W16A16 model. Dotted grey line denotes the Pareto frontier constructed from our corpus of randomly sampled configurat…
Figure 5
Figure 5. Figure 5: PixArt-Σ example images and comparison with related work. Resolution: 10242 . Method Precision FID Ó CLIP Ò Full Precision W16A16 34.05 0.3102 Q-Diffusion W4A16 41.93 0.2992 TFMQ-DM W4A16 38.67 0.2905 ViDiT-Q W4A16 33.98 0.2924 Qua2SeDiMo W4A16 29.02 0.3056 Qua2SeDiMo …
Figure 6
Figure 6. Figure 6: Hunyuan-DiT example images. Resolution: 10242 . Model Method Precision #Votes (%) Ò PixArt-α Q-Diffusion W4A8 34 (28.81%) TFMQ-DM W4A8 10 (8.47%) ViDiT-Q W4A8 10 (8.47%) Qua2SeDiMo W4A8 51 (43.22%) PixArt-Σ Q-Diffusion W4A8 28 (23.73%) TFMQ-DM W4A8 3 (2.54%) ViDiT-Q W4…
Figure 7
Figure 7. Figure 7: Block-level box-plots for sub 4-bit PixArt- [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Stacked quantization method bar plots for several [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Results on PixArt-α, PixArt-Σ, Hunyuan and SDXL maximing y “ ´F ID for pure performance. Dashed horizonal line denotes the FID of the W16A16 model. Dotted grey line denotes the Pareto frontier constructed from our corpus of randomly sampled configurations (yellow dots)…
Figure 10
Figure 10. Figure 10: Results on DiT-XL/2 and SDv1.5 for pure performance [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Block-level box-plots for the 3.65-bit Hunyuan [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Operation-wise score box plots for several DiT models at multiple weight bit-precision levels. Note several acronyms: [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Layer-wise score box plots for several DiT models at multiple weight bit-precision levels. [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Box plot block-level score distributions for the [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Stacked quantization method bar plots for re [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: Sample images by Qua2SeDiMo quantized models compared to the original FP16 weights. (a) Generated by DiT￾XL/2 and quantized either to to 4 (‘NDCG Op-Level’ predictor when y “ ´F ID in [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: Annotated images generated by SDv1.5 with [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: Validation set predictor performance across three types of hop-level losses [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 28 canonical work pages

  1. [1]

    AdaLN/Time Embedding: 2-hop subgraph covering the three time-embedding linear layers

    work page

  2. [3]

    Advances in Neural Information Processing Systems, 36

    Ptqd: Accurate post-training quantization for diffu- sion models. Advances in Neural Information Processing Systems, 36. Hessel, J.; Holtzman, A.; Forbes, M.; Bras, R. L.; and Choi, Y . 2021. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718. Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hoch...

    arXiv 2021

  3. [4]

    FRAP: Faithful and Realistic Text-to-Image Generation with Adaptive Prompt Weighting

    Talaria: Interactively optimizing machine learning models for efficient inference. In Proceedings of the CHI Conference on Human Factors in Computing Systems, 1–19. Huang, Y .; Gong, R.; Liu, J.; Chen, T.; and Liu, X. 2024. Tfmq-dm: Temporal feature maintenance quantization for diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Visi...

    work page Pith review arXiv 2024

  4. [6]

    1-hop subgraph containing 3-nodes:

    Cross-Attention (2): 2-hop subgraph containing the V and output projection weight layers, and rooted at the lat- ter. 1-hop subgraph containing 3-nodes:

    work page

  5. [9]

    SDXL contains a total of 343458 possible subgraphs split between 11 categories

    Projection Out: 0-hop subgraph containing the final ‘proj out’ weight layer in the DiT. SDXL contains a total of 343458 possible subgraphs split between 11 categories

    work page

  6. [10]

    Rooted at a dummy ‘Add’ node

    Input ResNet Block w/skip: 2-hop subgraph consist- ing of 5 weight layers: input, output, time-embed, skip- connection and a downsampling layer. Rooted at a dummy ‘Add’ node

    work page

  7. [12]

    Hunyuan-DiT contains a total of 317418 possible sub- graphs split between 9 categories

    Projection Out: 1-hop subgraph containing the final ‘norm out’ and ‘proj out’ layers and rooted at the latter. Hunyuan-DiT contains a total of 317418 possible sub- graphs split between 9 categories

    work page

  8. [13]

    Root is the final linear layer

    AdaLN/Time Embedding: 2-hop subgraph for the three time-step weight layers. Root is the final linear layer

    work page

Show all 31 references
  1. [14]

    Caption-Embedding: 1-hop subgraph for the two linear layers and rooted at the second layer

  2. [15]

    Patchify: 0-hop subgraph containing the patch- embedding convolution operation weight layer

  3. [16]

    Rooted at the output projection layer

    Self-Attention: 3-hop subgraph containing 7 nodes, 4 of which are weight layers: Q, K, V and output projection layer. Rooted at the output projection layer

  4. [17]

    Rooted at the ‘MatMul’ node

    Cross-Attention (1): 1-hop subgraph containing the Q and K weight layers and a dummy ‘MatMul’ node for the QK product. Rooted at the ‘MatMul’ node

  5. [18]

    Cross-Attention (2): 2-hop subgraph containing the V and output projection weight layers, and rooted at the lat- ter

  6. [20]

    Skip-connection: 1-hop subgraph rooted at an ‘Add’ node that sums the output of the previous transformer block with the input from the long residual skip-connection

  7. [32]

    Output ResNet Block: 2-hop subgraph consisting of 5 weight layers: input, output, time-embed and two skip- connection layers following (Li et al. 2023). Rooted at an output dummy ‘Add’ node. DiT contains a total of 257688 possible subgraphs split be- tween 4 categories

    2023

  8. [34]

    Attention of the first DiT block contains the ‘Patchify’ convolution

    Attention: 4-hop subgraph containing 8 nodes, at least 5 are weight layers: AdaLN linear layer, Q, K, V and output projection layer. Attention of the first DiT block contains the ‘Patchify’ convolution. Rooted at the output projection layer

  9. [36]

    Root is the second conv layer

    Projection Out: 1-hop subgraph for the final two convo- lution layers in the DiT, after the final Transformer block. Root is the second conv layer. SDv1.5 contains a total of 256488 possible subgraphs split between 12 categories

  10. [37]

    Root is the second linear layer

    Time-Embedding: 1-hop subgraph for the two time-step weight layers. Root is the second linear layer

  11. [38]

    Input Convolution: 0-hop subgraph consisting of the first convolution in the U-Net

  12. [39]

    Output Convolution: 0-hop subgraph consisting of the last convolution in the U-Net

  13. [40]

    The other downsam- pling convolutions are merged into ResNet block sub- graphs

    Block 9 Downsampler: 0-hop subgraph consisting of one of the downsampling convolutions. The other downsam- pling convolutions are merged into ResNet block sub- graphs

  14. [41]

    Upsampler: 0-hop subgraph consisting of an upsampling conv

  15. [42]

    Rooted at the output projection layer

    Self-Attention: 3-hop subgraph consisting of 5 weight layers: projection in convolution, Q, K, V and output projection linear layers. Rooted at the output projection layer

  16. [43]

    Rooted at the output projection layer

    Cross-Attention: 3-hop subgraph consisting of 4 weight layers: Q, K, V and output projection linear layers. Rooted at the output projection layer. Note: Where self- attention contains an input projection convolution, cross- attention contains a dummy ‘Add’ node from the self- ...

  17. [44]

    Root is the second layer

    Feedforward: 1-hop subgraph for the two linear layers. Root is the second layer

  18. [45]

    Transformer Projection Out: 0-hop subgraph consisting of the output projection layer following the feedforward

  19. [46]

    Rooted at the output layer

    Input ResNet Block w/out Skip: 1-hop subgraph con- sisting of the input, output and time-embedding layers. Rooted at the output layer

  20. [47]

    5th layer is a downsampling con- volution

    Input ResNet Block w/skip: 2-hop subgraph consisting of 4 or 5 weight layers: input, output, time-embed, and skip-connection layers. 5th layer is a downsampling con- volution. Rooted at a dummy ‘Add’ node

  21. [48]

    Output ResNet Block: 2-hop subgraph consisting of 5 weight layers: input, output, time-embed and two skip- connection layers following (Li et al. 2023). Rooted at an output dummy ‘Add’ node. Experimental Hardware and Software Resources All experiments conducted in this paper w...

    2023

  22. [91]

    ISBN 978-3-031-73411-3

    Springer Nature Switzerland. ISBN 978-3-031-73411-3. Chen, J.; Yu, J.; Ge, C.; Yao, L.; Xie, E.; Wang, Z.; Kwok, J. T.; Luo, P.; Lu, H.; and Li, Z. 2024. PixArt- α: Fast Training of Diffusion Transformer for Photorealistic Text- to-Image Synthesis. In The Twelfth International...

    2024 arXiv

  23. [2024]

    In Forty-first International Conference on Machine Learning

    Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. Fey, M.; and Lenssen, J. E. 2019. Fast Graph Representation Learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Gra...

    2019 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.