Pith. sign in

REVIEW 3 major objections 4 minor 52 references

Exploring Diffusion Transformer Designs via Grafting

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-stage 'grafting' procedure edits pretrained diffusion transformers into new architectures at a few percent of pretraining compute.

desk verdict Grafting is a genuinely useful low-cost way to materialize DiT edits; the paper's 'exploring designs' framing is a bit ahead of the evidence, and the authors admit it in Sec. 8. read the letter →

arxiv 2506.05340 v2 pith:Q4OMKE2O submitted 2025-06-05 cs.LG cs.AI

Keshigeyan Chandrasegaran , Michael Poli , Daniel Y. Fu , Dongjun Kim , Lea M. Hadzic , Manling Li , Agrim Gupta , Stefano Massaroli
show 4 more authors
Azalia Mirhoseini Juan Carlos Niebles Stefano Ermon Li Fei-Fei
This is my paper · ORCID
classification cs.LGcs.AI
keywords diffusiontransformersarchitectureeditinggraftingactivationdistillationoperatorreplacementhybridarchitecturestext-to-imagegenerationdepth-to-widthrestructuring
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Architecture design for generative models is usually gated by pretraining cost: you cannot know whether a new operator or block layout works without spending a large training budget to find out. This paper asks whether a pretrained diffusion transformer (DiT) can instead serve as a scaffold, so that new architectures are materialized by editing an existing checkpoint rather than by training from scratch. The answer it argues for is grafting, a two-stage recipe that first trains each replacement operator to reproduce the original operator's activations using only 8k samples, then fine-tunes the edited model end-to-end on 10-25% of the training data. The evidence spans three settings: class-conditional hybrids that stay within 0.1-0.4 FID of the DiT-XL/2 baseline (2.38-2.64 vs 2.27) at less than 2% of pretraining compute, a text-to-image model made 1.43x faster with a small GenEval drop, and a restructured 14-layer model obtained by parallelizing every pair of blocks, which beats other models of comparable depth. If the claim holds, the cost of exploring diffusion transformer designs drops from a full pretraining run to a few percent of it.

What carries the argument

The machinery is the two-stage grafting protocol plus two diagnostic tools that decide where grafting is safe. Stage 1, activation distillation, treats initialization as regression: a new operator $g^l_\theta$ is fit to match the pretrained operator $f^l_\phi$ on the same inputs under a loss $\mathcal{L}_{\mathrm{reg}}$ chosen per operator type—L1 for MHA, L2 for MLP—because activation variance, especially in deep layers, makes L2 unstable for attention. Stage 2, lightweight fine-tuning, runs the standard diffusion denoising objective on 10-25% of the data to absorb errors that accumulate when many operators are replaced at once. The diagnostics are the self-grafting baseline, which isolates the procedure's effect from the architecture change, and the band-k locality metric $L_k$, which measures the fraction of attention mass inside a local band and shows that most DiT layers are local enough for convolutional or windowed replacements.

What would settle it

Run the grafting pipeline on DiT-XL/2 with 50% interleaved MHA replacement using Hyena-X, but before Stage 2, compare the activations each surviving attention layer receives against the activations the same layer received in the unedited model on identical inputs, measuring the Wasserstein distance per layer. If the shift is large even though the final grafted model has good FID, then Stage 1's regression-fitted operators are not actually preserving the pretrained distribution and the method's success is carried by fine-tuning instead; if the shift is tiny, the distillation premise is doing the work. A second decisive check is to graft only the least-local layer (smallest $L_k$) and only the most-local layer, each with the same budget: the paper's locality analysis predicts a clear performance ordering, and a null result would falsify the band-$k$ guidance.

Watch

Extended reading notes

Core claim

The paper's central discovery is that wholesale operator replacement in a pretrained DiT does not have to degrade quality much, provided the new operators are initialized by regressing onto the old operators' activations and the assembled model is then lightly fine-tuned. The authors demonstrate this with a self-grafting control: replacing all 28 MHA or all 28 MLP blocks of DiT-XL/2 with randomly initialized copies, then applying the two stages, recovers FID 2.49 and 2.54 against a 2.27 baseline using 10% of ImageNet. They then show that genuinely new operators—gated convolutions (their Hyena-X and Hyena-Y), sliding-window attention, Mamba-2 linear attention, and MLPs with different expansion ratios—can be swapped in at 50% interleaved positions and land within about 0.5 FID of the baseline. The same protocol extends to high-resolution text-to-image generation, where replacing half the MHA layers in PixArt-Σ with Hyena-X gives a 1.43x wall-clock speedup at a GenEval cost of about two points, and to a block-level restructure that converts every sequential pair of DiT-XL/2 blocks into a parallel pair, halving depth to 14 layers at FID 2.77.

Load-bearing premise

The load-bearing premise is that after an operator is swapped in, the input distribution of every other operator stays close enough to the pretrained distribution that regression-fitted replacements trained on 8k examples continue to behave correctly, and that a short fine-tuning stage absorbs whatever error remains.

Editorial extensions

If this is right

  • Replacing every other attention block with local operators (gated convolution, sliding-window attention, or Mamba-2) yields hybrid DiTs within about 0.5 FID of the DiT-XL/2 baseline while cutting operator FLOPs by roughly half.
  • Full replacement of all MHA or MLP blocks, after the two-stage procedure with 10% data, recovers FID 2.49/2.54 versus 2.27, so operator-level edits at scale are feasible without full pretraining.
  • Grafting transfers to 2048x2048 text-to-image generation: a 50% Hyena-X replacement in PixArt-Σ runs 1.43x faster with GenEval 47.78 versus 49.75.
  • Parallelizing every pair of sequential DiT-XL/2 blocks halves depth (28 to 14) and yields FID 2.77, beating other 14-layer DiTs and pruning baselines; the paper frames this as converting depth into width.
  • Layer selection matters: interleaved (striped) replacement outperforms top-local, low-local, and deep-layer heuristics at the same 50% replacement ratio.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the transfer holds beyond the three settings tested, grafting becomes a cheap proxy for architecture search: any operator or block layout can be scored by grafting it onto a strong pretrained checkpoint rather than by a full pretraining run, which would make the space of testable designs far larger.
  • The paper leaves open whether architectures that win under grafting also win when trained from scratch; a natural next experiment is to train the best grafted hybrids from scratch and compare rankings.
  • The PixArt-Σ experiments rely on 12k synthetic image-text pairs, and the paper notes the synthetic data can propagate artifacts and biases; a testable extension is to re-run the same grafts with a curated (or larger) synthetic set and measure how much of the 2-point GenEval drop is data-limited rather than architecture-limited.
  • The depth-to-width restructure suggests an 'architecture arithmetic'—trading depth for width by rewiring existing blocks—that could be tried on other pretrained transformers (including language models), where parallel block execution may give latency wins at fixed quality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces grafting, a two-stage procedure for editing pretrained diffusion transformers (DiTs): Stage 1 initializes new operators by regression-based activation distillation on 8k samples, and Stage 2 performs lightweight end-to-end fine-tuning on 10--25% of the training data. The authors build a testbed on DiT-XL/2 and report that hybrid designs with gated convolutions, sliding-window attention, linear attention, and modified MLPs achieve FID 2.38--2.64 versus the 2.27 baseline, that full self-grafting recovers FID 2.49--2.54, that grafting PixArt-Sigma gives a 1.43x speedup with a small GenEval drop, and that a depth-to-width restructure (28 layers to 14) achieves FID 2.77. The paper argues that grafting enables the study of new DiT architectures under less than 2% of pretraining compute.

Significance. If the central claim holds, grafting would be a useful tool for cheaply prototyping and evaluating architectural edits in diffusion transformers, and the paper's strengths include a careful self-grafting control, a quantitative locality analysis of attention, a public testbed, wall-clock speedup measurements on PixArt-Sigma, and the explicit statement of the main limitation. The significance is currently qualified, however, by the fact that no grafted architecture is validated by from-scratch training; the paper's own Section 8 admits this. The empirical evidence supports the narrower claim that grafting can materialize edited architectures that perform well under the grafting protocol, but it does not yet establish that grafted performance transfers to from-scratch architecture quality.

major comments (3)
  1. [Section 8, Limitations] The sentence 'it remains an open question whether architectures that perform well under grafting also perform well when trained from scratch' directly qualifies the paper's central claim that grafting lets designers 'explore' DiT designs. Because every grafted model in Tables 4--6 is initialized from a pretrained checkpoint and then fine-tuned, the reported ranking (e.g., MLP r=6 FID 2.38, Hyena-X 50% FID 2.61, Hyena-X 75% FID 3.69) can in principle be driven by how much the pretrained rootstock absorbs the edit rather than by the intrinsic quality of the new architecture. Please either add a from-scratch training comparison for at least the top-performing grafted designs (e.g., MLP r=6 and the parallel depth-to-width variant) or explicitly narrow the abstract and conclusion claims to 'quality under the grafting protocol.'
  2. [Section 4.2 and Table 4] The headline comparison 'FID 2.38--2.64 versus 2.27 for DiT-XL/2' is confounded by the Stage 2 protocol: every grafted hybrid in Table 4 is fine-tuned on 128k ImageNet samples for 50K steps, while the DiT-XL/2 baseline is not fine-tuned at all. A control consisting of the unmodified DiT-XL/2 fine-tuned under the same Stage 2 schedule is needed to attribute the observed quality to the architectural edit rather than to the additional fine-tuning; without this control, claims such as 'variable expansion ratio r=6 is effective' (FID 2.38) are not yet isolated.
  3. [Section 6, Table 6] The depth-to-width case study compares the grafted 14-layer model (712M parameters, initialized from pretrained weights, fine-tuned on 25% data) against from-scratch DiT-L/2 (458M, 1,000K iterations), U-ViT-L (287M, 300K), and TinyDiT-D14 (340M, 500K). The claim that grafting 'outperforms other models of comparable depth' does not control for parameter count, compute, or initialization, so it overstates what the case study establishes. Please add a same-compute or parameter-matched baseline, or restrict the claim to 'at fixed depth and parameter count, grafting recovers quality from a pretrained checkpoint better than the listed alternatives.'
minor comments (4)
  1. [Section 3.3, Table 2] The main text states that full MHA self-grafting with 10% data achieves FID 2.53, but Table 2 reports FID 2.49 for that configuration; please align the text and table.
  2. [Section 4.2, Figure 4] The text cites Fig. 4(b) for the 20%-data results of Hyena-X (FID 2.61), but the data-scaling ablation is presented in Fig. 4(a); the layer-selection panel is Fig. 4(b).
  3. [Section 5, Table 5] The 1.43x speedup is reported for a single forward pass at batch size 2; please state explicitly in the main text that this is not an end-to-end generation latency measurement, since the abstract may otherwise be read as claiming an end-to-end speedup.
  4. [Appendix A] Standard deviations are reported for only two representative experiments; since several Table 4 rows differ by FID values as small as 0.02--0.03, please report variance or confidence intervals for the main comparisons that support ordering claims.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: all headline results are judged on external benchmarks; the only notable caveat is an admitted external-validity limitation, not a derivation loop.

full rationale

Grafting's two stages are defined independently of the quality claims: Stage 1 minimizes a regression objective (Eq. 2) that fits a new operator's activations to the frozen pretrained operator's activations, and Stage 2 uses the standard diffusion objective (Eq. 1). Both stages are evaluated end-to-end against external benchmarks (ImageNet FID, GenEval), compared with the standard DiT-XL/2 baseline, so no reported FID or GenEval number is true by construction. The self-grafting control strengthens this reading: randomly reinitializing every MHA or MLP operator and then running the same two-stage procedure recovers near-baseline quality, which would be redundant if the procedure were definitionally guaranteed to recover quality. The paper's self-citations (Hyena operators [52], mechanistic design of hybrid architectures [23], striped designs [24, 25]) are used as background and operator vocabulary, not as evidence that grafting works. The one significant caveat is explicitly flagged in Sec. 8: 'it remains an open question whether architectures that perform well under grafting also perform well when trained from scratch.' This is an external-validity limitation about whether grafted rankings transfer to from-scratch training, not a circular step in the derivation of grafting's own results. No fitted parameter is relabeled as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in solely by citation. Score 1 reflects minor self-citation with no load-bearing circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The central claim rests on several empirical choices: the regression objective, the fine-tuning data amount, operator kernel/window/state hyperparameters, the PixArt layer set, and LoRA rank. The key axioms are smoothness of activations, validity of the band-k locality proxy, and transfer of limited-data fine-tuning. Hyena-X and Hyena-Y are new operators whose evidence is entirely internal to the paper.

free parameters (5)
  • Operator-specific regression objective = L1 for MHA, L2 for MLP
    Chosen in Sec. 3.3 based on validation loss across five layers; affects Stage 1 quality and final FID.
  • Stage 2 fine-tuning data fraction = 10% (hybrids), 25% (depth-to-width), 12k synthetic pairs (PixArt)
    Compute/quality tradeoff; the 10% versus 20% comparison in Fig. 4a shows FID changes with data.
  • Replacement operator hyperparameters = K=4 (Hyena-X/Y), w=4 (SWA), ds=64 E=2 (Mamba-2), r=3/6 (MLP), r=2 (Hyena-X MLP)
    Testbed configurations in Table 3; results are specific to these settings.
  • PixArt replacement layer set = layers 8, 10, 12, 14, 16, 18, 20-27
    Described as 'empirically found' in Sec. 5; not derived from a fixed rule, so the GenEval result is partly the result of layer selection.
  • LoRA rank = 64
    Used in PixArt Stage 2 to manage memory; affects adaptation capacity and reported artifacts.
assumptions (5)
  • domain assumption DiT activations are continuous and smooth, so a new operator can be trained to match them with an L2 or L1 regression objective.
    Invoked in Sec. 3.1 before Eq. (2); if activations are not smooth, 8k-sample distillation would not transfer.
  • domain assumption The band-k locality measured on 250 ImageNet samples with 50-step DDIM summarizes the attention behavior used during generation.
    Sec. 3.4, used to justify local replacements; small sample and specific sampler.
  • domain assumption End-to-end fine-tuning on 10-25% of ImageNet or 12k synthetic pairs preserves enough pretrained capability after operator replacement.
    Sec. 3.1 Stage 2 and Table 2; empirically validated but not guaranteed for other edits.
  • standard math Standard diffusion training objective (Eq. 1) and the DiT architecture are taken as given.
    Sec. 2; background from [12] and [1].
  • domain assumption ImageNet FID and GenEval are valid external measures of generative quality.
    Used throughout to compare baseline and grafted models.
invented entities (2)
  • Hyena-X operator
    purpose: Drop-in replacement for MHA and MLP; local gated convolution with explicit short causal depthwise convolutions and G=I.
    Introduced in this paper; its only performance evidence is the paper's own FID/GenEval measurements, with no external falsifiable handle.
  • Hyena-Y operator
    purpose: Drop-in replacement for MHA; removes featurizer convolutions and uses a short explicit convolution in G.
    Introduced in this paper; its only performance evidence is the paper's own FID measurements.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Diffusion Transformer Designs via Grafting." pith.science (2026). https://pith.science/paper/Q4OMKE2O

@misc{pith2026250605340,
  author       = {Pith},
  title        = {Pith review of: Exploring Diffusion Transformer Designs via Grafting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q4OMKE2O}},
  note         = {Machine review of arXiv:2506.05340}
}
read the original abstract

Designing model architectures requires decisions such as selecting operators (e.g., attention, convolution) and configurations (e.g., depth, width). However, evaluating the impact of these decisions on model quality requires costly pretraining, limiting architectural investigation. Inspired by how new software is built on existing code, we ask: can new architecture designs be studied using pretrained models? To this end, we present grafting, a simple approach for editing pretrained diffusion transformers (DiTs) to materialize new architectures under small compute budgets. Informed by our analysis of activation behavior and attention locality, we construct a testbed based on the DiT-XL/2 design to study the impact of grafting on model quality. Using this testbed, we develop a family of hybrid designs via grafting: replacing softmax attention with gated convolution, local attention, and linear attention, and replacing MLPs with variable expansion ratio and convolutional variants. Notably, many hybrid designs achieve good quality (FID: 2.38-2.64 vs. 2.27 for DiT-XL/2) using <2% pretraining compute. We then graft a text-to-image model (PixArt-Sigma), achieving a 1.43x speedup with less than a 2% drop in GenEval score. Finally, we present a case study that restructures DiT-XL/2 by converting every pair of sequential transformer blocks into parallel blocks via grafting. This reduces model depth by 2x and yields better quality (FID: 2.77) than other models of comparable depth. Together, we show that new diffusion model designs can be explored by grafting pretrained DiTs, with edits ranging from operator replacement to architecture restructuring. Code and grafted models: https://grafting.stanford.edu

Figures

Figures reproduced from arXiv: 2506.05340 by the authors.

Figure 1
Figure 1. Grafting overview. (a,b) Model architecture design via grafting. Studying new model architecture designs requires costly pretraining. Grafting materializes new architectures by editing pretrained models under small compute budgets (Sec. 3). (c) Class-conditional image generation. Samples generated by hybrid architectures obtained via grafting (Sec. 4). (d) High-resolution text-to￾image generation. 2048×2048 samples … view at source ↗
Figure 2
Figure 2. Locality of self-attention in DiT￾XL/2. We plot Lk values for all 28 MHA operators, averaged over timesteps and sam￾ples. At k=32, 15 out of 28 layers exhibit values exceeding 0.5, indicating that several MHA operators model local interactions. MHA scales quadratically with sequence length, mak￾ing it a computational bottleneck. A natural idea is to replace it with local operators, such as convolu￾tion or local atte… view at source ↗
Figure 3
Figure 3. Our proposed Hyena-X and Hyena-Y, efficient local gated convolution operators used as drop-in replacements for MHA. We introduce Hyena-X and Hyena-Y—two new efficient gated convolution operators designed as drop-in replacements for MHA. While our study includes several off-the-shelf efficient alternatives, we also contribute new operator designs motivated by our MHA locality analysis. This allows us to test novel ar… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Ablation studies. (a) Data scale: Increasing fine-tuning data from 10% to 20% improves FID. (b) Layer selection strategies: Interleaved replacement outperforms other heuristics. MHA results. Replacing MHA operators in DiT-XL/2 via grafting yields strong quality-efficie…
Figure 5
Figure 5. Figure 5: Convert model depth → width via grafting: (a) Two sequential transformer lay￾ers. (b) Rewiring in parallel via grafting (includes skip connections). Can we rewire two sequential transformer blocks to run in parallel? Our MLP grafting results showed that MLPs are amenab…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 30 canonical work pages

  1. [1]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023. 1, 3, 10

    work page 2023

  2. [2]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. 1

    work page 2024

  3. [3]

    Photorealistic video generation with diffusion models

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662, 2023. 1

    arXiv 2023

  4. [4]

    Plant grafting: new mechanisms, evolutionary implications

    Eliezer Goldschmidt. Plant grafting: new mechanisms, evolutionary implications. Frontiers in Plant Science, 5:727, 12 2014. 2

    work page 2014

  5. [5]

    Lolcats: On low-rank linearizing of large language models

    Michael Zhang, Simran Arora, Rahul Chalamala, Benjamin Frederick Spector, Alan Wu, Krithik Ramesh, Aaryan Singhal, and Christopher Re. Lolcats: On low-rank linearizing of large language models. In The Thirteenth International Conference on Learning Representations, 2025. 2, 11

    work page 2025

  6. [6]

    The mamba in the llama: Distilling and accelerating hybrid models

    Junxiong Wang, Daniele Paliotta, Avner May, Alexander M Rush, and Tri Dao. The mamba in the llama: Distilling and accelerating hybrid models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 2, 11

    work page 2024

  7. [7]

    Transformers to ssms: Distilling quadratic knowledge to subquadratic models

    Aviv Bick, Kevin Li, Eric Xing, J Zico Kolter, and Albert Gu. Transformers to ssms: Distilling quadratic knowledge to subquadratic models. Advances in Neural Information Processing Systems, 37:31788–31812,

    work page

  8. [8]

    Monarch mixer: A simple sub-quadratic gemm- based architecture

    Dan Fu, Simran Arora, Jessica Grogan, Isys Johnson, Evan Sabri Eyuboglu, Armin Thomas, Benjamin Spector, Michael Poli, Atri Rudra, and Christopher Ré. Monarch mixer: A simple sub-quadratic gemm- based architecture. Advances in Neural Information Processing Systems , 36:77546–77603, 2023. 2, 6

    work page 2023

Show all 52 references
  1. [9]

    Sparse upcycling: Training mixture-of-experts from dense checkpoints

    Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-experts from dense checkpoints. In The Eleventh International Conference on Learning Repre...

    2023

  2. [10]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. 2, 6

    2001 arXiv

  3. [11]

    Pixart- P: Weak-to-strong training of diffusion transformer for 4k text-to-image generation, 2024

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- P: Weak-to-strong training of diffusion transformer for 4k text-to-image generation, 2024. 3, 9

    2024

  4. [12]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 3

    2020

  5. [13]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. 3

    2022 arXiv

  6. [14]

    Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, YU Jincheng, GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In The Twelfth International Conference on Learning Repr...

    2023

  7. [15]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255,

    2009

  8. [16]

    Geneval: An object-focused framework for evaluating text-to-image alignment

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132–52152,

  9. [17]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NeurIPS Deep Learning and Representation Learning Workshop, 2015. 4

    2015

  10. [18]

    Peter J. Huber. Robust Estimation of a Location Parameter.The Annals of Mathematical Statistics, 35(1):73 – 101, 1964. 4

    1964

  11. [19]

    Benign overfitting in linear regression

    Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):30063–30070, 2020. 5 12

    2020

  12. [20]

    Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling

    Sanghoon Kim, Dahyun Kim, Chanjun Park, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, et al. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. In Proceedings of the 2024 Conference of the North Americ...

  13. [21]

    Compute better spent: Replacing dense layers with structured matrices

    Shikai Qiu, Andres Potapczynski, Marc Anton Finzi, Micah Goldblum, and Andrew Gordon Wilson. Compute better spent: Replacing dense layers with structured matrices. In International Conference on Machine Learning, pages 41698–41716. PMLR, 2024. 6

    2024

  14. [22]

    The impact of depth on compositional generalization in transformer language models

    Jackson Petty, Sjoerd Steenkiste, Ishita Dasgupta, Fei Sha, Dan Garrette, and Tal Linzen. The impact of depth on compositional generalization in transformer language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computationa...

    2024

  15. [23]

    Mechanistic design and scaling of hybrid architectures

    Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Björn Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, et al. Mechanistic design and scaling of hybrid architectures. arXiv preprint arXiv:2403.17844, 2024. 6

    2024 arXiv

  16. [24]

    Systems and algorithms for convo- lutional multi-hybrid language models at scale

    Jerome Ku, Eric Nguyen, David W Romero, Garyk Brixi, Brandon Yang, Anton V orontsov, Ali Taghibakhshi, Amy X Lu, Dave P Burke, Greg Brockman, et al. Systems and algorithms for convo- lutional multi-hybrid language models at scale. arXiv preprint arXiv:2503.01868, 2025. 6, 7

    2025 arXiv

  17. [25]

    Durrant, Jerome Ku, Michael Poli, Greg Brockman, Daniel Chang, Gabriel A

    Garyk Brixi, Matthew G. Durrant, Jerome Ku, Michael Poli, Greg Brockman, Daniel Chang, Gabriel A. Gonzalez, Samuel H. King, David B. Li, Aditi T. Merchant, Mohsen Naghipourfar, Eric Nguyen, Chiara Ricci-Tam, David W. Romero, Gwanggyu Sun, Ali Taghibakshi, Anton V orontsov, Bra...

    2025

  18. [26]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv:2004.05150, 2020. 7

    2004 arXiv

  19. [27]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. 7

    1904 arXiv

  20. [28]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In Forty-first International Conference on Machine Learning, 2024. 7

    2024

  21. [29]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. 9

    2022

  22. [30]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775–5787, 2022. 9

    2022

  23. [31]

    Bk-sdm: A lightweight, fast, and cheap version of stable diffusion

    Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells, and Shinkook Choi. Bk-sdm: A lightweight, fast, and cheap version of stable diffusion. In European Conference on Computer Vision, pages 381–399. Springer, 2024. 10

    2024

  24. [32]

    Tinyfusion: Diffusion transformers learned shallow

    Gongfan Fang, Kunjun Li, Xinyin Ma, and Xinchao Wang. Tinyfusion: Diffusion transformers learned shallow. arXiv preprint arXiv:2412.01199, 2024. 10

    2024 arXiv

  25. [33]

    All are worth words: A vit backbone for diffusion models

    Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22669–22679, 2023. 10

    2023

  26. [34]

    Sana: Efficient high-resolution image synthesis with linear diffusion transformers

    Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. arXiv preprint arXiv:2410.10629, 2024. 10

    arXiv

  27. [35]

    Diffusion models without attention

    Jing Nathan Yan, Jiatao Gu, and Alexander M Rush. Diffusion models without attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8239–8249, 2024. 10 13

    2024

  28. [36]

    Scalable diffusion models with state space backbone

    Zhengcong Fei, Mingyuan Fan, Changqian Yu, and Junshi Huang. Scalable diffusion models with state space backbone. arXiv preprint arXiv:2402.05608, 2024. 10

    2024 arXiv

  29. [37]

    Zigma: Zigzag mamba diffusion model

    Vincent Tao Hu, Stefan Andreas Baumann, Ming Gui, Olga Grebenkova, Pingchuan Ma, Johannes Fischer, and Bjorn Ommer. Zigma: Zigzag mamba diffusion model. arXiv preprint arXiv:2403.13802, 2024. 10

    2024 arXiv

  30. [38]

    Dim: Diffusion mamba for efficient high-resolution image synthesis

    Yao Teng, Yue Wu, Han Shi, Xuefei Ning, Guohao Dai, Yu Wang, Zhenguo Li, and Xihui Liu. Dim: Diffusion mamba for efficient high-resolution image synthesis. arXiv preprint arXiv:2405.14224, 2024. 10

    2024 arXiv

  31. [39]

    Dig: Scalable and efficient diffusion models with gated linear attention

    Lianghui Zhu, Zilong Huang, Bencheng Liao, Jun Hao Liew, Hanshu Yan, Jiashi Feng, and Xinggang Wang. Dig: Scalable and efficient diffusion models with gated linear attention. arXiv preprint arXiv:2405.18428,

    arXiv

  32. [40]

    Seaweed-7b: Cost-effective training of video generation foundation model

    Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, Feng Cheng, Feilong Zuo Xuejiao Zeng, Ziyan Yang, Fangyuan Kong, Zhiwu Qing, Fei Xiao, Meng Wei, Tuyen Hoang, Siyu Zhang, Peihao Zhu, Qi Zhao, Jiangqiao Yan, Lia...

    2025

  33. [41]

    A survey on video diffusion models

    Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang. A survey on video diffusion models. ACM Comput. Surv., 57(2), November 2024. 10

    2024

  34. [42]

    Matten: Video generation with mamba-attention

    Yu Gao, Jiancheng Huang, Xiaopeng Sun, Zequn Jie, Yujie Zhong, and Lin Ma. Matten: Video generation with mamba-attention. arXiv preprint arXiv:2405.03025, 2024. 10

    2024 arXiv

  35. [43]

    Lingen: Towards high-resolution minute-length text-to-video generation with linear computational complexity

    Hongjie Wang, Chih-Yao Ma, Yen-Cheng Liu, Ji Hou, Tao Xu, Jialiang Wang, Felix Juefei-Xu, Yaqiao Luo, Peizhao Zhang, Tingbo Hou, et al. Lingen: Towards high-resolution minute-length text-to-video generation with linear computational complexity. arXiv preprint arXiv:2412.09856,...

    2024 arXiv

  36. [44]

    Scaling diffusion transform- ers to 16 billion parameters

    Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, and Junshi Huang. Scaling diffusion transform- ers to 16 billion parameters. arXiv preprint arXiv:2407.11633, 2024. 10

    2024 arXiv

  37. [45]

    Star: Syn- thesis of tailored architectures

    Armin W Thomas, Rom Parnichkun, Alexander Amini, Stefano Massaroli, and Michael Poli. Star: Syn- thesis of tailored architectures. In The Thirteenth International Conference on Learning Representations,

  38. [46]

    Clear: Conv-like linearization revs pre-trained diffusion transformers up

    Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Clear: Conv-like linearization revs pre-trained diffusion transformers up. arXiv preprint arXiv:2412.16112, 2024. 11

    2024 arXiv

  39. [47]

    Linfusion: 1 gpu, 1 minute, 16k image

    Songhua Liu, Weihao Yu, Zhenxiong Tan, and Xinchao Wang. Linfusion: 1 gpu, 1 minute, 16k image. arXiv preprint arXiv:2409.02097, 2024. 11

    2024 arXiv

  40. [48]

    Edit: Efficient diffusion transformers with linear compressed attention

    Philipp Becker, Abhinav Mehrotra, Ruchika Chavhan, Malcolm Chadwick, Luca Morreale, Mehdi Noroozi, Alberto Gil Ramos, and Sourav Bhattacharya. Edit: Efficient diffusion transformers with linear compressed attention. arXiv preprint arXiv:2503.16726, 2025. 11

    2025 arXiv

  41. [49]

    Ffn fusion: Rethinking sequential computation in large language models

    Akhiad Bercovich, Mohammad Dabbah, Omri Puny, Ido Galil, Amnon Geifman, Yonatan Geifman, Izhak Golan, Ehud Karpas, Itay Levy, Zach Moshe, et al. Ffn fusion: Rethinking sequential computation in large language models. arXiv preprint arXiv:2503.18908, 2025. 11

    2025 arXiv

  42. [50]

    Hadzic, Taran Kota, Jimming He, Cristobal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Fei-Fei Li

    Keshigeyan Chandrasegaran, Agrim Gupta, Lea M. Hadzic, Taran Kota, Jimming He, Cristobal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Fei-Fei Li. Hourvideo: 1-hour video-language understanding. In Advances in Neural Information Processing Systems, volume 37, 2024. 11

    2024

  43. [51]

    Eagle 2.5: Boosting long-context post-training for frontier vision-language models

    Guo Chen, Zhiqi Li, Shihao Wang, Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Tuomas Rintamaki, et al. Eagle 2.5: Boosting long-context post-training for frontier vision-language models. arXiv preprint arXiv:2504.15271, 2025. 11

    2025

  44. [52]

    Hyena hierarchy: Towards larger convolutional language models

    Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pages 28043–28078. PMLR, 2023. ...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.