Pith. sign in

REVIEW 4 major objections 6 minor 70 references

Diffusion Transformer-to-Mamba Distillation for High-Resolution Image Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By distilling a diffusion transformer layer-by-layer into a hybrid Mamba model, this paper claims a 0.7B model matches teacher-level text-to-image quality while gaining 1.5–2.1× speed at 2048 and 4K resolution.

desk verdict Useful distillation recipe with credible 512 results, but the headline claim about causal Mamba standing alone is not supported—the model's global context really comes from its 4 self-attention layers. read the letter →

arxiv 2506.18999 v1 pith:KLJPPLOD submitted 2025-06-23 cs.CV

Yuan Yao , Yicong Hong , Difan Liu , Long Mai , Feng Liu , Jiebo Luo This is my paper · ORCID
classification cs.CV
keywords diffusiontransformersstatespacemodelsMambaknowledgedistillationhigh-resolutionimagegenerationtext-to-imagesynthesisteacherforcinghybridarchitecture
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a diffusion image-generation model built mostly from causal Mamba layers can be trained to match a transformer-based teacher by distilling the teacher layer by layer, rather than training the Mamba model from scratch. It claims the resulting 0.7B hybrid model, with 86% Mamba and 14% self-attention blocks, reaches teacher-level text-to-image quality (GenEval 0.485 vs teacher 0.481) while generating 2048×2048 images 1.5× faster and 4K images 2.1× faster. The deeper claim is that a sequential, causal state-space model can capture the non-causal global dependencies needed for coherent images, if the distillation feeds it the teacher's intermediate features as pseudo-ground truth. If this holds, it opens a cheaper path to high-resolution generation without quadratic attention cost.

What carries the argument

The load-bearing mechanism is layer-level teacher forcing: for each Mamba block, the student receives the teacher's intermediate input $h_{\theta'}^{(n)}$ as pseudo-ground-truth input and is trained to minimize the distance between its token-mixer output and the teacher's self-attention output at that same layer, $\lVert MA_\theta^{(n)}(h_{\theta'}^{(n)}) - SA_{\theta'}^{(n)}(h_{\theta'}^{(n)}) \rVert_2^2$, with the loss applied only at Mamba blocks. This converts a long causal chain into a set of per-layer imitation tasks, preventing early deviations from snowballing. The second mechanism is the bidirectional scanning scheme: width-first and height-first interleaved scans with shared Mamba weights whose outputs are combined by a linear projection, giving the causal 1D state-space model access to 2D context in both orientations.

What would settle it

Measure the trained student's effective receptive field by perturbing one patch of the noisy latent and measuring how the prediction for a distant patch changes, and compare against the teacher's attention map; if the student's influence decays sharply with distance while the teacher's does not, the model is not actually capturing long-range non-causal dependencies, and the GenEval match would be explained by local statistics rather than the claimed mechanism.

Watch

Extended reading notes

Core claim

The paper's central claim is that a diffusion model whose backbone is 86% causal Mamba layers can be brought to teacher-level generation quality by distilling a pretrained non-causal diffusion transformer instead of training the Mamba model from scratch. The recipe has three stages. First, layer-level teacher forcing feeds each student Mamba block the intermediate features that the teacher's corresponding self-attention block receives, and trains the Mamba block to output that block's attention result, so errors from earlier layers do not compound. Second, knowledge distillation keeps all non-token-mixer weights frozen and trains the token mixers against a diffusion MSE loss, teacher pseudo-labels, and per-layer mixer outputs. Third, model adaptation replaces the positional encoding and VAE before two rounds of high-resolution fine-tuning to 1024×1024 and 2048×2048. The resulting 0.7B model scores GenEval 0.485 versus the teacher's 0.481, reaches 8.63 FID-30K on COCO versus 7.32 for the teacher, and speeds up sampling by 1.5× at 2048×2048 and 2.1× at 3840×2160.

Load-bearing premise

The training pipeline works only if the bidirectional interleaved scanning gives the causal Mamba layers enough global, non-causal 2D context to imitate the teacher's self-attention; if that scan cannot represent long-range dependencies, the quality match and the feasibility claim collapse even though training remains stable.

Editorial extensions

If this is right

  • A pretrained diffusion transformer can be converted into a mostly-Mamba diffusion model by training only the token mixers, yielding a GenEval score (0.485) that slightly exceeds the teacher (0.481).
  • High-resolution generation no longer requires quadratic self-attention: the hybrid model produces 2048×2048 images 1.5× faster and 3840×2160 images 2.1× faster than the DiT baseline on the same GPU.
  • Keeping a small number of attention blocks matters: removing them drops the overall GenEval score from 0.485 to 0.420, while the 86/14 hybrid retains global dependencies.
  • After model adaptation, the distilled model can generate zero-shot 4K images even though it was fine-tuned only up to 2048×2048.
  • The layerwise teacher forcing is the key ingredient: adding it raises the two-object GenEval subscore and the overall score, confirming that it prevents error accumulation across Mamba layers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 'Mamba is hard to train' problem is mainly cumulative error propagation rather than a representational limit, the same layerwise teacher-forcing recipe should transfer to other non-causal-to-causal distillations, such as video or audio diffusion.
  • A direct testable extension would be to apply T2MD to a larger or stronger teacher and measure whether the speed advantage grows with token count; the linear-complexity argument predicts the gap widens at higher resolutions.
  • The paper stops short of analyzing effective receptive fields; one would expect the student's bidirectional scans to approximate the teacher's global attention on natural-image statistics but to diverge on images with long-range periodic or relational structure.
  • The model-adaptation stage recovers within 100k steps after swapping text encoder, VAE, and positional encoding, suggesting the distilled Mamba backbone is a stable platform; swapping in new conditioning encoders could be a cheap way to test that generality.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes T2MD, a multi-stage distillation pipeline that transfers a pre-trained PixArt-α diffusion transformer into a 0.7B hybrid diffusion model containing 24 Mamba layers and 4 self-attention layers. The stages are layer-level teacher forcing, feature-based knowledge distillation, component adaptation, and high-resolution fine-tuning, with the end goal of efficient 2048x2048 and zero-shot 4K text-to-image generation. The paper reports a GenEval score of 0.485 for the distilled model versus 0.481 for the teacher, an MS-COCO FID-30K of 8.63, and 1.5x to 2.1x latency speedups over a DiT baseline at high resolutions. It also makes a broader feasibility claim: that sequential and causal Mamba models can generate non-causal visual output.

Significance. If the claims hold, the work would be a useful contribution to efficient high-resolution text-to-image generation: it offers a concrete training recipe for avoiding expensive from-scratch training of state-space diffusion models and provides quantitative evidence that a hybrid Mamba model can approach a DiT teacher on GenEval. The layer-level teacher-forcing idea is plausible and the ablation table gives useful evidence about which components contribute. However, the strongest scientific claim, that causal Mamba alone can handle non-causal visual dependencies, is not supported by the architecture as described, since the two scan directions are both causal and the model contains self-attention layers. The high-resolution quality claims also currently rest on qualitative examples only. The paper is therefore potentially valuable, but the central claims need significant additional evidence or careful restatement.

major comments (4)
  1. [Sec 3.3, Table 1] The "bidirectional scanning strategy" is described only as width-first and height-first interleaved scans with shared weights and a linear fusion; both scans are causal in their traversal order, so no Mamba block has access to tokens to the right or below. The only non-causal global mechanism in the model is therefore the four self-attention blocks in the Sec 4.1 pattern. The GenEval drop in the "No SA" row (0.420 vs 0.485) and the "Bi-dir→Uni-dir" row (0.448 vs 0.485) show that these design choices matter, but they do not test whether a pure causal-Mamba stack can handle non-causal visual dependencies. Please add a reverse-scan ablation and a receptive-field analysis, or restrict the feasibility claim to the hybrid architecture.
  2. [Abstract and Table 1] The abstract and Sec 1 state that T2MD improves GenEval by 0.28, while Table 1 reports 0.301 to 0.485, an improvement of 0.184. This discrepancy concerns the headline quantitative result and must be corrected in the final version.
  3. [Sec 4.3, Figs 1 and 6] High-resolution and zero-shot 4K claims are supported only by qualitative figures and by the latency/throughput measurements in Table 3; no FID, GenEval, or user-study numbers are reported for 1024, 2048, or 4K, and no comparison is made to high-resolution-capable baselines such as SDXL, PixArt-sigma, or LinFusion at those resolutions. Since the first beyond-2k SSM text-to-image claim is a central contribution, quantitative quality evaluation at high resolution is required.
  4. [Sec 4.3.2, Table 2] The claim that the student "reaches the teacher model in quality" is only supported on GenEval; on MS-COCO FID-30K the student is worse than the teacher (8.63 vs 7.32, where lower is better). Please qualify the claim or provide additional metrics that substantiate teacher-level quality.
minor comments (6)
  1. [Sec 1 and Sec 4.3.2] The word "casual" appears where "causal" is meant in the introduction and in Sec 4.3.2; please fix throughout.
  2. [Sec 4.3.2] The model is referred to as "T2DM" twice in Sec 4.3.2; use "T2MD" consistently.
  3. [Table 3] The "SA latency" column is not defined; label the baseline explicitly and state whether it is the teacher PixArt-α or another DiT configuration.
  4. [Table 1] The "Bi-dir→Uni-dir" ablation is not described in the text; please define exactly what change is applied and what the row is meant to show.
  5. [Sec 3.4.2, Eq (10)] The token-mixer loss is written in terms of ε[n] without defining the per-block output convention; clarify the notation so the reader can tell what is compared at each layer.
  6. [Sec 1] The contribution claim of being the first SSM-based beyond-2k text-to-image model is not verified against recent concurrent work; consider a more careful survey or a qualified statement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: T2MD's distillation pipeline is self-contained, its central claims are evaluated on external benchmarks, and no fitted parameter is renamed as a prediction.

full rationale

The paper's derivation chain is a standard teacher-student distillation setup: a frozen PixArt-alpha teacher provides pseudo-ground-truth features (Eq. 6), soft labels (Eq. 9), and token-mixer targets (Eq. 10), while the student Mamba is trained to minimize those objectives plus the diffusion MSE loss (Eq. 8). These objectives define the training procedure rather than serving as the paper's evidence. The central quantitative claims are checked against external benchmarks (GenEval and MS-COCO FID-30K) and against other published models, not against the training losses themselves. The teacher is used as an upper bound in Table 1, but reaching or approaching an upper bound is an empirical outcome, not a logical consequence of the loss design. No load-bearing result is justified solely by a self-citation; the cited Mamba 2 and state-space-duality works are external, and the paper does not invoke a uniqueness theorem or an author-generated constraint to force its architectural choice. The skeptical concern that the width-first and height-first scans are both causal, leaving the four self-attention layers to carry non-causal context, is a correctness or empirical-support issue about whether the feasibility claim is fully established; it is not circularity, because the claim is not true by construction and is in principle falsifiable by ablations such as the 'No SA' row in Table 1. Accordingly, the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on a small set of hand-chosen hyperparameters, a frozen strong teacher, and an unverified assumption that bidirectional scanning gives causal Mamba access to global 2D context. No new physical or conceptual entities are introduced.

free parameters (4)
  • Distillation loss weights lambda1, lambda2 = 0.5, 0.2
    Set in Sec 4.1 without reported sensitivity study; they control the balance between pseudo-loss and token-mixer loss and directly shape distilled quality.
  • Hybrid architecture ratio = 4 self-attention blocks, 24 Mamba blocks
    Chosen by hand in Sec 3.3; only a no-SA ablation is reported, with no search over intermediate SA/Mamba ratios.
  • High-resolution fine-tuning schedule = 40k steps with 80% 1024 data, then 20k steps at 2048
    Hand-set in Sec 3.4.4; no ablations are reported for these durations or data mixes.
  • Mamba model configuration = state dim 256, expand factor 2, hidden 1152, patch size 2
    Architecture choices in Sec 4.1 are stated without ablations or justification beyond matching teacher scale.
assumptions (4)
  • domain assumption State-space duality between attention and Mamba layers makes layer-wise feature matching a sufficient transfer mechanism.
    Invoked in Sec 3.1 as 'state space duality provides theoretical soundness', but the actual method uses heuristic L2 feature matching and no formal guarantee is given.
  • domain assumption Bidirectional scanning with shared weights over width-first and height-first raster orders captures global 2D non-causal context for image generation.
    Stated in Sec 3.3; no receptive-field analysis is provided, and the central feasibility claim for causal Mamba in non-causal visual tasks rests on this.
  • domain assumption The frozen PixArt-alpha teacher is an adequate upper-bound teacher for a 512x512 diffusion Mamba base model.
    Used throughout Sec 3.4; the teacher's limitations bound the student, and model adaptation later replaces VAE, T5, and positional encodings without re-deriving teacher constraints.
  • domain assumption Mamba models are harder to train from scratch than transformers, motivating distillation.
    The paper relies on cited empirical results (Waleffe et al. 2024) rather than its own controlled comparison, and this premise drives the entire multi-stage design.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Diffusion Transformer-to-Mamba Distillation for High-Resolution Image Generation." pith.science (2026). https://pith.science/paper/KLJPPLOD

@misc{pith2026250618999,
  author       = {Pith},
  title        = {Pith review of: Diffusion Transformer-to-Mamba Distillation for High-Resolution Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KLJPPLOD}},
  note         = {Machine review of arXiv:2506.18999}
}
abstract

The quadratic computational complexity of self-attention in diffusion transformers (DiT) introduces substantial computational costs in high-resolution image generation. While the linear-complexity Mamba model emerges as a potential alternative, direct Mamba training remains empirically challenging. To address this issue, this paper introduces diffusion transformer-to-mamba distillation (T2MD), forming an efficient training pipeline that facilitates the transition from the self-attention-based transformer to the linear complexity state-space model Mamba. We establish a diffusion self-attention and Mamba hybrid model that simultaneously achieves efficiency and global dependencies. With the proposed layer-level teacher forcing and feature-based knowledge distillation, T2MD alleviates the training difficulty and high cost of a state space model from scratch. Starting from the distilled 512$\times$512 resolution base model, we push the generation towards 2048$\times$2048 images via lightweight adaptation and high-resolution fine-tuning. Experiments demonstrate that our training path leads to low overhead but high-quality text-to-image generation. Importantly, our results also justify the feasibility of using sequential and causal Mamba models for generating non-causal visual output, suggesting the potential for future exploration.

Figures

Figures reproduced from arXiv: 2506.18999 by the authors.

Figure 1
Figure 1. High-resolution generation results from our 0.7B T2MD model. The resolutions of these images in the top and bottom rows are [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The architecture of our hybrid Mamba model. During [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the teacher-student alignment in layer [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Results for each 512×512 stage. During the teacher forcing stage and knowledge distillation stage, where we only train the token mixers, most of the image details have been preserved. The orange case demonstrates that the Mamba layers are capable of learning the semant…
Figure 5
Figure 5. Figure 5: Without model adaptation, the original 512 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Examples of zero-shot 4k generation. Although our model has not been trained with 4k data, it is capable generate 4k examples. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 21 canonical work pages

  1. [1]

    One transformer fits all distributions in multi-modal diffu- sion at scale

    Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffu- sion at scale. InInternational Conference on Machine Learn- ing, pages 1692–1717. PMLR, 2023. 3

    2023

  2. [2]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  3. [3]

    Cross-layer distillation with semantic calibration

    Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, and Chun Chen. Cross-layer distillation with semantic calibration. In Proceedings of the AAAI Con- ference on Artificial Intelligence, pages 7028–7036, 2021. 3

    work page 2021

  4. [4]

    Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. 3, 4, 8

    arXiv

  5. [5]

    Pixart-\sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-\sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403.04692, 2024. 6

    arXiv 2024

  6. [6]

    Gentron: Delving deep into dif- fusion transformers for image and video generation

    Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. Gentron: Delving deep into dif- fusion transformers for image and video generation. arXiv preprint arXiv:2312.04557, 2023. 3

    arXiv 2023

  7. [7]

    Pali- x: On scaling up a multilingual vision and language model,

    Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, Siamak Shak- eri, Mostafa Dehghani, Daniel Salz, Mario Lucic, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang, Ceslee Montgomery, Paulina Pietrzyk, Marvin Ritter, AJ Piergiovanni, Matthias ...

    work page

  8. [8]

    Scaling instruction- finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024. 6

    2024

Show all 70 references
  1. [9]

    Transformers are ssms: General- ized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: General- ized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024. 3, 4

    2024 arXiv

  2. [10]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural informa- tion processing systems, 34:8780–8794, 2021. 2

    2021

  3. [11]

    Dimba: Transformer- mamba diffusion models

    Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, Youqiang Zhang, and Junshi Huang. Dimba: Transformer- mamba diffusion models. arXiv preprint arXiv:2406.01159,

    arXiv

  4. [12]

    Hungry hungry hippos: To- wards language modeling with state space models

    Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher R´e. Hungry hungry hippos: To- wards language modeling with state space models. arXiv preprint arXiv:2212.14052, 2022. 3

    2022 arXiv

  5. [13]

    Simple hardware-efficient long convolutions for se- quence modeling

    Daniel Y Fu, Elliot L Epstein, Eric Nguyen, Armin W Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christo- pher R´e. Simple hardware-efficient long convolutions for se- quence modeling. In International Conference on Machine Learning, pages 10373–10391. PMLR, 2023. 3

    2023

  6. [14]

    Lumina-t2x: Trans- forming text into any modality, resolution, and duration via flow-based large diffusion transformers

    Peng Gao, Le Zhuo, Chris Liu, , Ruoyi Du, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Trans- forming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arXiv:2405.05945, 2024. 3

    2024 arXiv

  7. [15]

    Masked diffusion transformer is a strong image synthesizer, 2023

    Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is a strong image synthesizer, 2023. 3

    2023

  8. [16]

    Matten: Video generation with mamba- attention

    Yu Gao, Jiancheng Huang, Xiaopeng Sun, Zequn Jie, Yujie Zhong, and Lin Ma. Matten: Video generation with mamba- attention. arXiv preprint arXiv:2405.03025, 2024. 3

    2024 arXiv

  9. [17]

    Geneval: An object-focused framework for evaluating text- to-image alignment

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text- to-image alignment. Advances in Neural Information Pro- cessing Systems, 36:52132–52152, 2023. 6

    2023

  10. [18]

    Knowledge distillation: A survey

    Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. Interna- tional Journal of Computer Vision, 129(6):1789–1819, 2021. 3

    2021

  11. [19]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. 1, 3, 4

    2023 arXiv

  12. [20]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher R ´e. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021. 3

    2021 arXiv

  13. [21]

    State-space models

    James D Hamilton. State-space models. Handbook of econo- metrics, 4:3039–3080, 1994. 3

    1994

  14. [22]

    A comprehensive overhaul of feature distillation

    Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, No- jun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. InProceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 1921–1930,

    1921

  15. [23]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 7

    2017

  16. [24]

    Distill- ing the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distill- ing the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2(7), 2015. 3

    2015 arXiv

  17. [25]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 2

    2020

  18. [26]

    Cascaded diffusion models for high fidelity image generation

    Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. arXiv preprint arXiv:2106.15282, 2021. 2

    2021 arXiv

  19. [27]

    Zigma: Zigzag mamba diffusion model

    Vincent Tao Hu, Stefan Andreas Baumann, Ming Gui, Olga Grebenkova, Pingchuan Ma, Johannes Fischer, and Bjorn Ommer. Zigma: Zigzag mamba diffusion model. arXiv preprint arXiv:2403.13802, 2024. 1, 3

    2024 arXiv

  20. [28]

    Ml-mamba: Efficient multi-modal large language model utilizing mamba-2

    Wenjun Huang and Jianguo Hu. Ml-mamba: Efficient multi-modal large language model utilizing mamba-2. arXiv preprint arXiv:2407.19832, 2024. 1, 3

    2024 arXiv

  21. [29]

    Efficient movie scene detection using state-space transformers

    Md Mohaiminul Islam, Mahmudul Hasan, Kishan Shamsun- dar Athrey, Tony Braskich, and Gedas Bertasius. Efficient movie scene detection using state-space transformers. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 18749–18758, 2023. 1, 3

    2023

  22. [30]

    Just chop: Em- barrassingly simple llm compression, 2024

    Ananya Harsh Jha, Tom Sherborne, Evan Pete Walsh, Dirk Groeneveld, Emma Strubell, and Iz Beltagy. Just chop: Em- barrassingly simple llm compression, 2024. 3

    2024

  23. [31]

    Knowledge dis- tillation via route constrained optimization

    Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, and Xiaolin Hu. Knowledge dis- tillation via route constrained optimization. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 1345–1354, 2019. 3

    2019

  24. [32]

    Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, and Noah A. Smith. Finetuning pretrained transformers into rnns, 2021. 3

    2021

  25. [33]

    Videomamba: State space model for efficient video understanding

    Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding. In European Conference on Computer Vision, pages 237–255. Springer, 2024. 3

    2024

  26. [34]

    Jamba: A hybrid transformer-mamba language model

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887, 2024. 1, 3

    2024 arXiv

  27. [35]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  28. [36]

    Linfusion: 1 gpu, 1 minute, 16k image.arXiv preprint arXiv:2409.02097, 2024

    Songhua Liu, Weihao Yu, Zhenxiong Tan, and Xinchao Wang. Linfusion: 1 gpu, 1 minute, 16k image.arXiv preprint arXiv:2409.02097, 2024. 3, 7, 8

    2024 arXiv

  29. [37]

    Instaflow: One step is enough for high-quality diffusion- based text-to-image generation

    Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality diffusion- based text-to-image generation. In The Twelfth International Conference on Learning Representations, 2023. 7

    2023

  30. [38]

    Vmamba: Visual state space model

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. 2024. 1, 3

    2024

  31. [39]

    Long range language modeling via gated state spaces

    Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces. arXiv preprint arXiv:2206.13947, 2022. 3

    2022 arXiv

  32. [40]

    Lin- earizing large language models

    Jean Mercat, Igor Vasiljevic, Sedrick Scott Keh, Kushal Arora, Achal Dave, Adrien Gaidon, and Thomas Kollar. Lin- earizing large language models. 2024. 1, 3

    2024

  33. [41]

    Scaling diffusion mamba with bidirectional ssms for efficient image and video gener- ation

    Shentong Mo and Yapeng Tian. Scaling diffusion mamba with bidirectional ssms for efficient image and video gener- ation. arXiv preprint arXiv:2405.15881, 2024. 1, 3

    2024 arXiv

  34. [42]

    S4nd: Modeling images and videos as multidimensional signals with state spaces

    Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher R ´e. S4nd: Modeling images and videos as multidimensional signals with state spaces. Advances in neural information processing systems, 35:2846–2861, 2022. 1, 3

    2022

  35. [43]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021. 8

    2021 arXiv

  36. [44]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022. 2

    2022

  37. [45]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  38. [46]

    Rwkv: Reinventing rnns for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023. 3

    2023 arXiv

  39. [47]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 2, 6

    2023 arXiv

  40. [48]

    Hyena hierarchy: Towards larger con- volutional language models

    Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher R ´e. Hyena hierarchy: Towards larger con- volutional language models. In International Conference on Machine Learning, pages 28043–28078. PMLR, 2023. 1, 3

    2023

  41. [49]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 4

    2020

  42. [50]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International confer- ence on machine learning, pages 8821–8831. Pmlr, 2021. 8

    2021

  43. [51]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2, 7, 8

    2022

  44. [52]

    U-net: Convolutional networks for biomedical image segmentation,

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation,

  45. [53]

    Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion mod...

    2022

  46. [54]

    Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis

    Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. In International con- ference on machine learning , pages 30105–30118. PMLR,

  47. [55]

    Convolutional state space models for long-range spatiotemporal modeling.Advances in Neural Information Processing Systems, 36, 2024

    Jimmy Smith, Shalini De Mello, Jan Kautz, Scott Linder- man, and Wonmin Byeon. Convolutional state space models for long-range spatiotemporal modeling.Advances in Neural Information Processing Systems, 36, 2024. 3

    2024

  48. [56]

    Dim: Diffusion mamba for efficient high-resolution image synthesis

    Yao Teng, Yue Wu, Han Shi, Xuefei Ning, Guohao Dai, Yu Wang, Zhenguo Li, and Xihui Liu. Dim: Diffusion mamba for efficient high-resolution image synthesis. arXiv preprint arXiv:2405.14224, 2024. 1, 3

    2024 arXiv

  49. [57]

    Contrastive representation distillation

    Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv:1910.10699, 2019. 3

    1910 arXiv

  50. [58]

    Similarity-preserving knowl- edge distillation

    Frederick Tung and Greg Mori. Similarity-preserving knowl- edge distillation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision , pages 1365–1374,

  51. [59]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Red Hook, NY , USA,

  52. [60]

    An empirical study of mamba-based language models

    Roger Waleffe, Wonmin Byeon, Duncan Riach, Bran- don Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of mamba-based language models. arXiv preprint arXiv:2406.07887, 2024. 1, 3, 4, 5

    2024 arXiv

  53. [61]

    The mamba in the llama: Distilling and accelerating hybrid models

    Junxiong Wang, Daniele Paliotta, Avner May, Alexan- der M Rush, and Tri Dao. The mamba in the llama: Distilling and accelerating hybrid models. arXiv preprint arXiv:2408.15237, 2024. 7

    2024 arXiv

  54. [62]

    Sheared llama: Accelerating language model pre-training via structured pruning

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694, 2023. 3

    2023 arXiv

  55. [63]

    Difffit: Un- locking transferability of large diffusion models via sim- ple parameter-efficient fine-tuning

    Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Un- locking transferability of large diffusion models via sim- ple parameter-efficient fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Visi...

    2023

  56. [64]

    A survey on vision mamba: Models, applications and chal- lenges

    Rui Xu, Shu Yang, Yihui Wang, Bo Du, and Hao Chen. A survey on vision mamba: Models, applications and chal- lenges. arXiv preprint arXiv:2404.18861, 2024. 1, 3

    2024 arXiv

  57. [65]

    The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry, 2024

    Michael Zhang, Kush Bhatia, Hermann Kumbong, and Christopher R´e. The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry, 2024. 1, 3

    2024

  58. [66]

    Cobra: Extending mamba to multi-modal large language model for efficient inference

    Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang, and Donglin Wang. Cobra: Extending mamba to multi-modal large language model for efficient inference. arXiv preprint arXiv:2403.14520, 2024. 1, 3

    2024 arXiv

  59. [67]

    Fast training of diffusion models with masked transformers

    Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers. In Transactions on Machine Learning Re- search (TMLR), 2024. 3

    2024

  60. [68]

    Rethinking soft la- bels for knowledge distillation: A bias-variance tradeoff per- spective

    Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft la- bels for knowledge distillation: A bias-variance tradeoff per- spective. arXiv preprint arXiv:2102.00650, 2021. 3

    2021 arXiv

  61. [69]

    Lumina-next: Making lumina-t2x stronger and faster with next-dit

    Le Zhuo, Ruoyi Du, Xiao Han, Yangguang Li, Dongyang Liu, Rongjie Huang, Wenze Liu, et al. Lumina-next: Making lumina-t2x stronger and faster with next-dit. arXiv preprint arXiv:2406.18583, 2024. 3

    2024 arXiv

  62. [2017]

    Curran Associates Inc. 3

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.