Pith. sign in

REVIEW 3 major objections 5 minor 51 references

Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Deep fusion of a frozen LLM and a trainable DiT outperforms shallow fusion for text-to-image alignment.

desk verdict A careful, reproducible empirical study of deep fusion for text-to-image, but the headline gap sits at the GenEval noise floor and needs repeated runs to support the 'significantly better' claim. read the letter →

arxiv 2505.10046 v1 pith:4JTHLNII submitted 2025-05-15 cs.CV

Bingda Tang , Boyang Zheng , Xichen Pan , Sayak Paul , Saining Xie This is my paper · ORCID
classification cs.CV
keywords text-to-imagesynthesisdiffusiontransformerslargelanguagemodelsdeepfusionimage-textalignmentrectifiedflowtimestepconditioningcontrolledcomparison
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether deep fusion—feeding a frozen large language model and a trainable diffusion transformer through shared self-attention at every layer—earns its keep for text-to-image synthesis, compared with simpler ways of injecting LLM text into a DiT. Under matched architecture and training, the deep-fusion model reaches GenEval 0.51, above a self-attention DiT's 0.42 and a cross-attention DiT's 0.49, at similar or slightly lower inference latency. The paper maps the design space: dropping timestep-conditioning parameters improves FID and cuts parameters by about 20 percent, separating 1D rotary position embedding for text from 2D rotary position embedding for images works best, and upgrading the base LLM boosts alignment, while instruction-tuned variants do not. The resulting 2B-parameter model, FuseDiT, trained on roughly 26M image-caption pairs, reaches GenEval 0.60 and FID 7.54, competitive with several much larger systems on a benchmark where data and compute are limited. For a reader, the value is a reproducible recipe and concrete evidence about which fusion mechanism actually drives image-text alignment.

What carries the argument

The central object is the layer-wise shared self-attention module that concatenates the token sequences of the frozen LLM stream and the trainable DiT stream at every layer. The text sequence keeps a causal attention mask and the image sequence a bidirectional mask, so image tokens can attend to text tokens but not vice versa; the LLM's key and value states are constant across diffusion timesteps and can be cached for inference. The model is trained with a rectified-flow objective on velocity prediction, then sampled with 25-step Euler discretization and classifier-free guidance. This two-stream arrangement is what lets the DiT draw on each LLM layer's internal information flow instead of a single pooled text representation.

What would settle it

Retrain the three main architectures under the paper's exact recipe with three or more random seeds each and check whether the GenEval ordering (deep fusion above cross-attention above self-attention) and the size of the gaps persist. If the gaps fall within seed-to-seed variance, the central comparison collapses. A second check is to re-run the timestep-free variant on the 26M-image dataset: if FID no longer improves, the parameter-removal result is scale-dependent.

Watch

Extended reading notes

Core claim

The central claim is that layer-wise shared self-attention between a frozen LLM and a trainable DiT gives better image-text alignment than shallow fusion, in which each DiT layer ingests key and value states projected from a single LLM layer's hidden states. The authors reinterpret deep fusion as a variant of self-attention DiT whose conditional key and value states come from the LLM's internal layers rather than from a trainable projection, and they show this design beats both self-attention and cross-attention shallow baselines on GenEval and DPG-Bench under controlled conditions. They also report several counterintuitive findings: the adaLN-Zero timestep-conditioning branch can be removed with better FID and a 20 percent parameter saving; instruction-tuned LLMs slightly hurt; and moving to a stronger base LLM gives a large alignment boost, indicating that the DiT's performance is tightly coupled to the LLM's capabilities.

Load-bearing premise

The load-bearing premise is that a single training run per configuration is enough to rank these architectures; none of the results tables reports multiple seeds or error bars, so the headline gaps (for example GenEval 0.51 versus 0.42) could in principle be run-to-run variance.

Editorial extensions

If this is right

  • Deep fusion becomes a directly competitive conditioning mechanism: under the paper's controlled setup it surpasses cross-attention DiTs in image-text alignment at lower inference latency, so alignment gains do not have to be bought with slower sampling.
  • Because the frozen LLM's text key and value states are constant across timesteps, KV caching lets this architecture scale to longer or repeated text conditioning at little extra cost.
  • Removing adaLN-Zero timestep conditioning improves FID and removes about 20 percent of parameters, suggesting rectified-flow DiTs can be simplified substantially without losing alignment.
  • Base-LLM quality transfers to the DiT: swapping in a stronger base LLM raised GenEval from 0.51 to 0.54, pointing to LLM capability as a primary lever for image-text alignment.
  • Text and image sequences want different position encodings: 1D RoPE for text with 2D RoPE for images outperforms unified 1D RoPE and M-RoPE, giving concrete guidance for multimodal transformer design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The instruction-tuning result is probably about pretraining objectives, not instruction-following ability: if true, further gains should come from base LLMs trained for representation quality or longer context rather than from chat-tuned variants. A direct test would compare several base-versus-instructed LLM pairs at the same size.
  • The graceful degradation when shrinking the DiT hidden size or layer count suggests LLM and DiT can be scaled independently, so future systems might pair a small DiT with a much larger frozen LLM instead of matching their backbones.
  • The KV-caching property plus the timestep-free finding point toward a simple unified multimodal architecture in which one frozen LLM conditions generation at every layer with almost no conditioning-specific parameters.
  • The timestep-free result at 10.9M images is surprising; scale may reverse it. Retraining FuseDiT with and without timestep conditioning on the full 26M dataset would tell whether the parameter saving survives at scale.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents an empirical study of deep fusion of a frozen decoder-only LLM with a trainable diffusion transformer (DiT) for text-to-image synthesis. The authors introduce two shallow fusion baselines (self-attention DiT and cross-attention DiT) and compare them against deep fusion under controlled conditions (same data, same training budget, same inference procedure). They then explore key design choices: timestep conditioning strategies, positional encodings, base LLM selection and instruction tuning, and finally present FuseDiT, a model trained with the chosen recipe on roughly 26M image-text pairs. The central empirical claim is that deep fusion achieves significantly better image-text alignment (GenEval 0.51) than the self-attention DiT (0.42) and cross-attention DiT (0.49) under identical training, and that FuseDiT reaches GenEval 0.60, surpassing several established systems while using much less training data.

Significance. If the central comparison is statistically robust, this paper is a valuable contribution to the text-to-image literature. It systematically investigates an important and under-documented architecture family, provides an unusually detailed and reproducible training recipe (including datasets, hyperparameters, and code release), and offers concrete data points on design choices such as the surprising effectiveness of removing timestep conditioning. The controlled comparison between deep and shallow fusion is a useful reference for practitioners. However, the strength of the main claim depends critically on the statistical reliability of the reported differences, and the current single-run evidence does not fully support the word 'significantly' for the deep-versus-cross-attention gap.

major comments (3)
  1. [Sec. 5.2, Table 1] The claim that deep fusion achieves 'significantly better performance' than the shallow fusion baselines is not supported by the evidence presented. Each configuration is trained once, with no multiple seeds or error bars. GenEval is computed on roughly 553 prompts, so for a success probability near 0.5 the binomial standard error is about 0.02. The deep fusion versus cross-attention gap is 0.51 versus 0.49, which is within one standard error of the evaluation metric alone, before considering training stochasticity. Because this comparison is the paper's central result and motivates the entire subsequent design-space exploration, the lack of repeated runs is load-bearing. Please provide at least three training runs per configuration (or equivalent uncertainty quantification) for the central comparison, or substantially temper the language.
  2. [Sec. 6.1, Table 3] The decision to eliminate timestep conditioning entirely is based on a single run per strategy. The reported GenEval difference between adaLN-Zero (0.51) and w/o timestep (0.49) is again within the evaluation noise floor, even though the FID difference (27.33 versus 21.27) is larger. Given that removing timestep conditioning is surprising and contradicts the standard diffusion formulation, the empirical justification should be strengthened. Additional runs or stability analysis (e.g., training the same configuration with different seeds) are needed before this choice is incorporated into the final recipe. At minimum, state explicitly that this is a tentative observation from a single run.
  3. [Sec. 6.2 and 6.3, Tables 4 and 5] Several design-selection decisions are made based on differences of 0.01–0.02 in GenEval (e.g., 1D-RoPE+APE 0.51 versus 1D+2D-RoPE 0.51; Gemma 2B 0.51 versus + instruction tuning 0.49). These differences are all within the noise floor of a single evaluation set of this size. The paper should acknowledge explicitly that these pairwise differences are not reliably distinguishable from zero with a single run, and should either provide repeated runs or frame these conclusions as preliminary. This is not merely a presentation issue, because the final FuseDiT recipe is assembled from these choices.
minor comments (5)
  1. [Sec. 6.1, Fig. 4] There is a typo in the figure caption: 'Removing timesetp conditioning' should be 'Removing timestep conditioning'.
  2. [Table 3] The row for 'w/o timestep' reads '0.4976.7 21.27'; there is a missing space between the GenEval and DPG values. Please fix the table formatting.
  3. [Sec. 4, Inference paragraph] The text says 'a classifier-free guidance scale of 6 which we find to be near optimal for text-image alignment.' Since the paper later shows that removing timestep conditioning changes the model behavior, the optimal guidance scale for the final FuseDiT may differ; please state whether the same scale was used for all models and whether any re-tuning was considered.
  4. [Sec. 7.2, Table 6] The column header 'Gen.' is used for the GenEval metric; for consistency with the rest of the paper, write 'GenEval'. Also, the table is adapted from [48,50] but no footnote explains which numbers were taken from those sources versus reproduced by the authors; please clarify.
  5. [Sec. 2, Related Work] The related work section is thorough, but the discussion of LLM-based conditioning mechanisms would benefit from citing recent work on instruction-tuned LLM text encoders beyond [25,47], such as the role of prompt templates in diffusion models.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmarking study; its central claims are measured on external benchmarks rather than derived from fitted inputs or self-citations.

full rationale

The paper does not present a derivation chain in the sense of equations that reduce to their inputs. Its claims are empirical measurements from controlled training runs evaluated on external benchmarks (GenEval, DPG-Bench, MJHQ-30K FID). The central comparison in Table 1 trains self-attention DiT, cross-attention DiT, and deep fusion under the same setup and reports measured alignment and quality scores; no fitted parameter is renamed as a prediction. The design-space exploration in Sec. 6 selects options (e.g., 1D+2D RoPE, removing timestep conditioning) using the same GenEval/DPG/FID metrics that subsequently report the final FuseDiT numbers. This is a standard model-selection practice and a legitimate validity concern, but it is not a circular reduction: the final scores are measured outcomes of a chosen configuration, not quantities forced by construction. The only notable self-citation is [29] (DiT), which is foundational external architecture by one of the present authors and is not used to forbid alternatives or to justify the central deep-fusion claim. No uniqueness theorem, ansatz smuggling, renaming of a known result, or self-citation load-bearing step is present. The paper is self-contained against external benchmarks, so the circularity score is 0.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claims rest on standard diffusion/rectified-flow assumptions, the validity of the chosen evaluation benchmarks, and the representativeness of the constructed baselines. No invented entities are introduced.

free parameters (1)
  • Classifier-free guidance scale = 6
    Chosen empirically as near-optimal for text-image alignment on the used benchmarks; used across all models.
assumptions (5)
  • domain assumption The rectified flow objective (Eq. 1) is a valid and effective training objective for text-to-image synthesis.
    The paper uses rectified flow without questioning its suitability; standard in recent diffusion models.
  • domain assumption The frozen Gemma 2B LLM provides useful representations for image synthesis when fused with a DiT through shared self-attention.
    This is the central premise being tested; the paper assumes frozen LLM is sufficient and does not unfreeze it.
  • domain assumption GenEval, DPG-Bench, and FID on MJHQ-30K are reliable metrics for image-text alignment and visual quality, and differences on these benchmarks are meaningful.
    The paper relies on these benchmarks for all conclusions and notes DPG-Bench has limitations but still uses it.
  • domain assumption The shallow fusion baselines (self-attention DiT and cross-attention DiT) are representative of established text-to-image conditioning architectures.
    The paper constructs these baselines to represent the standard paradigm; if they were unfairly weak, the comparison would be biased.
  • domain assumption Removing timestep conditioning is a viable design for this scale, following [42], without breaking training stability.
    The paper adopts w/o timestep based on empirical results and prior work, but it is an architectural assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis." pith.science (2026). https://pith.science/paper/4JTHLNII

@misc{pith2026250510046,
  author       = {Pith},
  title        = {Pith review of: Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4JTHLNII}},
  note         = {Machine review of arXiv:2505.10046}
}
read the original abstract

This paper does not describe a new method; instead, it provides a thorough exploration of an important yet understudied design space related to recent advances in text-to-image synthesis -- specifically, the deep fusion of large language models (LLMs) and diffusion transformers (DiTs) for multi-modal generation. Previous studies mainly focused on overall system performance rather than detailed comparisons with alternative methods, and key design details and training recipes were often left undisclosed. These gaps create uncertainty about the real potential of this approach. To fill these gaps, we conduct an empirical study on text-to-image generation, performing controlled comparisons with established baselines, analyzing important design choices, and providing a clear, reproducible recipe for training at scale. We hope this work offers meaningful data points and practical guidelines for future research in multi-modal generation.

Figures

Figures reproduced from arXiv: 2505.10046 by the authors.

Figure 1
Figure 1. Illustration of the deep fusion approach and base￾lines. We conduct controlled comparisons with baseline methods that incorporate text representations from a single text encoder layer into each DiT layer using late fusion within the attention mechanism, a strategy we term as the “shallow fusion” approach. fusion within a single transformer [43], enabling seamless multi-modal generation. This unified approach support… view at source ↗
Figure 2
Figure 2. Illustration of the attention mask. Each dotted square indicates whether the row can attend to the column. To integrate LLMs into diffusion models, previous works attempt to replace text encoders with LLMs, either by training from scratch [25, 47, 51] or by aligning feature spaces [7, 15, 22, 28]. However, this substitution alone has not yielded the expected performance gains unless paired with sophisticated archite… view at source ↗
Figure 3
Figure 3. Illustration of cross-modal attention in the shallow fusion baselines. The key and query states of the condition are directly projected from text representations. • Self-attention DiT. In this design, text representations are projected to key and value states and then concatenated with those of image hidden states in self-attention, which can also be decoupled by running self-attention and cross￾attention in paralle… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Illustration of RoPE. The indices denote position IDs. tently enhances visual quality, whereas the performance in image-text alignment exhibits fluctuations. Notably, the strategy that completely removes timestep conditioning not only achieves significantly better FID …
Figure 6
Figure 6. Figure 6: Illustration of architecture alignment. Dashed boxes indicate parameters that have been reduced by decreasing either the hidden size or the number of layers. CC12M and SA-1B, we utilize synthetic captions [6, 9]. Other experimental setup follows Sec. 4. 7.2. Performanc…
Figure 7
Figure 7. Figure 7: Samples generated by FuseDiT. Method GenEval ↑ DPG ↑ FID ↓ self-attention 0.51 76.6 27.33 cross-attention 0.52 76.5 26.57 This modification yields minor gains, though at a cost to the LLM-DiT parity. Additionally, we find that although cross-attention introduces a negl…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 17 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. InNeurIPS,

  2. [2]

    Improving image generation with bet- ter captions.https://cdn

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with bet- ter captions.https://cdn. openai. com/papers/dall-e-3. pdf,

    work page

  3. [3]

    Paligemma: A versatile 3b vlm for trans- fer.arXiv:2407.07726, 2024

    Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexan- der Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for trans- fer.arXiv:2407.07726, 2024. 6

    arXiv 2024

  4. [4]

    Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. InCVPR,

    work page

  5. [5]

    Pixart-σ: Weak-to-strong training of diffu- sion transformer for 4k text-to-image generation

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-σ: Weak-to-strong training of diffu- sion transformer for 4k text-to-image generation. InECCV,

  6. [6]

    Pixart-α: Fast training of diffusion trans- former for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-α: Fast training of diffusion trans- former for photorealistic text-to-image synthesis. InICLR,

  7. [7]

    Dreamllm: Synergistic multimodal com- prehension and creation

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal com- prehension and creation. InICLR, 2024. 1, 2

    work page 2024

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InICLR, 2021. 2, 3

    2021

Show all 51 references
  1. [9]

    conceptual-captions-cc12m- llavanext.https://huggingface.co/datasets/ CaptionEmporium / conceptual - captions - cc12m-llavanext, 2024

    Caption Emporium. conceptual-captions-cc12m- llavanext.https://huggingface.co/datasets/ CaptionEmporium / conceptual - captions - cc12m-llavanext, 2024. 3, 7

    2024

  2. [10]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. InICML, 2024. 1, 2, 3, 4, 5, 7

    2024

  3. [11]

    Lumina-t2x: Transforming text into any modality, reso- lution, and duration via flow-based large diffusion transform- ers.arXiv:2405.05945, 2024

    Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Transforming text into any modality, reso- lution, and duration via flow-based large diffusion transform- ers.arXiv:2405.05945, 2024. 2, 4, 5, 7

    2024 arXiv

  4. [12]

    Geneval: An object-focused framework for evaluating text- to-image alignment.arXiv:2310.11513, 2023

    Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text- to-image alignment.arXiv:2310.11513, 2023. 3

    2023 arXiv

  5. [13]

    The llama 3 herd of models.arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Ab- hinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv:2407.21783, 2024. 7

    2024 arXiv

  6. [14]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. InNeurIPS, 2017. 3

    2017

  7. [15]

    Ella: Equip diffusion models with llm for en- hanced semantic alignment.arXiv:2403.05135, 2024

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for en- hanced semantic alignment.arXiv:2403.05135, 2024. 1, 2, 3

    2024 arXiv

  8. [16]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InICCV, 2023. 6

    2023

  9. [17]

    Announcing black forest labs.https: / / blackforestlabs

    Black Forest Labs. Announcing black forest labs.https: / / blackforestlabs . ai / announcing - black - forest-labs/, 2024. 1, 2, 5, 7

    2024

  10. [18]

    Playground v2

    Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Lin- miao Xu, and Suhail Doshi. Playground v2. 5: Three in- sights towards enhancing aesthetic quality in text-to-image generation.arXiv:2402.17245, 2024. 2, 3, 7

    2024 arXiv

  11. [19]

    Llm- grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models

    Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Llm- grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv:2305.13655, 2023. 2

    2023 arXiv

  12. [20]

    Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models.arXiv:2411.04996, 2024

    Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, et al. Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models.arXiv:2411.04996, 2024. 1, 2

    2024 arXiv

  13. [21]

    Playground v3: Improving text-to- image alignment with deep-fusion large language models

    Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Improving text-to- image alignment with deep-fusion large language models. arXiv:2409.10695, 2024. 1, 2, 3, 7

    2024 arXiv

  14. [22]

    Llm4gen: Leveraging semantic representation of llms for text-to-image generation.arXiv:2407.00737, 2024

    Mushui Liu, Yuhang Ma, Xinfeng Zhang, Yang Zhen, Zeng Zhao, Zhipeng Hu, Bai Liu, and Changjie Fan. Llm4gen: Leveraging semantic representation of llms for text-to-image generation.arXiv:2407.00737, 2024. 1, 2

    2024 arXiv

  15. [23]

    Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv:2209.03003, 2022

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv:2209.03003, 2022. 3

    2022 arXiv

  16. [24]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InICLR, 2019. 3

    2019

  17. [25]

    Exploring the role of large language models in prompt encoding for diffusion models.arXiv:2406.11831,

    Bingqi Ma, Zhuofan Zong, Guanglu Song, Hongsheng Li, and Yu Liu. Exploring the role of large language models in prompt encoding for diffusion models.arXiv:2406.11831,

    arXiv

  18. [26]

    Janusflow: Harmonizing autore- gression and rectified flow for unified multimodal under- standing and generation.arXiv:2411.07975, 2024

    Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Liang Zhao, et al. Janusflow: Harmonizing autore- gression and rectified flow for unified multimodal under- standing and generation.arXiv:2411.07975, 2024. 1, 2

    2024 arXiv

  19. [27]

    Gemma: Open models based on gemini research and technology

    Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi`ere, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv:2403.08295, 2024. 3

    2024 arXiv

  20. [28]

    Kosmos-g: Generating images in context with multimodal large language models

    Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos-g: Generating images in context with multimodal large language models. InICLR,

  21. [29]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InICCV, 2023. 1, 2, 3, 4, 5

    2023

  22. [30]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis.arXiv:2307.01952,

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis.arXiv:2307.01952,

    arXiv

  23. [31]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InICML, 2021. 1, 2, 5

    2021

  24. [32]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. InJMLR, 2020. 1, 2

    2020

  25. [33]

    Hierarchical text-conditional image gener- ation with clip latents.arXiv:2204.06125, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents.arXiv:2204.06125, 2022. 7

    2022 arXiv

  26. [34]

    Gemma 2: Improving open language models at a practical size

    Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cas- sidy Hardin, Surya Bhupatiraju, L´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram ´e, et al. Gemma 2: Improving open language models at a practical size. arXiv:2408.00118, 2024. 6

    2024 arXiv

  27. [35]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. InCVPR, 2022. 2, 7

    2022

  28. [36]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. InMICCAI, 2015. 2

    2015

  29. [37]

    Stretching each dol- lar: Diffusion training from scratch on a micro-budget

    Vikash Sehwag, Xianghao Kong, Jingtao Li, Michael Spranger, and Lingjuan Lyu. Stretching each dol- lar: Diffusion training from scratch on a micro-budget. arXiv:2407.15811, 2024. 7

    2024 arXiv

  30. [38]

    Lmfusion: Adapting pretrained language models for multimodal gener- ation.arXiv:2412.15188, 2024

    Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang, Xi Victoria Lin, Luke Zettlemoyer, and Lili Yu. Lmfusion: Adapting pretrained language models for multimodal gener- ation.arXiv:2412.15188, 2024. 1, 2

    2024 arXiv

  31. [39]

    Layer by layer: Uncovering hidden representations in language mod- els.arXiv:2502.02013, 2025

    Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language mod- els.arXiv:2502.02013, 2025. 7

    2025 arXiv

  32. [40]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 2024. 5

    2024

  33. [41]

    Journeydb: A benchmark for generative im- age understanding

    Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative im- age understanding. InNeurIPS, 2024. 6

    2024

  34. [42]

    Is noise conditioning necessary for denoising genera- tive models?arXiv:2502.13129, 2025

    Qiao Sun, Zhicheng Jiang, Hanhong Zhao, and Kaiming He. Is noise conditioning necessary for denoising genera- tive models?arXiv:2502.13129, 2025. 5

    2025

  35. [43]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeurIPS, 2017. 1

    2017

  36. [44]

    Divide and conquer: Language mod- els can plan and self-correct for compositional text-to-image generation.arXiv:2401.15688, 2024

    Zhenyu Wang, Enze Xie, Aoxue Li, Zhongdao Wang, Xihui Liu, and Zhenguo Li. Divide and conquer: Language mod- els can plan and self-correct for compositional text-to-image generation.arXiv:2401.15688, 2024. 2

    2024 arXiv

  37. [45]

    Self-correcting llm-controlled diffusion models

    Tsung-Han Wu, Long Lian, Joseph E Gonzalez, Boyi Li, and Trevor Darrell. Self-correcting llm-controlled diffusion models. InCVPR, 2024. 2

    2024

  38. [46]

    Omnigen: Unified image genera- tion.arXiv:2409.11340, 2024

    Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xin- grun Xing, Ruiran Yan, Chaofan Li, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image genera- tion.arXiv:2409.11340, 2024. 1, 2

    2024 arXiv

  39. [47]

    Sana: Ef- ficient high-resolution image synthesis with linear diffusion transformers.arXiv:2410.10629, 2024

    Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Yujun Lin, Zhekai Zhang, Muyang Li, Yao Lu, and Song Han. Sana: Ef- ficient high-resolution image synthesis with linear diffusion transformers.arXiv:2410.10629, 2024. 1, 2, 4, 6, 7

    2024 arXiv

  40. [48]

    Sana 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer

    Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, Han Cai, et al. Sana 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer. arXiv:2501.18427, 2025. 1, 7

    2025 arXiv

  41. [49]

    Mastering text-to-image diffu- sion: Recaptioning, planning, and generating with multi- modal llms

    Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Ste- fano Ermon, and Bin Cui. Mastering text-to-image diffu- sion: Recaptioning, planning, and generating with multi- modal llms. InICML, 2024. 2

    2024

  42. [50]

    Transfusion: Pre- dict the next token and diffuse images with one multi-modal model.arXiv:2408.11039, 2024

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Pre- dict the next token and diffuse images with one multi-modal model.arXiv:2408.11039, 2024. 1, 2, 5, 7

    2024 arXiv

  43. [51]

    Lumina-next: Making lumina-t2x stronger and faster with next-dit.arXiv:2406.18583, 2024

    Le Zhuo, Ruoyi Du, Han Xiao, Yangguang Li, Dongyang Liu, Rongjie Huang, Wenze Liu, Lirui Zhao, Fu-Yun Wang, Zhanyu Ma, et al. Lumina-next: Making lumina-t2x stronger and faster with next-dit.arXiv:2406.18583, 2024. 1, 2, 4

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.