Pith. sign in

REVIEW 3 major objections 4 minor 51 references

Lay-Your-Scene: Natural Scene Layout Generation with Diffusion Transformers

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A small open LLM plus an aspect-aware diffusion Transformer generates scene layouts from text

desk verdict A solid architectural contribution undermined by a training/evaluation overlap on NSR-1K that makes the headline spatial SOTA unsupported as written. read the letter →

arxiv 2505.04718 v1 pith:SBRU4ZBA submitted 2025-05-07 cs.CV cs.LG

Divyansh Srivastava , Xiang Zhang , He Wen , Chenru Wen , Zhuowen Tu This is my paper · ORCID
classification cs.CVcs.LG
keywords text-to-layoutgenerationopen-vocabularylayoutdiffusiontransformerspatialreasoningnumericalcontrollableimageNSR-1K
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that open-vocabulary scene layout generation does not need proprietary large language models. It proposes a two-stage pipeline, LayouSyn, in which a small open-source Llama model turns a text prompt into a set of object names with counts, and an aspect-ratio-aware diffusion Transformer turns those names into bounding boxes. The paper reports that this pipeline beats proprietary-LLM baselines on the NSR-1K spatial and numerical reasoning benchmark and lowers Layout-FID on a grounded COCO dataset, while using only an 18M-parameter denoiser. If the result holds, controllable image generation becomes cheaper and more transparent, and object parsing can be cleanly separated from geometric placement.

What carries the argument

The load-bearing component is the layout diffusion Transformer (LDiT) block. It concatenates bounding-box tokens and T5-encoded object-description tokens, modulates the description tokens with the same adaptive layer norm used for scalar conditions, and lets description tokens attend to the global prompt through an added cross-attention layer, so local object information is aligned with the scene-level text. The architecture also makes generation aspect-ratio aware by normalizing box coordinates to [-1,1] and feeding the ratio W/H as a scalar condition. A second mechanism is the scaled noise schedule of Eq. (3), which slows the destruction of box-coordinate information in early diffusion steps; the paper uses scale s=2.0 and classifier-free guidance 2.0 after ablating both.

What would settle it

Compare the NSR-1K prompts used in training against the benchmark's evaluation prompts; if any overlap exists, retrain on a disjoint subset and measure spatial accuracy. A large drop from the reported 92.58% would show the result is memorization rather than generalization.

Watch

Extended reading notes

Core claim

On its own terms, LayouSyn's central claim is that decomposing text-to-layout into a language step and a geometry step, with a diffusion Transformer trained directly on bounding-box coordinates, achieves better spatial and counting accuracy than layouts written entirely by a proprietary LLM. The language step is a prompted open-source Llama-3.1-8B model that outputs a JSON description set of noun phrases and counts. The geometry step is an 18M-parameter LDiT denoiser that adds noise to box coordinates, normalizes coordinates by layout size for aspect-ratio-agnostic training, conditions on the aspect ratio and timestep via adaLN, and aligns local description embeddings with the global T5 prompt embedding through cross-attention and modulation. A scaled noise schedule with s=2.0 keeps low-dimensional box information from being destroyed too early. The paper reports 92.58% spatial accuracy and 95.14% numerical accuracy on NSR-1K, L-FID 3.07 on COCO-GR, and improvements over GPT-based LayoutGPT with equal or fewer sampling steps.

Load-bearing premise

The central performance claim assumes the NSR-1K benchmark is a held-out test set, yet the paper trains on the NSR-1K spatial prompts and never documents a train/test split before reporting spatial accuracy on that same benchmark.

Editorial extensions

If this is right

  • Open-vocabulary layout generation works with a lightweight open-source LLM in the loop; the paper reports that Llama-3.1-8B description sets give lower L-FID than GPT-3.5 or GPT-4o-mini description sets.
  • The layout denoiser alone contributes most of the gain: when given the same description sets as LayoutGPT, LayouSyn cuts L-FID by up to 50.1%.
  • Coarse layouts from any LLM can be refined by DDIM inversion in about 15 steps, improving spatial accuracy over the LLM's raw output.
  • The same model can be used for automated object addition: detect existing objects, let LayouSyn complete the layout for the new object, and inpaint it with GLIGEN.
  • Because the noise schedule is built into alpha_t rather than into input normalization, the method is usable with standard stable training at any aspect ratio.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the NSR-1K prompts used for training are also present in the reported evaluation set, the spatial-accuracy numbers may partly reflect memorization; a disjoint held-out split would settle this.
  • Editorial inference: the scaled noise schedule should transfer to other low-dimensional diffusion targets where information is destroyed quickly, such as keypoints or object-detection boxes.
  • Editorial inference: the paper's separation of noun-phrase parsing from geometric placement suggests a modular recipe in which compact specialist models can replace proprietary LLM components in other controllable-generation pipelines.
  • Editorial inference: LLM initialization followed by DDIM inversion may work as a general refinement operator for coarse structured predictions beyond layouts, such as 3D boxes or scene graphs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. LayouSyn is a two-stage text-to-layout pipeline: a lightweight open-source LLM (Llama-3.1-8B) extracts object descriptions with counts, and an aspect-aware diffusion Transformer (LDiT) generates bounding boxes conditioned on the prompt, the object descriptions, and the aspect ratio. The paper evaluates on COCO-GR layout quality (L-FID), on the NSR-1K spatial and numerical reasoning benchmark, and in two applications (LLM initialization and object addition to images). The central claim is that LayouSyn achieves state-of-the-art spatial and numerical reasoning while using only lightweight open-source components, making controllable image generation cheaper and more transparent than proprietary-LLM-based approaches.

Significance. If the spatial and numerical results were obtained on properly held-out data, the contribution would be meaningful: an open-vocabulary, transparent, and cost-effective alternative to proprietary-LLM layout generators. The COCO-GR L-FID comparison, the noise-schedule scaling analysis, and the ablations that separate description-set generation from layout generation are informative and show careful engineering. However, the central spatial-reasoning claim is currently undermined by benchmark contamination, and the numerical SOTA claim is only partially supported by the reported metrics. Until the evaluation is corrected, the magnitude of the reported gains cannot be taken as evidence of open-vocabulary generalization.

major comments (3)
  1. [Sec. 4.1 / Sec. 4.4 / Table 3] The model is trained on the NSR-1K spatial dataset (738 prompts, Sec. 4.1) and then evaluated on the NSR-1K benchmark (Sec. 4.4, Table 3) with no documented train/test split or exclusion of these prompts from training. The NSR-1K benchmark was introduced as a zero-shot evaluation set for LLM-based planners, so the reported 92.58% spatial accuracy is not evidence of open-vocabulary generalization unless a held-out portion is shown. This also makes the comparison against zero-shot LayoutGPT baselines unfair, since LayouSyn has seen the spatial prompts during training. Please provide a proper split, or retrain without NSR-spatial and report the resulting spatial accuracy; this is a necessary condition for the abstract's spatial SOTA claim.
  2. [Sec. 4.4, Table 3] The abstract and Section 1 claim state-of-the-art numerical reasoning, but Table 3 shows that on the numerical GLIP accuracy metric LayouSyn (56.17) and LayouSyn-GRIT (56.20) are below LayoutGPT (GPT-4o-mini) (57.96). The numerical SOTA claim is therefore only supported for Precision/Recall/Accuracy, not for the GLIP-based metric; please qualify the claim or explain why the GLIP column should not be considered when assessing numerical reasoning.
  3. [Appendix A.1, Eq. (6)] The proof of the noise-schedule scaling is inconsistent as written. Eq. (5) defines the normalized process with a sqrt(tilde-alpha_t) coefficient, but Eq. (6) sets tilde-alpha_t equal to that coefficient itself (s sqrt(alpha_t)/sqrt((s^2-1)alpha_t+1)). The variance parameter corresponding to Eq. (3) is s^2 alpha_t/((s^2-1)alpha_t+1), not the expression in Eq. (6). Please correct the theorem or align the notation between Eq. (3) and the appendix.
minor comments (4)
  1. [Sec. 4.3] The L-FID evaluation uses the first 8,700 captions from the COCO-GR validation set; please specify whether this is a random sampled subset and report the subset construction, since an ordered prefix may introduce selection bias.
  2. [Table 3] The table note says 'shaded columns represent metrics computed directly on the generated layouts', but it is unclear which columns are shaded in monochrome printing; please make the notation explicit.
  3. [Table 7] The L-FID improvement from cross-attention and modulation is small (2.82 to 2.74); please report variance or statistical significance before claiming that these architectural components are responsible for the improvement.
  4. [Page 2, affiliations] The affiliation 'Tsingua University' appears to be a typo for 'Tsinghua University'.

Circularity Check

1 steps flagged · score 6.0 of 10

The spatial half of the NSR-1K SOTA claim is circular: the model is trained on the NSR-1K spatial dataset and then evaluated on the same NSR-1K benchmark with no documented train/test split.

  1. other [Section 4.1 (Datasets) and Section 4.4 (Spatial and Numerical Evaluation), Table 3]
    "We use the NSR-1K spatial dataset proposed in LayoutGPT [12] to train our model for understanding the spatial relationship between objects present in the scene. ... We evaluate LayouSyn on the NSR-1K spatial and numerical reasoning benchmark ... The results on the NSR-1K benchmark are reported in Tab. 3. LayouSyn achieves superior performance across multiple metrics, including 92.58% accuracy in spatial reasoning."

    Section 4.1 states that the model is trained on the NSR-1K spatial dataset (738 prompts with above/below/left/right relations). Section 4.4 then evaluates the model on the NSR-1K benchmark and reports 92.58% spatial accuracy. No train/test split, held-out subset, or exclusion of NSR-spatial prompts is documented. Because the evaluation prompts are a subset of the training data, the reported spatial accuracy is a memorization score by construction, not an open-vocabulary generalization result. The comparison to zero-shot LLM baselines is therefore not evidence of superiority on unseen spatial prompts. The numerical-reasoning column is less directly contaminated because only the spatial subset is mentioned in Section 4.1, but the abstract's SOTA claim bundles both benchmarks.

full rationale

No equation-level or derivation-level circularity was found: the noise-schedule scaling in Eq. (3) is derived from the scaled forward process in the appendix, and the diffusion model is trained on externally sourced captions and detections (COCO-GR, GRIT). The L-FID evaluation uses a validation split and is internally consistent. The one load-bearing circularity is dataset-level: the same NSR-1K spatial benchmark used for training is then presented as an evaluation benchmark with no documented split, making the spatial SOTA figures partly a measure of memorization. Because the abstract's central claim is state-of-the-art performance on 'spatial and numerical reasoning benchmarks,' this contamination directly undercuts the headline result. Consequently the score is 6: partial circularity in the central evaluation, not in the derivation itself.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a few free hyperparameters (noise schedule scale, CFG scale, sampling steps) chosen by ablation, and on domain assumptions about the reliability of GroundingDINO boxes, GLIP/GLIGEN evaluation, and the NSR-1K benchmark structure. The most consequential assumption is that NSR-1K can be used for training and then serve as a clean evaluation set; the paper provides no split documentation.

free parameters (3)
  • Noise schedule scale s = 2.0
    Selected via L-FID ablation in Sec 4.5 (Fig 5); directly controls how gradually bounding-box information is destroyed and affects all generation results.
  • Classifier-free guidance scale = 2.0
    Jointly chosen with noise schedule scale in the same ablation to minimize L-FID.
  • DDIM sampling steps = 15
    Chosen as a speed-quality tradeoff in Tab 6; L-FID keeps improving up to 100 steps, so using 15 is a deliberate tradeoff for the reported inference time.
assumptions (4)
  • ad hoc to paper NSR-1K spatial prompts are appropriate as both training data and evaluation benchmark without a documented split.
    Sec 4.1 trains on the 738-prompt NSR-1K spatial set and Sec 4.4 evaluates on NSR-1K; the identification of training and test prompts is not stated.
  • domain assumption GroundingDINO detections are accurate enough to serve as layout ground truth in COCO-GR.
    COCO-GR is built by taking Llama-extracted nouns and GroundingDINO boxes (Sec 4.1); detector false positives and false negatives become training signal and can bias layout quality.
  • domain assumption GLIGEN plus GLIP is a valid proxy for judging whether a layout satisfies spatial and numerical constraints.
    NSR-1K GLIP metrics in Sec 4.4 rely on GLIGEN to render images and GLIP to re-detect objects; the ground-truth layouts do not reach 100 percent, showing that detector noise affects the metric.
  • domain assumption Bounding-box tuples plus text descriptions fully capture scene layout semantics.
    The formulation L = {(d_i, b_i)} in Sec 3 drops occlusion, depth, and size relationships beyond the box; Sec 6 acknowledges future depth constraints.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lay-Your-Scene: Natural Scene Layout Generation with Diffusion Transformers." pith.science (2026). https://pith.science/paper/SBRU4ZBA

@misc{pith2026250504718,
  author       = {Pith},
  title        = {Pith review of: Lay-Your-Scene: Natural Scene Layout Generation with Diffusion Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SBRU4ZBA}},
  note         = {Machine review of arXiv:2505.04718}
}
read the original abstract

We present Lay-Your-Scene (shorthand LayouSyn), a novel text-to-layout generation pipeline for natural scenes. Prior scene layout generation methods are either closed-vocabulary or use proprietary large language models for open-vocabulary generation, limiting their modeling capabilities and broader applicability in controllable image generation. In this work, we propose to use lightweight open-source language models to obtain scene elements from text prompts and a novel aspect-aware diffusion Transformer architecture trained in an open-vocabulary manner for conditional layout generation. Extensive experiments demonstrate that LayouSyn outperforms existing methods and achieves state-of-the-art performance on challenging spatial and numerical reasoning benchmarks. Additionally, we present two applications of LayouSyn. First, we show that coarse initialization from large language models can be seamlessly combined with our method to achieve better results. Second, we present a pipeline for adding objects to images, demonstrating the potential of LayouSyn in image editing applications.

Figures

Figures reproduced from arXiv: 2505.04718 by the authors.

Figure 1
Figure 1. Text to natural scene layout generation with [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of inference pipeline for LayouSyn. We frame the scene layout generation task as a two-stage process. First, a lightweight language model extracts a set of relevant object descriptions from the text prompt describing the scene. Second, a trained diffusion model generates layouts conditioned on the text prompt and object descriptions. Note that Concat in the LDiT block refers to the concatenation of descript… view at source ↗
Figure 3
Figure 3. Diversity of layouts generated by LayouSyn [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparative analysis with LayoutGPT. LayouSyn can generate complex layouts with multiple objects following spa￾tial constraints in the prompt. 1 2 4 6 8 Noise Schedule Scaling 2.5 3.0 3.5 4.0 L-FID (Lower is Better) CFG=1 CFG=2 CFG=4 CFG=8 [PITH_FULL_IMAGE:figures/ful…
Figure 5
Figure 5. Figure 5: Effect of noise schedule scaling on layout fidelity (L￾FID) at different CFG scales. A noise schedule scale of 2 and a CFG scale of 2 achieves the lowest L-FID. unless otherwise stated. Noise schedule scaling We report the quantitative re￾sults for different noise sche…
Figure 6
Figure 6. Figure 6: Examples of automated object addition using Lay￾ouSyn Objects to add are highlighted in red in the prompt. Our pipeline consists of four steps: extracting relevant objects from the prompt with LLM, detecting objects present in the scene with GroundingDINO [29], layout …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 22 canonical work pages

  1. [1]

    Variational transformer networks for layout generation

    Diego Martin Arroyo, Janis Postels, and Federico Tombari. Variational transformer networks for layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 13642–13652, 2021. 2

    2021

  2. [2]

    Make it count: Text-to-image gen- eration with an accurate number of objects

    Lital Binyamin, Yoad Tewel, Hilit Segev, Eran Hirsch, Royi Rassin, and Gal Chechik. Make it count: Text-to-image gen- eration with an accurate number of objects. arXiv preprint arXiv:2406.10210, 2024. 1

    arXiv 2024

  3. [3]

    Video generation models as world sim- ulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world sim- ulators. 2024-03-03]. https://openai. com/research/video- generation-modelsas-world-simulators, 2024. 3

    work page 2024

  4. [4]

    Lay- outdm: Transformer-based diffusion model for layout gener- ation

    Shang Chai, Liansheng Zhuang, and Fengying Yan. Lay- outdm: Transformer-based diffusion model for layout gener- ation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 18349–18358,

  5. [5]

    Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. 1

    arXiv

  6. [6]

    Diffu- siondet: Diffusion model for object detection, 2023

    Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffu- siondet: Diffusion model for object detection, 2023. 4

    work page 2023

  7. [7]

    On the importance of noise scheduling for diffu- sion models, 2023

    Ting Chen. On the importance of noise scheduling for diffu- sion models, 2023. 4

    work page 2023

  8. [8]

    Ting Chen, Lala Li, Saurabh Saxena, Geoffrey Hinton, and David J. Fleet. A generalist framework for panoptic segmen- tation of images and videos, 2023. 4

    work page 2023

Show all 51 references
  1. [9]

    Diffedit: Diffusion-based seman- tic image editing with mask guidance

    Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based seman- tic image editing with mask guidance. arXiv preprint arXiv:2210.11427, 2022. 8

    2022 arXiv

  2. [10]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  3. [11]

    The llama 3 herd of models, 2024

    Abhimanyu Dubey et al. The llama 3 herd of models, 2024. 3, 6

    2024

  4. [12]

    Layoutgpt: Compositional visual plan- ning and generation with large language models

    Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Ar- jun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual plan- ning and generation with large language models. arXiv preprint arXiv:2305.15393, 2023. 2, 5, 6

    2023 arXiv

  5. [13]

    Ranni: Taming text-to-image diffu- sion for accurate instruction following

    Yutong Feng, Biao Gong, Di Chen, Yujun Shen, Yu Liu, and Jingren Zhou. Ranni: Taming text-to-image diffu- sion for accurate instruction following. arXiv preprint arXiv:2311.17002, 2023. 6

    2023 arXiv

  6. [14]

    Llm blueprint: Enabling text-to-image generation with complex and detailed prompts

    Hanan Gani, Shariq Farooq Bhat, Muzammal Naseer, Salman Khan, and Peter Wonka. Llm blueprint: Enabling text-to-image generation with complex and detailed prompts. In The Twelfth International Conference on Learning Repre- sentations, 2024. 2, 3, 6

    2024

  7. [15]

    Layout- transformer: Layout generation and completion with self- attention

    Kamal Gupta, Justin Lazarow, Alessandro Achille, Larry S Davis, Vijay Mahadevan, and Abhinav Shrivastava. Layout- transformer: Layout generation and completion with self- attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1004–1014, 20...

    2021

  8. [16]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium, 2018

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium, 2018. 5

    2018

  9. [17]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 7

    2022 arXiv

  10. [18]

    Denoising diffu- sion probabilistic models, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models, 2020. 4

    2020

  11. [19]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. 5

    2022

  12. [20]

    Layoutdm: Discrete diffusion model for controllable layout generation

    Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Layoutdm: Discrete diffusion model for controllable layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10167–10176, 2023. 1, 2, 3

    2023

  13. [21]

    Layoutformer++: Con- ditional graphic layout generation via constraint serializa- tion and decoding space restriction

    Zhaoyuz Jiang, Jiaqi Guo, Shizhao Sun, Huayu Deng, Zhongkai Wu, Vuksan Mijovic, Zijiang James Yang, Jian- Guang Lou, and Dongmei Zhang. Layoutformer++: Con- ditional graphic layout generation via constraint serializa- tion and decoding space restriction. In Proceedings of the ...

    2023

  14. [22]

    Layoutvae: Stochastic scene layout generation from a description set

    Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Si- gal, and Greg Mori. Layoutvae: Stochastic scene layout generation from a description set. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9895–9904, 2019. 1, 2

    2019

  15. [23]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. 5

    2017

  16. [24]

    Blt: Bidirectional layout transformer for controllable layout generation

    Xiang Kong, Lu Jiang, Huiwen Chang, Han Zhang, Yuan Hao, Haifeng Gong, and Irfan Essa. Blt: Bidirectional layout transformer for controllable layout generation. In European Conference on Computer Vision , pages 474–490. Springer,

  17. [25]

    Layoutgan: Generating graphic layouts with wireframe discriminators

    Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. Layoutgan: Generating graphic layouts with wireframe discriminators. arXiv preprint arXiv:1901.06767,

    1901 arXiv

  18. [26]

    Gligen: Open-set grounded text-to-image generation, 2023

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jian- wei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation, 2023. 1, 2, 6, 8 9

    2023

  19. [27]

    Llm- grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models

    Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Llm- grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305.13655, 2023. 2, 6

    2023 arXiv

  20. [28]

    Lawrence Zitnick, and Piotr Doll ´ar

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Doll ´ar. Microsoft coco: Common objects in context, 2015. 2, 5

    2015

  21. [29]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 8

    2023 arXiv

  22. [30]

    Repaint: Inpainting using denoising diffusion probabilistic models, 2022

    Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models, 2022. 8

    2022

  23. [31]

    Dit-3d: Exploring plain diffusion transformers for 3d shape generation

    Shentong Mo, Enze Xie, Ruihang Chu, Lanqing Hong, Matthias Niessner, and Zhenguo Li. Dit-3d: Exploring plain diffusion transformers for 3d shape generation. Advances in neural information processing systems, 36:67960–67971,

  24. [32]

    Hall, Daniel Cer, and Yinfei Yang

    Jianmo Ni, Gustavo Hern ´andez ´Abrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfei Yang. Sentence- t5: Scalable sentence encoders from pre-trained text-to-text models, 2021. 4

    2021

  25. [33]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021. 1

    2021 arXiv

  26. [34]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162–8171. PMLR,

  27. [35]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Ad- vances in neural information processing systems, 35:...

    2022

  28. [36]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  29. [37]

    Kosmos-2: Grounding multimodal large language models to the world, 2023

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world, 2023. 5

    2023

  30. [38]

    Film: Visual reasoning with a general conditioning layer, 2017

    Ethan Perez, Florian Strub, Harm de Vries, Vincent Du- moulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer, 2017. 4

    2017

  31. [39]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 5

    2021

  32. [40]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learn- ing Research, 21(140):1–67, 2020. 4

    2020

  33. [41]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International confer- ence on machine learning, pages 8821–8831. Pmlr, 2021. 1

    2021

  34. [42]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1

    2022

  35. [43]

    Instancediffusion: Instance- level control for image generation

    Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Ro- hit Girdhar, and Ishan Misra. Instancediffusion: Instance- level control for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6232–6242, 2024. 1

    2024

  36. [44]

    Instancediffusion: Instance-level control for image generation, 2024

    Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancediffusion: Instance-level control for image generation, 2024. 2

    2024

  37. [45]

    Dolfin: Diffusion layout transformers without autoencoder

    Yilin Wang, Zeyuan Chen, Liangjun Zhong, Zheng Ding, Zhizhou Sha, and Zhuowen Tu. Dolfin: Diffusion layout transformers without autoencoder. In European Conference on Computer Vision, 2024. 1, 2, 3, 5

    2024

  38. [46]

    Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion

    Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wen- tian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7452–7461, 2023. 1, 2

    2023

  39. [47]

    Bayesian diffusion models for 3d shape reconstruction

    Haiyang Xu, Yu Lei, Zeyuan Chen, Xiang Zhang, Yue Zhao, Yilin Wang, and Zhuowen Tu. Bayesian diffusion models for 3d shape reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10628–10638, 2024. 3

    2024

  40. [48]

    Raphael: Text-to-image generation via large mixture of diffusion paths

    Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuo- fan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. Advances in Neural Information Processing Systems, 36, 2024. 1

    2024

  41. [49]

    Reco: Region-controlled text-to-image genera- tion

    Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region-controlled text-to-image genera- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 14246–14255,

  42. [50]

    Layoutdiffusion: Improving graphic lay- out generation by discrete diffusion probabilistic models

    Junyi Zhang, Jiaqi Guo, Shizhao Sun, Jian-Guang Lou, and Dongmei Zhang. Layoutdiffusion: Improving graphic lay- out generation by discrete diffusion probabilistic models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7226–7236, 2023. 1, 2

    2023

  43. [51]

    phrase1": count1,

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 1 10 A. Appendix A.1. Scaling Factor Figure A.1. Effect of scaling fact...

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.