Pith. sign in

REVIEW 4 major objections 4 minor 78 references

Grounding Text-to-Image Diffusion Models for Controlled High-Quality Image Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ObjectDiffusion grounds text-to-image diffusion on bounding boxes, reporting AP50 46.6, AR 44.5, and FID 19.8, beating open-source-trained layout-to-image models on all three metrics.

desk verdict The SOTA claim rests on an unablated inference-time swap from SD to GLIGEN plus uncontrolled baselines; the architecture is a plausible incremental combination, but the numbers as reported don't support the headline. read the letter →

arxiv 2501.09194 v2 pith:WT4QMFDB submitted 2025-01-15 cs.CV cs.AI

Ahmad Süleyman , Göksel Biricik This is my paper · ORCID
classification cs.CVcs.AI
keywords text-to-imagegenerationlayout-to-imageboundingboxgroundingdiffusionmodelsControlNetGLIGENcontrollableimagesynthesisopen-set
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ObjectDiffusion is a conditional image generation model that lets a pretrained text-to-image diffusion model place objects described in free-form text into specified bounding boxes. The authors claim that by combining the frozen-backbone architecture of ControlNet with the gated-attention grounding mechanism of GLIGEN, they get both high placement precision and high image quality. Fine-tuned on COCO2017 and evaluated on its validation split, the model reports AP50 of 46.6, AR of 44.5, and FID of 19.8, beating open-source-trained baselines on all three metrics. The point of the work is to show that an existing frozen diffusion base can be extended with grounding capability without retraining the base and without losing its generation knowledge.

What carries the argument

GroundNet: a trainable parallel network formed from the encoder and middle blocks of the diffusion backbone, modified by inserting a gated self-attention layer (with tanh gating) into each attention block, and connected to the frozen base via 1x1 zero-initialized convolution layers. It processes a control layout g = [(e1,b1),...,(eN,bN)] where each entity ei is embedded by a frozen CLIP text encoder and each box bi by a Fourier positional embedding; an MLP fuses the two into a 768-dimensional token that the gated self-attention layers inject into the visual features. The zero-convolutions and the tanh gates protect the pretrained weights from the newly introduced grounding signal during early training.

What would settle it

Run the exact same training procedure but keep the frozen Stable Diffusion base at inference (no swap to GLIGEN), then recompute AP50, AR, and FID on the same COCO2017 validation annotations; if the numbers collapse toward or below the GLIGEN fine-tuned baseline, the reported gains depend on the swapped base rather than on the GroundNet training.

Watch

Extended reading notes

Core claim

The central claim is that a trainable GroundNet—a cloned, modified copy of the diffusion model's encoder and middle blocks with GLIGEN-style gated self-attention layers folded in—can be attached to a frozen Stable Diffusion model through zero-initialized convolutions and fine-tuned on detection annotations, yielding a layout-to-image model whose grounding precision and image quality exceed the current state of the art among models trained on open-source data. The grounding input is a set of tokens pairing a CLIP-encoded open-ended entity description with a Fourier-embedded bounding box, fused by an MLP and injected at every encoder layer and the middle block. At inference the frozen base is swapped for a pretrained GLIGEN, which the authors say produces more controllable images; all reported numbers come from that hybrid.

Load-bearing premise

The reported metrics all come from a hybrid where the fine-tuned GroundNet is attached to a pretrained GLIGEN base at inference instead of the Stable Diffusion base it was trained with; if the zero-convolution injections trained against Stable Diffusion do not transfer to GLIGEN's own grounding pathway, the numbers would not reflect the proposed training method.

Editorial extensions

If this is right

  • Layout-to-image generation can be added to a frozen diffusion backbone by fine-tuning only a small parallel network, so the base model's broad generation knowledge is preserved.
  • The reported AP50 46.6 versus 42.3 for GLIGEN (fine-tuned) and AR 44.5 versus 30.7 suggest that multi-scale injection of grounding tokens reduces both object misplacement and object omission.
  • An FID of 19.8 versus 21.58 for GLIGEN (fine-tuned) indicates that image quality does not have to be sacrificed for control.
  • Open-set qualitative results imply that the grounding works beyond the 80 COCO categories, as long as the CLIP text encoder recognizes the entity phrase.
  • Initializing GroundNet from a pretrained GLIGEN checkpoint and fine-tuning only 460.8M of 1.32B parameters substantially reduces training cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A controlled comparison that keeps the frozen base fixed (Stable Diffusion throughout training and inference) would separate how much of the reported precision comes from the GroundNet fine-tuning and how much from swapping in GLIGEN at inference.
  • The large per-class AP spread (cat AP50 96.7 versus sports ball 10.9) suggests that reweighting the grounding loss or oversampling low-AP classes could lift overall precision without changing the architecture.
  • Because the grounding tokens are produced by a frozen CLIP encoder and injected through gated self-attention, the same recipe could likely be attached to other frozen text-to-image backbones, making box grounding a modular add-on rather than a per-model retraining task.
  • Testing the model with deliberately corrupted or partially dropped grounding tokens would reveal whether the 10% random-drop training makes it robust to missing boxes, which is directly relevant to real-world use.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes ObjectDiffusion, a layout-to-image model that combines ControlNet-style zero-convolution injection with GLIGEN-style gated self-attention grounding. The authors freeze a Stable Diffusion v1.4 base and fine-tune a parallel GroundNet on COCO2017; at inference, however, they replace the frozen SD base with a pretrained GLIGEN network. They report AP50=46.6, AR=44.5, and FID=19.8 and claim state-of-the-art performance among models trained on open-source datasets, supported by qualitative closed-set and open-set examples.

Significance. If the quantitative claims were obtained by the proposed training method, the paper would offer a lightweight approach for adding bounding-box grounding to a frozen text-to-image model. The authors are transparent about their open-source evaluation and include a limitations section. However, the headline results are generated by a hybrid system that is not the model defined by the training objective, and the baseline numbers are not measured under the same protocol; the central empirical claim is therefore currently unsubstantiated.

major comments (4)
  1. [§4.1 (Inference), §3.3 (Eq. 8), Figure 4] The reported metrics are produced by replacing the Stable Diffusion base used during training with a pretrained GLIGEN network, while GroundNet's zero-convolution outputs were trained as additive corrections to SD features. Equation (8) defines the model as f_{θ,θ'} = f_θ(SD) + Z(f_θ'), but at inference f_θ is a GLIGEN network whose internal gated self-attention layers alter the feature activations at the injection points. The paper offers no ablation comparing (i) SD + GroundNet, (ii) GLIGEN + GroundNet, (iii) GLIGEN alone, and (iv) SD alone under the same evaluation protocol. Without such an ablation, the reported AP50=46.6, AR=44.5, and FID=19.8 cannot be attributed to the proposed fine-tuning; they may stem entirely from the pretrained GLIGEN base.
  2. [§4.1 (Evaluation Benchmarks), Tables 1 and 4] The state-of-the-art comparison is not controlled. Tables 1, 3, and 4 state that baseline values are taken from [28], while the authors compute their own numbers with a pretrained YOLOv8m detector and the pytorch-fid implementation [56]. If GLIGEN's published numbers used a different detector or a different FID implementation and image preprocessing, part or all of the reported improvement may be an artifact of the evaluation pipeline. The preprocessing also differs from GLIGEN's: the authors resize with bicubic interpolation and do not crop, whereas GLIGEN center-crops (§4.1, Figure 3). The authors should rerun GLIGEN and GLIGEN (fine-tuned) under the exact same preprocessing, sampling, detection, and FID protocol before claiming superiority on AP50, AR, and FID.
  3. [§4.2 (Table 3)] The AR upper bound is reported as 49.7, the same value as the AP upper bound in Table 1. Average recall is a different evaluation statistic, and its upper bound should be derived from the detector's recall on the ground-truth boxes, not from the AP value. The statement that ObjectDiffusion's AR of 44.5 is 'only five points behind the upper bound' is therefore unsupported unless the upper bound is explicitly defined and recomputed.
  4. [§3.3, Figure 2, §4.1 (Initialization)] The description of GroundNet's construction and initialization is internally inconsistent. Section 3.3 says the encoder and middle blocks are cloned from the frozen Stable Diffusion model and then augmented with gated self-attention layers, while Figure 2 and Section 4.1 state that GroundNet consists of, and is initialized from, the encoder and middle blocks of GLIGEN. Because GroundNet is the only trainable component of the proposed method, this ambiguity prevents readers from knowing which weights are actually fine-tuned and is a barrier to reproducibility.
minor comments (4)
  1. [§4.1 (Evaluation Benchmarks)] The detector is referred to as YOLOv8m with citation [49], but [49] is the original YOLO paper; please cite the Ultralytics YOLOv8 implementation actually used.
  2. [References] Reference [13] contains a malformed URL ('https://https://huggingface.co/gligen/diffusers-generation-text-box'); it should be corrected.
  3. [§4.2 (Quantitative Results)] The sentence 'Our model increased the zero-shot AP score of GLIGEN (zero-shot) from 19.1 to 27.4' is misleading because ObjectDiffusion is fine-tuned on COCO; this is a comparison against a zero-shot baseline, not an improvement in zero-shot capability.
  4. [§4.1 (Dataset)] AP/AR are computed on 5k generated images while FID uses 20k generated images; please clarify whether the 5k set is a subset of the 20k set and report variance over multiple random seeds or sample sets.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training objective and evaluations are external, and the disclosed inference-time base-model swap is a correctness/attribution concern, not a circular derivation.

full rationale

Walking the claimed derivation chain, no step reduces to its own input by construction. The model is trained with the standard diffusion L2 objective (Eq. 9), with GroundNet parameters optimized on COCO2017 training annotations and evaluated on held-out COCO2017 validation annotations via a pretrained YOLOv8m detector and FID against real images; these are external benchmarks rather than quantities defined by the model or fitted from the evaluation set. The paper's most unusual design choice is the disclosed inference-time substitution of the frozen Stable Diffusion base used in training with a pretrained GLIGEN network, justified only by 'we have empirically verified that GLIGEN yields more controllable images.' This is an uncontrolled architectural change and a legitimate threat to attribution of the reported gains, but it is a methodological/correctness gap, not circularity: the reported AP50/AR/FID are not equal to any training loss, fitted parameter, or self-citation by construction. Similarly, taking some baseline numbers from [28] while recomputing the proposed model's numbers may raise comparability concerns, but uneven benchmarking is not a self-referential derivation. No load-bearing self-citation chain or imported uniqueness theorem exists; the cited prior work (ControlNet, GLIGEN) supplies components and checkpoints openly rather than an unverified premise that alone forces the conclusion. I therefore find no significant circularity and assign a score of 0.

Assumptions & free parameters 12 free parameters · 6 assumptions · 0 invented entities

The central claim rests on several domain assumptions about pretrained models and dataset quality, plus two ad hoc assumptions introduced specifically for this paper: that baseline numbers from [28] are directly comparable, and that a test-time swap of the frozen base from SD to GLIGEN preserves or improves the trained GroundNet's behavior. The hand-chosen hyperparameters listed above are typical for ML fine-tuning but are not derived from first principles.

free parameters (12)
  • learning_rate = 5e-5
    Chosen by hand for Adam optimizer; affects convergence and final performance.
  • training_iterations = 100k
    Chosen by hand; no convergence analysis or long-run comparison.
  • batch_size (with gradient accumulation) = 16 (4 x 4)
    Chosen to fit GPU memory; affects optimization.
  • warmup_iterations = 4k
    Chosen by hand as part of the learning rate schedule.
  • max_entities_per_image = 30
    Retain only the 30 largest boxes; a hyperparameter that discards annotations.
  • bbox_area_threshold = 1% of image area
    Boxes smaller than this are discarded; affects training distribution and evaluation potential.
  • caption_drop_probability = 0.1
    Randomly drop caption 10% of the time to encourage robustness.
  • condition_drop_probability = 0.1
    Randomly drop grounding condition 10% of the time.
  • guidance_scale = 7.5
    Used at inference; affects prompt adherence and quality.
  • sampler_steps = 50 (PLMS)
    Inference-time hyperparameter for sampling.
  • fourier_frequencies_M = 8
    Gives a 64-dimensional box embedding; fixed, not learned.
  • inference_base_swap = SD v1.4 to GLIGEN diffusers checkpoint
    Post-hoc choice not trained for; all reported results use this swap.
assumptions (6)
  • domain assumption Pretrained Stable Diffusion v1.4 provides a sufficiently strong generative prior.
    The frozen SD base is assumed to contribute high-quality generation knowledge during training and inference.
  • domain assumption Pretrained GLIGEN checkpoint provides a grounding prior that transfers to COCO.
    GroundNet is initialized from GLIGEN weights; the paper assumes this initialization plus COCO fine-tuning yields better grounding than GLIGEN alone.
  • domain assumption COCO2017 annotations are accurate and a valid training and evaluation signal.
    The entire method is fine-tuned and evaluated on COCO boxes and labels; no noise or annotation issues are considered.
  • domain assumption YOLOv8m detection scores are a valid proxy for object placement accuracy.
    AP and AR are computed using YOLOv8m predictions on generated images, implicitly assuming detector errors are negligible or uniform.
  • ad hoc to paper Baseline numbers from [28] are comparable despite different evaluation pipelines.
    The paper uses YOLOv8m and pytorch-fid for its own numbers but takes baseline AP and FID values from the GLIGEN paper, without re-running baselines.
  • ad hoc to paper GroundNet trained with SD base remains compatible with GLIGEN base at inference.
    The inference-time swap is asserted based on empirical verification, but no ablation or analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Grounding Text-to-Image Diffusion Models for Controlled High-Quality Image Generation." pith.science (2026). https://pith.science/paper/WT4QMFDB

@misc{pith2026250109194,
  author       = {Pith},
  title        = {Pith review of: Grounding Text-to-Image Diffusion Models for Controlled High-Quality Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WT4QMFDB}},
  note         = {Machine review of arXiv:2501.09194}
}
abstract

Text-to-image (T2I) generative diffusion models have demonstrated outstanding performance in synthesizing diverse, high-quality visuals from text captions. Several layout-to-image models have been developed to control the generation process by utilizing a wide range of layouts, such as segmentation maps, edges, and human keypoints. In this work, we propose ObjectDiffusion, a model that conditions T2I diffusion models on semantic and spatial grounding information, enabling the precise rendering and placement of desired objects in specific locations defined by bounding boxes. To achieve this, we make substantial modifications to the network architecture introduced in ControlNet to integrate it with the grounding method proposed in GLIGEN. We fine-tune ObjectDiffusion on the COCO2017 training dataset and evaluate it on the COCO2017 validation dataset. Our model improves the precision and quality of controllable image generation, achieving an AP$_{\text{50}}$ of 46.6, an AR of 44.5, and an FID of 19.8, outperforming the current SOTA model trained on open-source datasets across all three metrics. ObjectDiffusion demonstrates a distinctive capability in synthesizing diverse, high-quality, high-fidelity images that seamlessly conform to the semantic and spatial control layout. Evaluated in qualitative and quantitative tests, ObjectDiffusion exhibits remarkable grounding capabilities in closed-set and open-set vocabulary settings across a wide variety of contexts. The qualitative assessment verifies the ability of ObjectDiffusion to generate multiple detailed objects in varying sizes, forms, and locations.

Figures

Figures reproduced from arXiv: 2501.09194 by the authors.

Figure 1
Figure 1. The grounding tokens are formed by fusing the CLIP [46] encoded object entities with their Fourier [34] embedded bounding boxes. The concatenation vector is processed through an MLP [43], which produces a standard-sized output vector of 768. can prolong sampling time and decrease perceptual qual￾ity. As a result, they are typically used in conjunction with regular training-based conditional models. 3. Method We intr… view at source ↗
Figure 2
Figure 2. ObjectDiffusion architecture is divided into two parallel networks. For the first network, we utilize Stable Diffusion 1.4v [50] as a pretrained text-to-image model, depicted in blue. The second network is a trainable GroundNet, which consists of the encoder and middle blocks from GLIGEN [28], represented in red. GroundNet injects the encoded conditional layout g, which consists of the positional and semantic tokens… view at source ↗
Figure 3
Figure 3. This figure highlights the difference between the re￾sizing via Bicubic [22] interpolation that we apply and the center cropping implemented in GLIGEN. The original image is on the left, the GLIGEN [28] preprocessed image is in the middle, and our preprocessed image is on the right. COCO2017 training split, which contains 118k annotated images, for the purpose of fine-tuning, and the COCO2017 validation split, consi… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The inference schema consists of two networks. The first network, displayed in red, is our GroundNet fine-tuned on COCO2017 [30] object detection annotations. The second net￾work, displayed in blue, is a pretrained GLIGEN [28]. We re￾place the pretrained Stable Diffusi…
Figure 5
Figure 5. Figure 5: This figure showcases limitations of our model in generating faces and hands of people, as shown in image (a). It also highlights the limitations of rendering legible, correct text, as seen in image (b). ObjectDiffusion may synthesize images that precisely adhere to th…
Figure 6
Figure 6. Figure 6: This figure represents qualitative examples of closed-set configuration. Our model demonstrates precise generation capabilities for complex indoor visuals, such as living rooms, kitchens, and bathrooms. Images are generated from COCO2017 [30] validation set annotations…
Figure 7
Figure 7. Figure 7: This figure presents qualitative examples of closed-set configuration. Our model is capable of accurately rendering various classes of vehicles, such as cars, trains, motorcycles, and boats. Images are generated using grounding entities and image captions from the COCO…
Figure 8
Figure 8. Figure 8: This figure presents qualitative examples of closed-set configuration. The portraits generated by ObjectDiffusion exhibit realistic facial expressions. Images are generated using grounding entities and image captions from the COCO2017 [30] validation set annotations. 1…
Figure 9
Figure 9. Figure 9: This figure represents qualitative examples of closed-set configuration. Our model demonstrates proficiency in generating high￾resolution visuals of animals in their native habitat, achieving a very high AP precision score. Images are generated using grounding entities…
Figure 10
Figure 10. Figure 10: This figure represents qualitative examples of closed-set configuration. It shows both static and dynamic scenes of different object categories. ObjectDiffusion can synthesize images of dishes as well as visuals of sports activities, such as surfing and skiing. Images…
Figure 11
Figure 11. Figure 11: This figure represents realistic and artistic qualitative examples of images generated in an open-set configuration with manual annotations. The images demonstrate the high quality and precision of ObjectDiffusion. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 43 canonical work pages

  1. [28]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jian- wei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511–22521, 2023. 2, 3, 4, 5, 6, 7, 8, 9 17

    work page 2023

  2. [56]

    pytorch-fid: FID Score for PyTorch

    Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. https://github.com/mseitzer/pytorch-fid ,

  3. [1]

    https://www.midjourney.com , 2023

    Midjourney. https://www.midjourney.com , 2023. (accessed 15 July 2024). 1

    work page 2023

  4. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  5. [3]

    Universal guidance for diffusion models

    Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geip- ing, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 843–852,

  6. [4]

    A computational approach to edge detection

    John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelli- gence, (6):679–698, 1986. 1, 2

    work page 1986

  7. [5]

    Realtime multi-person 2d pose estimation using part affinity fields

    Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291–7299, 2017. 1, 2

    work page 2017

  8. [6]

    Training-free layout control with cross-attention guidance

    Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5343–5353, 2024. 3

    2024

Show all 78 references
  1. [7]

    https://huggingface.co/CompVis/ stable-diffusion-v-1-4-original , 2023

    CompVis. https://huggingface.co/CompVis/ stable-diffusion-v-1-4-original , 2023. Hug- ging Face, (accessed 27 January 2024). 6

    2023

  2. [8]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural informa- tion processing systems, 34:8780–8794, 2021. 2

    2021

  3. [9]

    Activation functions in deep learning: A com- prehensive survey and benchmark

    Shiv Ram Dubey, Satish Kumar Singh, and Bidyut Baran Chaudhuri. Activation functions in deep learning: A com- prehensive survey and benchmark. Neurocomputing, 503: 92–108, 2022. 4

    2022

  4. [10]

    Sigmoid- weighted linear units for neural network function approxima- tion in reinforcement learning

    Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid- weighted linear units for neural network function approxima- tion in reinforcement learning. Neural networks, 107:3–11,

  5. [11]

    Frido: Fea- ture pyramid diffusion for complex scene image synthesis

    Wan-Cyuan Fan, Yen-Chun Chen, DongDong Chen, Yu Cheng, Lu Yuan, and Yu-Chiang Frank Wang. Frido: Fea- ture pyramid diffusion for complex scene image synthesis. In Proceedings of the AAAI conference on artificial intelli- gence, pages 579–587, 2023. 3

    2023

  6. [12]

    Attrlost- gan: Attribute controlled image synthesis from reconfig- urable layout and style

    Stanislav Frolov, Avneesh Sharma, J ¨orn Hees, Tushar Karayil, Federico Raue, and Andreas Dengel. Attrlost- gan: Attribute controlled image synthesis from reconfig- urable layout and style. In DAGM German Conference on Pattern Recognition, pages 361–375. Springer, 2021. 3

    2021

  7. [13]

    https : / / https : / / huggingface

    gligen. https : / / https : / / huggingface . co/gligen/diffusers-generation-text-box ,

  8. [14]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014. 1, 2

    2014

  9. [15]

    Towards light-weight and real-time line segment detection

    Geonmo Gu, Byungsoo Ko, SeoungHyun Go, Sung-Hyun Lee, Jingeun Lee, and Minchul Shin. Towards light-weight and real-time line segment detection. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 726–734,

  10. [16]

    Hypernetworks

    David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016. 3

    2016 arXiv

  11. [17]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 2, 7, 9

    2017

  12. [18]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 6

    2020

  13. [19]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR, 2019. 2

    2019

  14. [20]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 3

    2021 arXiv

  15. [21]

    High- resolution complex scene synthesis with transformers

    Manuel Jahn, Robin Rombach, and Bj ¨orn Ommer. High- resolution complex scene synthesis with transformers. arXiv preprint arXiv:2105.06458, 2021. 2, 9

    2021 arXiv

  16. [22]

    Cubic convolution interpolation for digital im- age processing

    Robert Keys. Cubic convolution interpolation for digital im- age processing. IEEE transactions on acoustics, speech, and signal processing, 29(6):1153–1160, 1981. 6

    1981

  17. [23]

    Adam: A method for stochastic opti- mization

    Diederik P Kingma. Adam: A method for stochastic opti- mization. arXiv preprint arXiv:1412.6980, 2014. 6

    2014 arXiv

  18. [24]

    Gradient accumulation in pytorch

    Nikita Kozodoi. Gradient accumulation in pytorch. https: / / kozodoi . me / blog / 20210219 / gradient - accumulation, 2021. (accessed 7 February 2024). 7

    2021

  19. [25]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of compute...

    2017

  20. [26]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page...

    2022

  21. [27]

    Bachgan: High-resolution im- age synthesis from salient object layout

    Yandong Li, Yu Cheng, Zhe Gan, Licheng Yu, Liqiang Wang, and Jingjing Liu. Bachgan: High-resolution im- age synthesis from salient object layout. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8365–8374, 2020. 3

    2020

  22. [29]

    Image synthesis from layout with locality- aware mask adaption

    Zejian Li, Jingyu Wu, Immanuel Koh, Yongchuan Tang, and Lingyun Sun. Image synthesis from layout with locality- aware mask adaption. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 13819– 13828, 2021. 2, 3, 7, 8, 9

    2021

  23. [30]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  24. [31]

    Pseudo numerical methods for diffusion models on manifolds.arXiv preprint arXiv:2202.09778, 2022

    Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds.arXiv preprint arXiv:2202.09778, 2022. 7

    2022 arXiv

  25. [32]

    Design guidelines for prompt engineering text-to-image generative models

    Vivian Liu and Lydia B Chilton. Design guidelines for prompt engineering text-to-image generative models. InPro- ceedings of the 2022 CHI conference on human factors in computing systems, pages 1–23, 2022. 1

    2022

  26. [33]

    Mixed precision training

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017. 7

    arXiv

  27. [34]

    Representing scenes as neu- ral radiance fields for view synthesis., 2021, 65

    B Mildenhall, PP Srinivasan, M Tancik, JT Barron, R Ra- mamoorthi, and R Ng Nerf. Representing scenes as neu- ral radiance fields for view synthesis., 2021, 65. DOI: https://doi. org/10.1145/3503250, pages 99–106. 3, 4

    2021 doi

  28. [35]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4296–4304, 2024. 2

    2024

  29. [36]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021. 1

    2021 arXiv

  30. [37]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162–8171. PMLR,

  31. [38]

    Dall-e-3

    OpenAI. Dall-e-3. https://openai.com/index/ dall-e-3/, 2023. (accessed 4 May 2024). 1

    2023

  32. [39]

    Im2text: Describing images using 1 million captioned pho- tographs

    Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned pho- tographs. Advances in neural information processing sys- tems, 24, 2011. 2, 6, 7

    2011

  33. [40]

    A survey on performance metrics for object-detection algo- rithms

    Rafael Padilla, Sergio L Netto, and Eduardo AB Da Silva. A survey on performance metrics for object-detection algo- rithms. In 2020 international conference on systems, sig- nals and image processing (IWSSIP), pages 237–242. IEEE,

    2020

  34. [41]

    Best prompts for text-to-image models and how to find them

    Nikita Pavlichenko and Dmitry Ustalov. Best prompts for text-to-image models and how to find them. In Proceed- ings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 2067–2071, 2023. 1

    2023

  35. [42]

    Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. In Pro- ceedings of the IEEE international conference on computer ...

    2015

  36. [43]

    Multilayer percep- tron and neural networks

    Marius-Constantin Popescu, Valentina E Balas, Liliana Perescu-Popescu, and Nikos Mastorakis. Multilayer percep- tron and neural networks. WSEAS Transactions on Circuits and Systems, 8(7):579–588, 2009. 3, 4

    2009

  37. [44]

    Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation

    David MW Powers. Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation. arXiv preprint arXiv:2010.16061, 2020. 2

    2010 arXiv

  38. [45]

    Unicontrol: A unified diffusion model for controllable visual generation in the wild

    Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: A unified diffusion model for controllable visual generation in the wild. arXiv preprint arXiv:2305.11147, 2023. 2, 3

    2023 arXiv

  39. [46]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  40. [47]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International confer- ence on machine learning, pages 8821–8831. Pmlr, 2021. 1

    2021

  41. [48]

    Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer

    Ren ´e Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 44(3):1623–1637, 2020. 1, 2

    2020

  42. [49]

    You only look once: Unified, real-time object de- tection

    Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object de- tection. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 779–788, 2016. 1, 7, 8, 9

    2016

  43. [50]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2023. 1, 2, 3, 4, 5, 6, 7, 9

    2023

  44. [51]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...

    2015

  45. [52]

    Limitations of face image generation

    Harrison Rosenberg, Shimaa Ahmed, Guruprasad Ramesh, Kassem Fawaz, and Ramya Korlakai Vinayak. Limitations of face image generation. In Proceedings of the AAAI Con- ference on Artificial Intelligence, pages 14838–14846, 2024. 10

    2024

  46. [53]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven 18 generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven 18 generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2...

    2023

  47. [54]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  48. [55]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  49. [57]

    Objects365: A large-scale, high-quality dataset for object detection

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 8430–8439, 2019. 2, 6, 7

    2019

  50. [58]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2018

  51. [59]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International confer- ence on machine learning, pages 2256–2265. PMLR, 2015. 2, 6

    2015

  52. [60]

    Generative modeling by esti- mating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by esti- mating gradients of the data distribution. Advances in neural information processing systems, 32, 2019. 2

    2019

  53. [61]

    Score-based generative modeling through stochastic differential equa- tions

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions. arXiv preprint arXiv:2011.13456, 2020. 2

    2011 arXiv

  54. [62]

    Image synthesis from reconfig- urable layout and style

    Wei Sun and Tianfu Wu. Image synthesis from reconfig- urable layout and style. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10531– 10540, 2019. 2, 3

    2019

  55. [63]

    Learning layout and style recon- figurable gans for controllable image synthesis

    Wei Sun and Tianfu Wu. Learning layout and style recon- figurable gans for controllable image synthesis. IEEE trans- actions on pattern analysis and machine intelligence, 44(9): 5070–5087, 2021. 7, 8, 9

    2021

  56. [64]

    Object-centric image genera- tion from layouts

    Tristan Sylvain, Pengchuan Zhang, Yoshua Bengio, R Devon Hjelm, and Shikhar Sharma. Object-centric image genera- tion from layouts. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2647–2655, 2021. 2, 9

    2021

  57. [65]

    Diode: A dense indoor and outdoor depth dataset

    Igor Vasiljevic, Nick Kolkin, Shanyi Zhang, Ruotian Luo, Haochen Wang, Falcon Z Dai, Andrea F Daniele, Moham- madreza Mostajabi, Steven Basart, Matthew R Walter, et al. Diode: A dense indoor and outdoor depth dataset. arXiv preprint arXiv:1908.00463, 2019. 1, 2

    1908 arXiv

  58. [66]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 3

    2017

  59. [67]

    Continual learning with hy- pernetworks

    Johannes V on Oswald, Christian Henning, Benjamin F Grewe, and Jo ˜ao Sacramento. Continual learning with hy- pernetworks. arXiv preprint arXiv:1906.00695, 2019. 3

    1906 arXiv

  60. [68]

    Pretraining is all you need for image-to-image translation

    Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image translation. arXiv preprint arXiv:2205.12952, 2022. 2

    2022 arXiv

  61. [69]

    Instancediffusion: Instance- level control for image generation

    Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Ro- hit Girdhar, and Ishan Misra. Instancediffusion: Instance- level control for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6232–6242, 2024. 2, 3

    2024

  62. [70]

    Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion

    Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wen- tian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7452–7461, 2023. 3

    2023

  63. [71]

    Holistically-nested edge de- tection

    Saining Xie and Zhuowen Tu. Holistically-nested edge de- tection. In Proceedings of the IEEE international conference on computer vision, pages 1395–1403, 2015. 1

    2015

  64. [72]

    Modeling image composition for complex scene generation

    Zuopeng Yang, Daqing Liu, Chaoyue Wang, Jie Yang, and Dacheng Tao. Modeling image composition for complex scene generation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 7764–7773, 2022. 2, 3, 7, 8, 9

    2022

  65. [73]

    Reco: Region-controlled text-to-image genera- tion

    Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region-controlled text-to-image genera- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 14246–14255,

  66. [74]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 2, 3, 4, 5

    2023

  67. [75]

    Image generation from layout

    Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal. Image generation from layout. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8584–8593, 2019. 2, 3

    2019

  68. [76]

    Uni-controlnet: All-in-one control to text-to-image diffusion models

    Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Sys- tems, 36, 2024. 2, 3, 5

    2024

  69. [77]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 633–641,

  70. [2023]

    Hugging Face, (accessed 2 February 2024). 6

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.