Pith. sign in

REVIEW 3 major objections 5 minor 46 references

Bridging the Gap: Aligning Text-to-Image Diffusion Models with Specific Feedback

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning diffusion models on detection-based feedback sharpens text-image alignment in categories and quantities

desk verdict A plausible detection-based reward idea, but the claimed differentiability path is absent and the main metric is the same as the training reward. read the letter →

arxiv 2412.00122 v1 pith:VQXFZAJX submitted 2024-11-28 cs.CV

Xuexiang Niu , Jinping Tang , Lei Wang , Ge Zhu This is my paper · ORCID
classification cs.CV
keywords text-to-imagegenerationdiffusionmodelsfeedbacklearningrewardfine-tuningcompositionalobjectdetectioncategory-quantityalignmentCQScore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that text-to-image diffusion models can be aligned with prompts more effectively by fine-tuning on feedback that is specific to object categories and quantities, rather than on holistic similarity or human preference scores. It proposes a detection-based reward, the CQ Score, computed by comparing object detections from the generated image with quantity–category pairs parsed from the prompt, then fine-tunes Stable Diffusion v1.5 by backpropagating this reward through the last denoising step. On a new 1,700-pair compositional dataset, the method improves CLIP, BLIP, and CQ Scores over the base model by an average of 11.2%, and also improves FID. The paper further claims the reward is differentiable, enabling direct SGD optimization, and that the fine-tuned model can serve as a metric for evaluating other text-to-image generators.

What carries the argument

The CQ Score is the harmonic mean of two terms: $\text{Acc}$, the mean detection confidence per class for classes that appear in the prompt, and $\text{Aqc}$, the mean over classes of $\min(\text{count}_{\text{detected}}, \text{count}_{\text{prompt}}) / \max(\text{count}_{\text{detected}}, \text{count}_{\text{prompt}})$. It is computed by running the YOLOS object detector on the generated image, filtering detections by confidence threshold 0.8 and non-maximum suppression with IoU 0.5, and pairing each detected class with a count; the prompt is parsed with the spaCy tokenizer to extract quantity–category pairs. The reward is applied only to the last denoising step, and the total loss is the pretraining denoising loss plus a reweighted reward term, so the gradient of the reward is propagated through the decoder and the last denoising step.

What would settle it

Fine-tune the model with the quantity term Aqc removed from the reward and compare quantity accuracy against the full CQ Score; if quantity accuracy does not drop, the claimed quantity-feedback signal is not doing the work. Alternatively, compute the gradient of Aqc through the detection pipeline with respect to the denoising output and check whether it is nonzero at any sampled image.

Watch

Extended reading notes

Core claim

The central claim is that a reward built from object-detection results—specifically the average category confidence and the average quantity confidence, combined through a harmonic mean—can fine-tune a text-to-image diffusion model to produce images that match prompts in both which objects appear and how many of each appear. The authors show that this focused feedback outperforms more generic rewards such as CLIP similarity, ImageReward, and DDPO on alignment metrics, with the largest gains in multi-category, multi-quantity compositions. The paper also states the reward is differentiable and can be optimized by directly backpropagating its gradient, and that the resulting model can be used to score other generators.

Load-bearing premise

The paper assumes the detection-based reward is differentiable with respect to the diffusion model's parameters, even though the count of detected objects per class is an integer produced by thresholding and non-maximum suppression, so the gradient of the quantity term is zero wherever the count does not change.

Editorial extensions

If this is right

  • Fine-tuning on specific detection-based feedback yields measurable alignment gains over the base Stable Diffusion v1.5 model and over ImageReward and DDPO across CLIP, BLIP, and CQ Score metrics.
  • The proposed model can be used as a metric to score text-image alignment of other generators, since its reward is a computable matching score.
  • The new 1,700-pair text-image dataset of category-quantity compositions provides a benchmark for evaluating compositional generation.
  • Applying the reward only at the last denoising step makes the fine-tuning memory-efficient relative to stepwise reinforcement learning approaches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the approach is right, alignment methods should shift from holistic similarity to attribute-specific feedback; the same detection-based reward could be adapted to other diffusion backbones or to other attributes such as spatial position, color, and size.
  • The differentiability claim is questionable given the discrete counting step; a robust implementation would need a soft or approximate differentiable count, or a policy-gradient treatment for the quantity term.
  • The model's utility as a metric is limited to categories detectable by the fixed detector and its COCO label set; prompts with out-of-vocabulary objects would receive zero category confidence.
  • The dataset's filtering by ImageReward may select images that are easier to align, so observed gains could partly reflect distribution shift rather than compositional skill.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a fine-tuning method for text-to-image diffusion models that uses object detection feedback to improve alignment on object categories and quantities. The method constructs a matching score, called CQ Score, from YOLOS detection outputs and a tokenized prompt, uses this score as a reward, and claims to fine-tune Stable Diffusion by backpropagating reward gradients. The authors also introduce a dataset of 1,700 prompt-image pairs and report improvements over SD v1.5, ImageReward, and DDPO on CLIP, BLIP, and CQ Score metrics.

Significance. If the proposed mechanism worked as described, the paper would address a genuine limitation in compositional text-to-image generation, and the release of a dataset and code would be useful to the community. The idea of using detection-based feedback for category and quantity alignment is attractive. However, the central technical claim—that the reward is differentiable and can be optimized with SGD—is not supported by the manuscript as written. The detection pipeline involves hard thresholding, non-maximum suppression, and discrete counts, which are non-differentiable operations. In addition, the primary evaluation metric is identical to the training reward, making the headline improvements circular. These issues are load-bearing rather than cosmetic, so the contribution as presented is not established.

major comments (3)
  1. [Section 3.3 / Algorithm 1] The claim that the reward is differentiable and can be optimized with SGD is not supported. The reward r(x,z)=CQ Score is computed from YOLOS outputs after a hard confidence threshold of 0.8, NMS with IoU threshold 0.5, and integer box counts zni_b. These operations are piecewise constant functions of the image, so their gradients with respect to the diffusion model parameters are zero almost everywhere and undefined at the discrete transitions. Algorithm 1 (lines 11-18) performs the YOLOS forward pass and then computes Lreward, and line 18 applies ∇ω Lreward; however, no straight-through estimator, soft surrogate, or score-function estimator is described. Therefore the asserted learning signal for object quantity is absent, and the reported gains cannot be attributed to the proposed gradient-based fine-tuning.
  2. [Section 3.2, Eq. (2)] The average quantity confidence Aqc does not check category identity. For a perfectly aligned image with detected outputs {person:4; skis:1} and the prompt 'four person and one skis', Eq. (2) gives Aqc = 0.625 rather than 1.0, because it averages min/max count ratios over all (detected class, prompt class) pairs. Consequently, a detection of a wrong category with the same count as a prompt category contributes to Aqc, and even a perfect image is not assigned the maximal score. This structural defect means the CQ Score cannot correctly credit category-specific counts and is not a valid alignment measure.
  3. [Section 4.1 / Table 1] The primary evaluation metric CQ Score is the same function used as the training reward (Section 3.2 defines r(x,z)=CQ Score). Reporting an improvement in CQ Score on a benchmark built from the same prompt distribution and evaluated with the same YOLOS detector is therefore circular. The non-CQ improvements over SD v1.5 (CLIP +7.02%, BLIP +12.93%) are comparatively small, and the paper does not report a train/test split, the number of evaluation prompts, or statistical significance. As a result, the abstract's claim of an average 11.2% improvement over SD v1.5 is not well supported by the evidence presented.
minor comments (5)
  1. [Throughout] There are numerous typos and formatting errors, e.g., 'fine-turning' in the Abstract, 'sepecific' in Section 3.2, 'short for' should be 'shorted for' in Section 3.2, 'Monte Corlo' in Section 3.3, 'DDOP' on page 6, and 'bmy' in Algorithm 1 comments; these should be corrected.
  2. [Section 4.2 / Appendix A.2] The cross-references are inconsistent: the text says 'As shown in Table 2' after presenting Table 1, and the caption for Figure 2 is not descriptive. In Appendix A.2, the method called 'GORS' is attributed to reference [16], but reference [16] is T2I-CompBench, not a method named GORS; the actual GORS work does not appear to be cited.
  3. [Section 3.1 / Section 4.1] The dataset construction and evaluation protocol are underspecified: it is unclear whether the 1,700 image-text pairs are split into training and testing sets, and how the prompts for FID evaluation on MS-COCO are generated.
  4. [Section 3.2, Eq. (1)] The notation in Eq. (1) is confusing: p_i_c is defined as the sum of confidences over boxes of class i, but then p_i_c / zni_b is the average per-box confidence; the text describing 'total confidence score' should be clarified to avoid ambiguity.
  5. [Section 4.3 / Section 5] The claim that the model can serve as a metric for evaluating text-image alignment is not validated against human judgments or an independent benchmark; only a small qualitative comparison is provided.

Circularity Check

1 steps flagged · score 6.0 of 10

CQ Score is simultaneously the training reward and the headline evaluation metric, so its reported gains are forced by construction; CLIP/BLIP gains provide partial independent support.

  1. fitted input called prediction [Section 3.2, Eq. (3) and Section 4.1 'Evaluation Metrics'; Table 1]
    "Our CQ Scorecan guide the model for feedback learning in the form of a reward function, i.e., r(x, z) = CQ Scorefor given prompt x and the generated image z. ... For the alignment evaluation, the CLIP Score [13], BLIP Score [24] and CQ Score of our model improves the original SD v1.5 [31] by 7.02%, 12.93%, 13.65%, respectively."

    The paper trains the diffusion model to maximize CQ Score (L_reward = -CQ Score, with phi the negative reward) and then reports CQ Score as its headline alignment metric. The reported 13.65% CQ Score improvement is therefore the training objective's own improvement, not an independent measurement of text-image alignment. The evaluation metric is built from the same YOLOS detection pipeline, confidence threshold, NMS, and prompt parser used to construct the reward, so the CQ Score gain reduces by construction to 'the model improved at the objective it was optimized for.' This is a self-referential evaluation rather than an external validation.

full rationale

Most of the paper's derivation chain is self-contained: the detection-based reward is constructed from YOLOS outputs and spaCy-tokenized prompts, and fine-tuning combines the standard SD loss with a reward term. The central non-circularity, however, is that the headline alignment metric (CQ Score) is literally the reward function being optimized, r(x,z)=CQ Score; therefore the CQ Score improvement over SD v1.5 is the training objective's own value, not an independent alignment result. Because the same detector, thresholds, and prompt parser feed both reward construction and evaluation, this component of the reported gains is forced by construction. The CLIP and BLIP improvements are external metrics and provide partial independent support, which prevents the entire claim from being circular; there is also no load-bearing self-citation chain in the paper. Two additional correctness concerns are noted but are not forms of circularity: the hard confidence threshold, NMS, and discrete box counts make the reward gradient zero almost everywhere, so the claimed SGD backpropagation path in Section 3.3 is unsupported, and Eq. (2) computes quantity confidence without matching detected classes to prompt classes, so even a perfectly aligned image is not scored maximally.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several assumptions that are not independently verified: the reliability of the COCO-trained detector for the prompt vocabulary, the accuracy of spaCy parsing, the differentiability of the reward pipeline, and the representativeness of the synthetic dataset. The only hand-chosen numerical parameters that materially affect the reward and evaluation are lambda, the detection threshold, the NMS threshold, and the ImageReward filtering threshold.

free parameters (4)
  • lambda (reward weighting factor) = not specified
    In Eq. 6, L = L_pretrain + lambda * L_reward balances the pretraining and reward losses. The paper states 'By choosing an appropriate lambda' but never reports the value, making the trained behavior hard to reproduce.
  • Detection confidence threshold = 0.8
    Section 3.2 discards bounding boxes with confidence below 0.8 before computing counts and confidence sums. This threshold directly shapes the reward and the CQ Score evaluation metric.
  • NMS IoU threshold = 0.5
    Section 3.2 applies non-maximum suppression with IoU greater than 0.5. This affects which boxes survive and therefore affects counts and confidence sums in the reward.
  • ImageReward filtering threshold = not specified
    Section 3.1 selects images with matching scores above an unspecified threshold to build the 1,700-pair dataset. The threshold determines the training distribution and is not stated.
assumptions (4)
  • domain assumption YOLOS trained on MS-COCO 2017 provides reliable category labels and counts for the evaluated prompts.
    The reward and the primary metric rely on the detector's 80-class vocabulary and exact label matching. Prompts like 'polar bear', 'surfing', or 'spacesuits' may not map cleanly to COCO labels, and missing or extra detections directly change the reward (Section 3.2).
  • domain assumption spaCy tokenization extracts object categories and quantities accurately, including plural normalization and compound nouns.
    The reward compares prompt-derived category-quantity maps with detector outputs. Parsing errors, such as mishandling 'polar bears' or 'two dogs and two cats', propagate into both training and evaluation (Section 3.2).
  • ad hoc to paper The reward function is differentiable with respect to the model parameters despite thresholding, non-maximum suppression, and discrete counts.
    Section 3.3 asserts differentiability and uses SGD backpropagation, but the detection pipeline includes non-differentiable operations. This is a load-bearing assumption that is not justified and is central to the claimed learning mechanism.
  • domain assumption Fine-tuning on 1,700 synthetic text-image pairs filtered by ImageReward generalizes to broader compositional prompts.
    The dataset is generated by a controllable diffusion model and filtered automatically by ImageReward (Section 3.1). There is no human verification or diverse distribution described, and the evaluation uses prompts of the same style, so generalization is assumed rather than demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bridging the Gap: Aligning Text-to-Image Diffusion Models with Specific Feedback." pith.science (2026). https://pith.science/paper/VQXFZAJX

@misc{pith2026241200122,
  author       = {Pith},
  title        = {Pith review of: Bridging the Gap: Aligning Text-to-Image Diffusion Models with Specific Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VQXFZAJX}},
  note         = {Machine review of arXiv:2412.00122}
}
read the original abstract

Learning from feedback has been shown to enhance the alignment between text prompts and images in text-to-image diffusion models. However, due to the lack of focus in feedback content, especially regarding the object type and quantity, these techniques struggle to accurately match text and images when faced with specified prompts. To address this issue, we propose an efficient fine-turning method with specific reward objectives, including three stages. First, generated images from diffusion model are detected to obtain the object categories and quantities. Meanwhile, the confidence of category and quantity can be derived from the detection results and given prompts. Next, we define a novel matching score, based on above confidence, to measure text-image alignment. It can guide the model for feedback learning in the form of a reward function. Finally, we fine-tune the diffusion model by backpropagation the reward function gradients to generate semantically related images. Different from previous feedbacks that focus more on overall matching, we place more emphasis on the accuracy of entity categories and quantities. Besides, we construct a text-to-image dataset for studying the compositional generation, including 1.7 K pairs of text-image with diverse combinations of entities and quantities. Experimental results on this benchmark show that our model outperforms other SOTA methods in both alignment and fidelity. In addition, our model can also serve as a metric for evaluating text-image alignment in other models. All code and dataset are available at https://github.com/kingniu0329/Visions.

Figures

Figures reproduced from arXiv: 2412.00122 by the authors.

Figure 1
Figure 1. The steps in our fine-tuning method. (1) We create a text-to-image dataset containing different kinds of compositions. (2) We [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Comparison results for text-image alignment in three kinds of compositions on three metrics, including [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Qualitative comparison with three SOTA methods in three kinds of compositions. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparison of images generated by the original SD v1.5 [ [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Comparison of images generated by the original SD v1.5 [ [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Comparison of images generated by the original SD v1.5 [ [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Qualitative comparison with other methods in three kinds of compositions. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 35 canonical work pages

  1. [1]

    Aishwarya Agarwal, Srikrishna Karanam, K. J. Joseph, Apoorv Saxena, Koustava Goswami, and Balaji Vasan Srini- vasan. A-STAR: test-time attention segregation and reten- tion for text-to-image synthesis. In ICCV, pages 2283–2293,

    work page

  2. [2]

    Universal guidance for diffusion models

    Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geip- ing, and Tom Goldstein. Universal guidance for diffusion models. In ICLR. OpenReview.net, 2024. 2

    work page 2024

  3. [3]

    Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models

    Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models. ACM Trans. Graph., 42(4):148:1–148:10, 2023. 2, 10

    work page 2023

  4. [4]

    Training-free layout control with cross-attention guidance

    Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. pages 5331– 5341, 2024. 2

    work page 2024

  5. [5]

    Kevin Clark, Paul Vicol, Kevin Swersky, and David J. Fleet. Directly fine-tuning diffusion models on differentiable re- wards. In ICLR. OpenReview.net, 2024. 1, 2

    work page 2024

  6. [6]

    RAFT: reward ranked finetuning for generative foundation model alignment.Trans

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: reward ranked finetuning for generative foundation model alignment.Trans. Mach. Learn. Res., 2023, 2023. 2

    work page 2023

  7. [7]

    Optimizing DDPM sampling with shortcut fine-tuning

    Ying Fan and Kangwook Lee. Optimizing DDPM sampling with shortcut fine-tuning. pages 9623–9639. PMLR, 2023. 2

    work page 2023

  8. [8]

    DPOK: reinforcement learning for fine-tuning text-to-image diffu- sion models

    Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Moham- mad Ghavamzadeh, Kangwook Lee, and Kimin Lee. DPOK: reinforcement learning for fine-tuning text-to-image diffu- sion models. CoRR, abs/2305.16381, 2023. 2, 3

    arXiv 2023

Show all 46 references
  1. [9]

    You only look at one sequence: Rethinking transformer in vision through object detection

    Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. In NeurIPS, pages 26183–26197,

  2. [10]

    Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang

    Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Ar- jun R. Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured dif- fusion guidance for compositional text-to-image synthesis. In ICLR, 2023. 1, 2

    2023

  3. [11]

    Benchmarking spatial relationships in text-to-image generation

    Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vi- neet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation. CoRR, abs/2212.10015, 2022. 1

    2022 arXiv

  4. [12]

    Optimizing prompts for text-to-image generation

    Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image generation. In NeurIPS, 2023. 2

    2023

  5. [13]

    Clipscore: A reference-free evaluation met- ric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation met- ric for image captioning. pages 7514–7528. Association for Computational Linguistics, 2021. 1, 2, 5, 10

    2021

  6. [14]

    spaCy 2: Natural lan- guage understanding with Bloom embeddings, convolutional neural networks and incremental parsing

    Matthew Honnibal and Ines Montani. spaCy 2: Natural lan- guage understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2018. 1, 3

    2018

  7. [15]

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Os- tendorf, Ranjay Krishna, and Noah A. Smith. TIFA: accurate and interpretable text-to-image faithfulness evaluation with question answering. In ICCV, pages 20349–20360, 2023. 1

    2023

  8. [16]

    T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xi- hui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. 2023. 2, 10

    2023

  9. [17]

    Comat: Aligning text-to-image diffusion model with image- to-text concept matching

    Dongzhi Jiang, Guanglu Song, Xiaoshi Wu, Renrui Zhang, Dazhong Shen, Zhuofan Zong, Yu Liu, and Hongsheng Li. Comat: Aligning text-to-image diffusion model with image- to-text concept matching. CoRR, abs/2404.03653, 2024. 2

    2024 arXiv

  10. [18]

    Realigndiff: Boosting text-to-image diffusion model with coarse-to-fine semantic re-alignment

    Zutao Jiang, Guian Fang, Jianhua Han, Guansong Lu, Hang Xu, Shengcai Liao, Xiaojun Chang, and Xiaodan Liang. Realigndiff: Boosting text-to-image diffusion model with coarse-to-fine semantic re-alignment. arXiv preprint arXiv:2305.19599, 2023. 1

    2023 arXiv

  11. [19]

    Dif- fusionclip: Text-guided diffusion models for robust image manipulation

    Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Dif- fusionclip: Text-guided diffusion models for robust image manipulation. In CVPR, pages 2416–2425. IEEE, 2022. 2

    2022

  12. [20]

    Dense text-to-image generation with attention modulation

    Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. In ICCV, pages 7667–7677, 2023. 2

    2023

  13. [21]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Ma- tiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In NeurIPS, 2023. 1, 2 7

    2023

  14. [22]

    Maskgan: Towards diverse and interactive facial image ma- nipulation

    Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image ma- nipulation. In CVPR, 2020. 5

    2020

  15. [23]

    Align- ing text-to-image models using human feedback

    Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Moham- mad Ghavamzadeh, and Shixiang Shane Gu. Align- ing text-to-image models using human feedback. CoRR, abs/2302.12192, 2023. 1, 2

    2023 arXiv

  16. [24]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In In- ternational Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , pages ...

    2022

  17. [25]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , pages 19730...

    2023

  18. [26]

    GLIGEN: open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jian- wei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. GLIGEN: open-set grounded text-to-image generation. In CVPR, pages 22511–22521, 2023. 2, 3

    2023

  19. [27]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, pages 740–755. Springer, 2014. 2, 5

    2014

  20. [28]

    Training diffusion models towards diverse image generation with reinforcement learning

    Zichen Miao, Jiang Wang, Ze Wang, Zhengyuan Yang, Li- juan Wang, Qiang Qiu, and Zicheng Liu. Training diffusion models towards diverse image generation with reinforcement learning. In CVPR, pages 10844–10853. IEEE, 2024. 1, 2, 5, 6, 7, 9, 11, 12, 13, 14

    2024

  21. [29]

    Hierarchical text-conditional image gener- ation with CLIP latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with CLIP latents. CoRR, abs/2204.06125, 2022. 2

    2022 arXiv

  22. [30]

    Linguistic bind- ing in diffusion models: Enhancing attribute correspondence through attention map alignment

    Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Rav- fogel, Yoav Goldberg, and Gal Chechik. Linguistic bind- ing in diffusion models: Enhancing attribute correspondence through attention map alignment. In NeurIPS, 2023. 2

    2023

  23. [31]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, pages 10674– 10685. IEEE, 2022. 1, 2, 3, 5, 6, 9, 10, 11, 12, 13

    2022

  24. [32]

    Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with de...

    2022

  25. [33]

    LAION- 400M: open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION- 400M: open dataset of clip-filtered 400 million image-text pairs. CoRR, abs/2111.02114, 2021. 5

    2021 arXiv

  26. [34]

    LAION-5B: an open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAI...

    2022

  27. [35]

    Dream- sync: Aligning text-to-image generation with image under- standing feedback

    Jiao Sun, Deqing Fu, Yushi Hu, Su Wang, Royi Rassin, Da-Cheng Juan, Dana Alon, Charles Herrmann, Sjoerd van Steenkiste, Ranjay Krishna, and Cyrus Rashtchian. Dream- sync: Aligning text-to-image generation with image under- standing feedback. CoRR, abs/2311.17946, 2023. 2

    2023 arXiv

  28. [36]

    Tokencompose: Text-to-image diffusion with token-level supervision

    Zirui Wang, Zhizhou Sha, Zheng Ding, Yilin Wang, and Zhuowen Tu. Tokencompose: Text-to-image diffusion with token-level supervision. In CVPR, pages 8553–8564. IEEE,

  29. [37]

    Gonzalez, Boyi Li, and Trevor Darrell

    Tsung-Han Wu, Long Lian, Joseph E. Gonzalez, Boyi Li, and Trevor Darrell. Self-correcting llm-controlled diffusion models. In CVPR, pages 6327–6336, 2024. 2, 3

    2024

  30. [38]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. CoRR, abs/2306.09341, 2023. 1, 2

    2023 arXiv

  31. [39]

    Human preference score: Better aligning text-to- image models with human preference

    Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hong- sheng Li. Human preference score: Better aligning text-to- image models with human preference. InICCV, pages 2096–

  32. [40]

    Deep reward supervisions for tuning text-to-image diffusion models

    Xiaoshi Wu, Yiming Hao, Manyuan Zhang, Keqiang Sun, Zhaoyang Huang, Guanglu Song, Yu Liu, and Hongsheng Li. Deep reward supervisions for tuning text-to-image diffusion models. CoRR, abs/2405.00760, 2024. 1, 2

    2024 arXiv

  33. [41]

    Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion

    Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wen- tian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In ICCV, pages 7418–7427, 2023. 2

    2023

  34. [42]

    Imagere- ward: Learning and evaluating human preferences for text- to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagere- ward: Learning and evaluating human preferences for text- to-image generation. In NeurIPS, 2023. 1, 2, 3, 5, 6, 7, 9, 11, 12, 13, 14

    2023

  35. [43]

    Mastering text-to-image diffu- sion: Recaptioning, planning, and generating with multi- modal llms

    Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Ste- fano Ermon, and Bin Cui. Mastering text-to-image diffu- sion: Recaptioning, planning, and generating with multi- modal llms. 2024. 2, 3

    2024

  36. [44]

    A dense reward view on aligning text-to-image diffusion with prefer- ence

    Shentao Yang, Tianqi Chen, and Mingyuan Zhou. A dense reward view on aligning text-to-image diffusion with prefer- ence. OpenReview.net, 2024. 2

    2024

  37. [45]

    Freedom: Training-free energy-guided condi- tional diffusion model

    Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-free energy-guided condi- tional diffusion model. In ICCV, pages 23117–23127. IEEE,

  38. [46]

    Fixed Category & Incremental Quantity

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pages 3813–3824. IEEE, 2023. 1, 3 8 APPENDIX A. More Experimental Results A.1. Analysis on different types of compositions Quantitative Comparison. We design thr...

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.