Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Align Beyond Prompts: Evaluating World Knowledge Alignment in Text-to-Image Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read State-of-the-art text-to-image models fail to align generated images with world knowledge beyond the prompt, and ABPScore measures this failure better than existing metrics.

desk verdict ABP is a genuinely broader world-knowledge T2I benchmark with a human-grounded metric, but the headline 43% ITKI gain is insecure because the same GPT-4o both rewrites the prompt and grades the image. read the letter →

arxiv 2505.18730 v1 pith:AG37PQOM submitted 2025-05-24 cs.CV

Wenchao Zhang , Jiahe Tian , Runze He , Jizhong Han , Jiao Dai , Miaomiao Feng , Wei Mi , Xiaodan Zhang This is my paper
classification cs.CV
keywords text-to-imagegenerationworldknowledgebenchmarkABPScoremultimodalLLMevaluationcommonsensereasoninginference-timeinjectionimplicitprompt
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that text-to-image models are evaluated only on whether they follow what a prompt says, not on whether the picture matches what the prompt implies about the world. It introduces ABP, a benchmark of 2,060 prompts that quietly encode everyday facts across six domains, and ABPScore, an automatic metric that uses a multimodal language model to check whether a generated image actually contains the implied knowledge. On eight current models, the best performer, GPT-4o, still scores only 0.82 on average, and all models do particularly poorly on chemical scenes. The paper also reports that ABPScore tracks human judgments far better than six existing metrics, and that a training-free prompt-enrichment step called ITKI raises ABPScore by roughly 43 percent on the 200 hardest samples.

What carries the argument

The load-bearing components are the ABP prompt triplets, which encode implicit world knowledge in a visually checkable form, and the ABPScore formula, which averages an indicator over N extracted knowledge statements verified by a multimodal large language model. The third component is ITKI, whose Knowledge Infusor turns an implicit prompt into an enriched prompt that spells out the implied outcome, so the generator has to reason about the physics or commonsense rather than merely render the named objects.

What would settle it

Take a subset of ABP prompts, generate images where the explicit objects are present but the implied behavior is violated, such as a metal ball floating on water, and compare human judgments with ABPScore; if the metric sides with the explicit cue more often than humans do, it is measuring textual explicitness rather than world knowledge.

Watch

Extended reading notes

Core claim

The central claim is that explicit prompt-image alignment and real-world knowledge alignment are distinct capabilities, and that the latter is largely missing from existing text-to-image evaluations. In ABP's formulation each item is a triple of prompt, image, and implicit world knowledge, and ABPScore disaggregates that knowledge into roughly 8.9 atomic checks, each verified by GPT-4o as a visual judge. The paper's measurements show that the best evaluated model, GPT-4o with an overall score of 0.8213, still fails roughly one in five implicit-knowledge checks, that closed-source models outrun open-source ones, and that all eight models score worst on chemical scenes. It further claims that raising the reasoning component at inference time, by asking GPT-4o to rewrite prompts with the implied knowledge made explicit, improves ABPScore by approximately 43 percent across the eight models on 200 challenging samples.

Load-bearing premise

The metric is only as sound as GPT-4o's visual judgment: if the judge rewards explicit textual cues rather than the implied world knowledge, ABPScore could count wrong images as right and ITKI's reported gains could be inflated.

Editorial extensions

If this is right

  • Text-to-image evaluation should treat implicit world knowledge as a separate axis from caption following, otherwise benchmark scores overstate what models understand.
  • A metric like ABPScore, which decomposes an image into individual knowledge checks, can localize a model's failures by domain, and the paper shows chemical scenes are the weakest.
  • Because ITKI works without retraining, improving the reasoning module of a generator is a direct lever for world-knowledge alignment.
  • High fidelity and high textual alignment are not sufficient for applications where physical or factual correctness matters, such as scientific illustration or safety-relevant imagery.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An unstated consequence is that ABPScore is only as trustworthy as its GPT-4o judge; auditing it on the paper's own human-rated images, especially where explicit text and implied knowledge conflict, would separate genuine alignment from textual echo.
  • Because ITKI's 43 percent gain was measured on the 200 hardest prompts, a natural extension is to run it on the full 2,060-prompt set: the average gain should be smaller, but the per-model ordering should shift if reasoning ability is the bottleneck.
  • The same implicit-knowledge checks could be adapted to video generation, where physical expectations such as a shaken soda can fizzing play out over time and cannot be faked by a single static frame.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces ABP, a benchmark of 2,060 prompts designed to test whether text-to-image models align generated images with real-world knowledge that is implicit in the prompt. The prompts span six domains (physical, chemical, animal, plant, human, factual), and the authors collect 30,867 human judgments on images from eight T2I models. They also propose ABPScore, an automatic metric built on GPT-4o that extracts implicit knowledge from a prompt and verifies each knowledge item in the image; ABPScore correlates with human judgments at Spearman 0.434, exceeding existing metrics. The paper further proposes Inference-Time Knowledge Injection (ITKI), which rewrites prompts to make implicit world knowledge explicit, and reports an approximately 43% average improvement in ABPScore on 200 selected hard samples.

Significance. If the central claims hold, the ABP benchmark would be a useful resource: it is broader in domain coverage than prior commonsense/physics benchmarks, includes human judgments, and is publicly released. ABPScore's higher correlation with human judgments relative to existing automatic metrics is a meaningful contribution to evaluation methodology. The training-free ITKI idea is also timely and practical. However, the headline quantitative claim, the 43% improvement from ITKI, is currently insecure because the evaluation and the optimization share the same model family (GPT-4o), and because the 200 samples are selected post hoc from the same metric used for evaluation. These issues do not invalidate the benchmark itself, but they require additional validation before the improvement claim can be accepted.

major comments (4)
  1. [§4.4, Eq. (1), Figure 5] The reported 43% ABPScore improvement from ITKI is not yet convincingly established because the same model family (GPT-4o) is used both as the Knowledge Infusor that rewrites prompts and as the MLLM evaluator in Eq. (1). If the rewritten prompt explicitly states the knowledge that the evaluator is asked to verify (e.g., 'A red chameleon camouflaged on a red leaf'), the evaluator may answer the verification questions from textual cues or prompt-conditioned priors rather than from genuine visual evidence in the image. To support the claim, please add human judgments on the optimized 200 samples, or evaluate with an independent MLLM (e.g., a different model family) when computing ABPScore after ITKI, and report per-model scores with confidence intervals.
  2. [§4.4, Figure 5] The 200 'challenging samples' are selected as those with the lowest ABPScore from the same test set used to measure the improvement. This post hoc selection is vulnerable to regression to the mean and selection bias: random measurement noise at the low end will tend to produce larger apparent improvements on retest. The paper does not provide confidence intervals, significance tests, or a control condition. Please include a control in which prompts are expanded with explicit but irrelevant knowledge, and/or report the ITKI improvement on a randomly selected held-out subset, with per-model error bars and a paired significance test.
  3. [§3.4 and §4.2, Table 2] The correlation analysis supporting ABPScore lacks statistical detail. The paper reports Spearman's rho = 0.434 and Kendall's tau = 0.323 but does not specify the number of units over which the correlation is computed (e.g., images, prompt-model pairs), nor does it report confidence intervals or significance tests for the difference between ABPScore and the baselines. Additionally, the text in §4.2 says 'Pearson and Kendall' while Table 2 reports Spearman and Kendall; please correct this and provide bootstrap confidence intervals for the correlation coefficients.
  4. [§3.2, Eq. (1)] The definition of the ground-truth answer Ai in Eq. (1) is underspecified. The equation uses Ai before it is defined, and it is not clear how Ai is determined for each extracted knowledge item Ri, nor how the 'knowledge of the associated entity' is separated from the implicit knowledge itself. Clarifying this step is important for reproducibility, since the metric's validity depends on how these verification questions and ground-truth answers are constructed.
minor comments (6)
  1. [§4.2] The text says 'We utilize the Pearson and Kendall correlation coefficients' but Table 2 reports Spearman and Kendall; please align the text with the reported coefficients.
  2. [Figure 5] The bar chart shows per-model comparisons before and after ITKI but reports no numerical values or error bars; adding the underlying numbers and error bars would substantially strengthen the presentation of the 43% claim.
  3. [Figure 1] The caption states that correctly generated images are marked with a check mark, but the figure's inline symbols are not always visually clear; consider using larger, more distinguishable markers or a legend.
  4. [§3.3] The statement that each prompt incorporates multiple knowledge anchors is not quantified; please provide the average or distribution of the number of knowledge anchors per prompt.
  5. [§3.4] The citation for Krippendorff's Alpha is [13], which is a TIFA paper, not the standard reference for this statistic; please cite the appropriate methodology reference.
  6. [§3.4] Please clarify how the 5-point Likert scores are aggregated into the reported human-judgment numbers (e.g., averaged across evaluators and images), and how the filtering rule (score difference > 2) affects the final dataset size.

Circularity Check

1 steps flagged · score 6.0 of 10

ITKI's headline 43% gain is not independently established: the same GPT-4o model that expands prompts also judges the resulting images, so the improvement is partly wired into ABPScore by construction.

  1. self definitional [Section 4.4 / Figure 6; ABPScore definition in Section 3.2 Stage 3, Eq. (1)]
    "We utilize GPT-4o as a Knowledge Infusor (KI) to extract world knowledge beyond user-provided prompts. ... By comparing the ABPScores before and after optimization, we observed a significant improvement of approximately 43% across eight T2I models on average."

    ABPScore is defined as the fraction of knowledge items R_i, 'extracted from the prompt', that an MLLM verifies in the generated image (Eq. 1). ITKI rewrites the prompt so the originally implicit knowledge is stated explicitly (e.g., 'a red chameleon' is inserted into 'a chameleon camouflaged on a red leaf'). The verification model is the same GPT-4o family that performed the rewriting. Consequently, the 43% improvement largely measures whether the image contains entities named in the rewritten prompt, not whether the model inferred unstated world knowledge; the target of evaluation is changed by the intervention. The knowledge item and the verification condition come from the same source, so the improvement is partly an artifact of prompt-to-evaluator leakage.

full rationale

The ABP benchmark itself is not circular: the 2,060 prompts are manually filtered, 30,867 human judgments were collected, and ABPScore is externally validated by a Spearman correlation of 0.434 with human judgments. The circularity is confined to the ITKI result. Because ABPScore's knowledge items are extracted from the prompt (Eq. 1) and ITKI uses GPT-4o to inject exactly those items into the prompt, the measured improvement is partly a construct of the evaluation procedure rather than evidence of improved implicit world-knowledge alignment. The selection of the 200 'most challenging' samples by lowest ABPScore further inflates the reported gain through regression to the mean. These issues do not invalidate the dataset or the metric's ranking of models, but they mean the headline 43% improvement should be treated as a prompt-engineering effect requiring independent validation, not as a demonstrated increase in world-knowledge alignment.

Assumptions & free parameters 1 free parameters · 3 assumptions · 2 invented entities

The central claims rest on the reliability of GPT-4o as knowledge extractor, judge, and prompt infusor. The only independent anchor is human judgment correlation, which is moderate. The benchmark's scope is also assumed to be representative of visual world knowledge.

free parameters (1)
  • human judgment disagreement threshold = score difference > 2 excluded
    This threshold in Section 3.4 determines which human judgments are kept (30,867 final counts) and was chosen without sensitivity analysis, potentially affecting reliability statistics.
assumptions (3)
  • domain assumption GPT-4o can reliably extract all relevant world knowledge from a prompt and verify it in an image.
    ABPScore in Eq. (1) depends on this. The paper does not audit the extraction accuracy or compare against a fixed knowledge base.
  • domain assumption Human Likert judgments after filtering approximate ground truth for world-knowledge alignment.
    Section 3.4 uses these judgments to validate ABPScore and to draw conclusions about model competence.
  • domain assumption The six scene categories (physical, chemical, animal, plant, human, factual) adequately cover visually perceivable world knowledge.
    The benchmark's comprehensiveness claim in Section 3.3 rests on this coverage assumption.
invented entities (2)
  • ABPScore independent evidence
    purpose: Automated metric to quantify alignment of generated images with implicit world knowledge.
    It is correlated with human judgments (Spearman 0.434), providing evidence outside the metric itself.
  • Knowledge Infusor (KI)
    purpose: Rewrites prompts to make implicit world knowledge explicit during inference without retraining.
    Its effectiveness is measured only via ABPScore, which relies on GPT-4o, the same model family used by KI.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Align Beyond Prompts: Evaluating World Knowledge Alignment in Text-to-Image Generation." pith.science (2026). https://pith.science/paper/AG37PQOM

@misc{pith2026250518730,
  author       = {Pith},
  title        = {Pith review of: Align Beyond Prompts: Evaluating World Knowledge Alignment in Text-to-Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AG37PQOM}},
  note         = {Machine review of arXiv:2505.18730}
}
read the original abstract

Recent text-to-image (T2I) generation models have advanced significantly, enabling the creation of high-fidelity images from textual prompts. However, existing evaluation benchmarks primarily focus on the explicit alignment between generated images and prompts, neglecting the alignment with real-world knowledge beyond prompts. To address this gap, we introduce Align Beyond Prompts (ABP), a comprehensive benchmark designed to measure the alignment of generated images with real-world knowledge that extends beyond the explicit user prompts. ABP comprises over 2,000 meticulously crafted prompts, covering real-world knowledge across six distinct scenarios. We further introduce ABPScore, a metric that utilizes existing Multimodal Large Language Models (MLLMs) to assess the alignment between generated images and world knowledge beyond prompts, which demonstrates strong correlations with human judgments. Through a comprehensive evaluation of 8 popular T2I models using ABP, we find that even state-of-the-art models, such as GPT-4o, face limitations in integrating simple real-world knowledge into generated images. To mitigate this issue, we introduce a training-free strategy within ABP, named Inference-Time Knowledge Injection (ITKI). By applying this strategy to optimize 200 challenging samples, we achieved an improvement of approximately 43% in ABPScore. The dataset and code are available in https://github.com/smile365317/ABP.

Figures

Figures reproduced from arXiv: 2505.18730 by the authors.

Figure 1
Figure 1. Examples of ABP, example presents images before (fail to align with real-world knowl￾edge) on the left, and after (correction of alignment with world knowledge) optimization using ITKI on the right. Each image includes the ABPSCORE result, with correctly generated images marked with a check mark "✓", while incorrect images are marked with a cross "✗". Abstract Recent text-to-image (T2I) generation models have advanc… view at source ↗
Figure 2
Figure 2. The construction of ABP. (Upper) Collection of Knowledge Anchors. We manually collect and filter knowledge anchors from various online repositories, including Wikipedia and ConceptNet, across six different scenes; (Middle) Constructing Prompts. We use GPT-4o to generate prompts from the collected knowledge anchors, which are then filtered and optimized to align with the criteria of Reasonability, Implicature, and Vi… view at source ↗
Figure 3
Figure 3. Statistics for the ABP dataset. The inner ring illustrates the six world knowledge domains covered by ABP: physical scenes, chemical scenes, animal scenes, plant scenes, human scenes, and factual scenes. As individual prompts may span multiple knowledge domains, the total number of prompts across all domains exceeds 2,060. The outer ring illustrates the five most frequent specific knowledge categories within each do… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Human judgments. We show the average human judgments for eight T2I models, with the first four being open-source (SDXL, SD3-M, SD3.5-L, CogView4) and the remaining four being closed-source (Midjourney V6, Gemini 2.0, DALL-E 3, GPT-4o). Our analysis reveals two key insi…
Figure 5
Figure 5. Figure 5: Performance Comparison Before and After ITKI, each T2I model shows significant improvement. After analyzing the results in Section 4.3, we identified that T2I models exhibit limitations when generating images incorporating world knowl￾edge beyond the provided prompts. …
Figure 6
Figure 6. Figure 6: Examples optimized through ITKI. We utilize GPT-4o as a Knowledge Infusor (KI) to extract world knowledge beyond user-provided prompts. Examples optimized with ITKI achieve higher scores, and the corresponding images better represent the integration of world knowledge.…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    ToolArtist trains a unified multimodal model to reason, search the web, and generate images as one policy, improving scores on WISE and WorldGenBench-Humanities.

Reference graph

Works this paper leans on

53 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [13]

    Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20406–20417, 2023

    2023

  2. [1]

    Tallyqa: Answering complex counting questions

    Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 8076–8084, 2019

    2019

  3. [2]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023. 9

    work page 2023

  4. [3]

    Measuring Progress in Fine-grained Vision-and-Language Understanding

    Emanuele Bugliarello, Laurent Sartran, Aishwarya Agrawal, Lisa Anne Hendricks, and Aida Nematzadeh. Measuring progress in fine-grained vision-and-language understanding. arXiv preprint arXiv:2305.07558, 2023

    work page Pith review arXiv 2023

  5. [4]

    Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation

    Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. arXiv preprint arXiv:2310.18235, 2023

    arXiv

  6. [5]

    Diffusion bridges vector quantized variational autoencoders

    Max Cohen, Guillaume Quispe, Sylvain Le Corff, Charles Ollion, and Eric Moulines. Diffusion bridges vector quantized variational autoencoders. arXiv preprint arXiv:2202.04895, 2022

    work page arXiv 2022

  7. [6]

    Cogview: Mastering text-to-image generation via transformers

    Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in neural information processing systems, 34:19822–19835, 2021

    2021

  8. [7]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024

    2024

Show all 53 references
  1. [8]

    Commonsense-t2i challenge: Can text-to-image generation models understand commonsense? arXiv preprint arXiv:2406.07546, 2024

    Xingyu Fu, Muyu He, Yujie Lu, William Yang Wang, and Dan Roth. Commonsense-t2i challenge: Can text-to-image generation models understand commonsense? arXiv preprint arXiv:2406.07546, 2024

    2024 arXiv

  2. [9]

    Benchmarking spatial relationships in text-to-image generation

    Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation. arXiv preprint arXiv:2212.10015, 2022

    2022 arXiv

  3. [10]

    Generative adversarial networks

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 139–144, 2020

    2020

  4. [11]

    Clipscore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021

    2021 arXiv

  5. [12]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017

    2017

  6. [14]

    T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723–78747, 2023

    2023

  7. [15]

    Evaluating numerical reasoning in text-to-image models

    Ivana Kaji´c, Olivia Wiles, Isabela Albuquerque, Matthias Bauer, Su Wang, Jordi Pont-Tuset, and Aida Nematzadeh. Evaluating numerical reasoning in text-to-image models. Advances in Neural Information Processing Systems, 37:42211–42224, 2024

    2024

  8. [16]

    Diffusionclip: Text-guided diffusion models for robust image manipulation

    Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 2416–2425. IEEE, 2022. doi: 10....

    2022

  9. [17]

    Pick-a- pic: An open dataset of user preferences for text-to-image generation

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a- pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652–36663, 2023

    2023

  10. [18]

    Viescore: Towards explainable metrics for conditional image synthesis evaluation

    Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explainable metrics for conditional image synthesis evaluation. arXiv preprint arXiv:2312.14867, 2023

    2023 arXiv

  11. [19]

    Multi-concept customization of text-to-image diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1931–1941, 2023

    1931

  12. [20]

    Holistic evaluation of text-to-image models

    Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al. Holistic evaluation of text-to-image models. Advances in Neural Information Processing Systems, 36:69981–70011, 2023. 10

    2023

  13. [21]

    Genai-bench: Evaluating and improving compositional text-to-visual generation

    Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, et al. Genai-bench: Evaluating and improving compositional text-to-visual generation. arXiv preprint arXiv:2406.13743, 2024

    2024 arXiv

  14. [22]

    Evaluating and improving compositional text-to-visual generation

    Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Xide Xia, Pengchuan Zhang, Graham Neubig, and Deva Ramanan. Evaluating and improving compositional text-to-visual generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

    2024

  15. [23]

    Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing

    Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Processing Systems, 36:30146– 30166, 2023

    2023

  16. [24]

    Science-t2i: Addressing scientific illusions in image synthesis

    Jialuo Li, Wenhao Chai, Xingyu Fu, Haiyang Xu, and Saining Xie. Science-t2i: Addressing scientific illusions in image synthesis. arXiv preprint arXiv:2504.13129, 2025

    2025

  17. [25]

    Rich human feedback for text-to-image generation

    Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, et al. Rich human feedback for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2024

  18. [26]

    Evaluating text-to-visual generation with image-to-text generation

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pages 366–384. Springer, 2024

    2024

  19. [27]

    Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation

    Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, and William Yang Wang. Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation. Advances in Neural Information Processing Systems, 36:23075–23093, 2023

    2023

  20. [28]

    Inference-time scaling for diffusion models beyond scaling denoising steps

    Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, et al. Inference-time scaling for diffusion models beyond scaling denoising steps. arXiv preprint arXiv:2501.09732, 2025

    2025 arXiv

  21. [29]

    Phybench: A physical commonsense benchmark for evaluating text-to-image models

    Fanqing Meng, Wenqi Shao, Lixin Luo, Yahong Wang, Yiran Chen, Quanfeng Lu, Yue Yang, Tianshuo Yang, Kaipeng Zhang, Yu Qiao, et al. Phybench: A physical commonsense benchmark for evaluating text-to-image models. arXiv preprint arXiv:2406.11802, 2024

    2024 arXiv

  22. [30]

    Midjourney version 6, 2024

    Midjourney. Midjourney version 6, 2024. URL https://www.midjourney.com/. Accessed: 2025-05- 08

    2024

  23. [31]

    Addendum to gpt-4o system card: Native image generation, 2025

    OpenAI. Addendum to gpt-4o system card: Native image generation, 2025. URL https://openai.com/ index/gpt-4o-image-generation-system-card-addendum/ . Accessed: 2025-05-08

    2025

  24. [32]

    Toward verifiable and reproducible human evaluation for text-to-image generation

    Mayu Otani, Riku Togashi, Yu Sawai, Ryosuke Ishigami, Yuta Nakashima, Esa Rahtu, Janne Heikkilä, and Shin’ichi Satoh. Toward verifiable and reproducible human evaluation for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2023

  25. [33]

    Teaching clip to count to ten

    Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3170–3180, 2023

    2023

  26. [34]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  27. [35]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning , pages 8821–8831. Pmlr, 2021

    2021

  28. [36]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  29. [37]

    Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 225...

    2023

  30. [38]

    Photorealistic text-to- image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in neural informatio...

    2022

  31. [39]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016

    2016

  32. [40]

    Enhancing image generation by fusing auto encoder & transformative generation approach

    T Sanjay et al. Enhancing image generation by fusing auto encoder & transformative generation approach. In 2024 International Conference on Electrical Electronics and Computing Technologies (ICEECT) , volume 1, pages 1–6. IEEE, 2024

    2024

  33. [41]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural informa...

    2022

  34. [42]

    Conceptnet 5.5: An open multilingual graph of general knowledge

    Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017

    2017

  35. [43]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  36. [44]

    Revisiting text-to-image evaluation with gecko: On metrics, prompts, and human ratings

    Olivia Wiles, Chuhan Zhang, Isabela Albuquerque, Ivana Kaji´c, Su Wang, Emanuele Bugliarello, Yasumasa Onoe, Pinelopi Papalampidi, Ira Ktena, Chris Knutsen, et al. Revisiting text-to-image evaluation with gecko: On metrics, prompts, and human ratings. arXiv preprint arXiv:2404...

    2024 arXiv

  37. [45]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference...

    2023

  38. [46]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341, 2023

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341, 2023

    2023 arXiv

  39. [47]

    Conceptmix: A com- positional image generation benchmark with controllable difficulty

    Xindi Wu, Dingli Yu, Yangsibo Huang, Olga Russakovsky, and Sanjeev Arora. Conceptmix: A com- positional image generation benchmark with controllable difficulty. arXiv preprint arXiv:2408.14339, 2024

    2024 arXiv

  40. [48]

    Imagereward: Learning and evaluating human preferences for text-to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903–15935, 2023

    2023

  41. [49]

    What you see is what you read? improving text-image alignment evaluation

    Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation. Advances in Neural Information Processing Systems, 36:1601–1619, 2023

    2023

  42. [50]

    Scaling autoregressive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2(3):5, 2022

    2022 arXiv

  43. [51]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023

    2023

  44. [52]

    Gpt-4v (ision) as a generalist evaluator for vision-language tasks

    Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. Gpt-4v (ision) as a generalist evaluator for vision-language tasks. arXiv preprint arXiv:2311.01361, 2023

    2023 arXiv

  45. [53]

    A contrastive compositional benchmark for text-to-image synthesis: A study with unified text-to-image fidelity metrics

    Xiangru Zhu, Penglei Sun, Chengyu Wang, Jingping Liu, Zhixu Li, Yanghua Xiao, and Jun Huang. A contrastive compositional benchmark for text-to-image synthesis: A study with unified text-to-image fidelity metrics. arXiv preprint arXiv:2312.02338, 2023. 12

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.