REVIEW 3 major objections 6 minor 5 cited by
OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read OneIG-Bench's central claim is that text-to-image quality should be measured as six separately scoreable dimensions, and the benchmark's automated metrics make that decomposition reproducible.
desk verdict Solid multi-dimensional T2I benchmark with a valuable bilingual dataset, but the reasoning metric is self-referential and the human-validation claim is unsupported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the six-category taxonomy plus a purpose-built metric for each category. The named objects are: the question dependency graph from DSG for semantic alignment; the composite text score $S_{\text{text}}=1-\min(\phi,\mathrm{ED})(1-\mathrm{CR})(1-\mathrm{WAC})/\phi$, with $\phi=100$ for English and $\phi=50$ for Chinese; GPT-4o reference answers paired with LLM2CLIP cosine similarity for knowledge and reasoning; a dual style-encoding pipeline using CSD and a CLIP-based OneIG encoder, taking the maximum cosine similarity to reference images; and DreamSim-based pairwise diversity computed over multiple images per prompt. The framework's flexibility comes from treating each dimension as an independent subset, so a user can run one evaluation slice without generating images for the entire prompt set.
What would settle it
Take the 225 Knowledge and Reasoning prompts, replace GPT-4o's reference answers with answers written by a different large language model or by human experts, recompute the reasoning scores, and check whether the ranking of models changes; if leading models swap positions, the reasoning dimension is not measuring a stable capability.
Extended reading notes
Core claim
The central discovery is a measurement decomposition: OneIG-Bench claims to make fine-grained evaluation of text-to-image models reproducible across the capabilities that matter in practice. It provides curated prompts, roughly 200 per category, drawn from real-world inputs, deduplicated, and rewritten into short, middle, and long lengths, with each category assigned a metric designed for that capability. Alignment is scored by decomposing a prompt into a dependency graph and verifying root and leaf questions against the generated image with a vision-language model. Text rendering is scored by edit distance, exact completion rate, and word accuracy, fused into a single text score. Reasoning is scored by the cosine similarity between the generated image and a GPT-4o-generated textual answer, computed with LLM2CLIP. Style is scored by averaging two style encoders' similarities to reference images, and diversity is scored by pairwise DreamSim distance among images from the same prompt. The results rank closed-source models first on most dimensions and show that no single model dominates all six.
Load-bearing premise
The reasoning score assumes that GPT-4o's written answers are the correct reference for what a reasoning image should contain, and that the text-image similarity score used to compare those answers with generated images really measures reasoning quality, even though GPT-4o is also one of the models being ranked.
Editorial extensions
If this is right
- Because each dimension is scored independently, a model can be selected by application: exact text rendering, reasoning, style, alignment, or diversity can be compared without conflating qualities into one number.
- The bilingual comparison exposes a common weakness: most models can barely render Chinese text, while Seedream 3.0 reaches a text score of 0.928 on OneIG-Bench-ZH, so text rendering should be reported separately per script.
- The edit-distance rule that counts any capitalization mismatch as an error directly lowers visually strong renderers such as GPT-4o, so text rankings should be interpreted with that strictness in mind.
- The reasoning dimension gives reasoning-oriented text-to-image models a place to show capability that alignment-only benchmarks miss, and the paper's tiered ranking can serve as a target for open-source models.
- The deliberate 1:2:1 short, middle, and long prompt structure makes it possible to isolate where long-prompt failures begin, since alignment and text scores degrade with prompt length for most models.
Reading between the lines
- The reasoning score may be sensitive to which model writes the reference answer, because GPT-4o supplies the textual ground truth and is also one of the scored image generators; swapping the answer generator for an independent source would test whether the rankings persist.
- The diversity metric is informative mainly among models with comparable alignment, since outputs that ignore the prompt can look diverse; conditioning diversity on alignment or reporting both jointly would make the comparison more reliable.
- The same per-dimension construction could be extended to video or multi-image generation by reusing the prompt taxonomy and replacing the image-level metrics with temporal or cross-frame checks.
- Public release of per-prompt scores, not just averages, would let the community test whether dimension scores are stable under prompt resampling and would expose error patterns that aggregate scores hide.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces OneIG-Bench, an English-Chinese benchmark containing 2,440 prompts across six categories (General Object, Portrait, Anime & Stylization, Text Rendering, Knowledge & Reasoning, Multilingualism), together with automated metrics for semantic alignment, text rendering, style, diversity, and knowledge/reasoning. The benchmark is validated by evaluating 22 text-to-image models, including open-source, unified multimodal, and proprietary models, and reporting per-dimension scores with fine-grained breakdowns by prompt type and length. The authors make the dataset and code publicly available and highlight the benchmark's flexibility in evaluating user-selected subsets.
Significance. If the proposed metrics are valid, OneIG-Bench fills real gaps in the evaluation landscape: it explicitly targets reasoning, style, text rendering, and multilingual generation, dimensions that earlier benchmarks such as GenEval, DPG-Bench, and T2ICompBench only partially cover. The public release of code and data is a concrete strength that supports reproducibility and cross-model comparison. The most significant caveat is that the Knowledge and Reasoning metric depends on a circular, unvalidated reference, which currently undermines the benchmark's claim to provide a trustworthy multidimensional evaluation. Because the reasoning dimension is a headline contribution, this issue must be resolved before the benchmark can be relied upon.
major comments (3)
- [§4.1, Knowledge and Reasoning] The reasoning score uses GPT-4o-generated textual answers as the reference and LLM2CLIP cosine similarity between that text and the generated image as the score. Since GPT-4o image generation is itself one of the evaluated models (Table 2), the evaluation is self-referential: it measures how closely an image matches GPT-4o's own textual expression, not whether the image is factually correct or logically coherent. The consequence is visible in Tables 2 and 3, where GPT-4o receives the highest reasoning scores (0.345 and 0.300). No evidence is provided that LLM2CLIP cosine similarity tracks human judgments of reasoning quality. This is load-bearing because reasoning is one of the six benchmark dimensions and is emphasized in the abstract as a key contribution.
- [Limitation section] The single sentence claiming that "our metric rankings align closely with human evaluations" is not accompanied by any protocol, data, or statistical measure. For a benchmark whose purpose is to provide valid automated rankings, the absence of a human validation study is a serious omission. At minimum, the authors should report the number of annotators, the number of prompts sampled, the agreement metric (e.g., Spearman or Kendall correlation), and ideally a breakdown per dimension, with particular attention to the reasoning and style dimensions.
- [§4.1, Style] The style score relies on a newly introduced "OneIG style image encoder" fine-tuned from CLIP using CSGO-generated images, but the manuscript provides no details on the training data, architecture modifications, training procedure, or validation of this encoder. Also, the notation in Eq. (2) defines R as a set of three reference images while using l for the number of reference images and max over j, leaving the reference count ambiguous. These omissions prevent reproducibility and do not rule out bias introduced by using generated images to train the evaluator.
minor comments (6)
- [§1] The phrase "lackly coverage" should be "lacks coverage".
- [Table 1] The column header "Evalution" is misspelled as "Evalution".
- [Table 1 and References] GenEval is cited as reference [39] in Table 1, but reference [39] is GenAI-Bench; in the text GenEval is correctly cited as [25]. This inconsistency should be corrected.
- [Figure 3 caption] The caption includes "the the text rendering results," with a duplicated "the."
- [Figure 9 caption] The caption begins "Tha task is to give...", which should be "The task is to give...".
- [§4.1, Eq. (1)] The edit distance upper bound φ is introduced after the equation; for readability, define φ before presenting the formula.
Circularity Check
Knowledge & Reasoning score is defined as similarity to GPT-4o-generated reference text while GPT-4o is itself one of the scored models, making the reasoning ranking self-referential by construction.
-
self definitional
[Section 4.1 (Metrics), Knowledge and Reasoning paragraph; Tables 2 and 3]
"GPT-4o is responsible for generating the textual reasoning answers, which serve as the core reference for evaluation. LLM2CLIP then measures the alignment between text and image by calculating the cosine similarity between the GPT-4o-generated answer and the corresponding generated image."
The Knowledge & Reasoning score is defined as the cosine similarity between an image and a text answer that is itself generated by GPT-4o, a model that also appears as one of the evaluated systems in Tables 2 and 3. The reference standard for 'reasoning' is therefore not an externally verified ground truth but GPT-4o's own textual paraphrase of the expected answer. The metric consequently measures how closely each generated image matches GPT-4o's textual expression, so any model whose output distribution resembles GPT-4o's is rewarded by construction. This is not a speculative bias: GPT-4o obtains the highest Reasoning score on both OneIG-Bench-EN (0.345) and OneIG-Bench-ZH (0.300).
full rationale
OneIG-Bench is a benchmark paper rather than a mathematical derivation, and most of its evaluation dimensions are externally grounded: semantic alignment uses DSG-style question graphs with Qwen2.5-VL answers, text rendering compares extracted strings to the prompt text via edit distance, and diversity uses DreamSim pairwise similarity. These components are self-contained and do not reduce to their own inputs. The one load-bearing circular step is the Knowledge & Reasoning metric in Section 4.1. The paper defines the reasoning reference as GPT-4o-generated text and then scores all models, including GPT-4o image generation, by LLM2CLIP cosine similarity to that text. Since the ground truth is authored by the very model family being ranked, the reasoning score is self-definitional: the quantity the benchmark claims to measure (reasoning quality) is operationally defined as agreement with GPT-4o's output distribution. The paper's own Limitation section concedes that reasoning evaluation is novel and that the only support is an undocumented claim of alignment with human evaluations, which cannot break the circularity because no human study is presented. The style sub-score also uses an encoder fine-tuned on images from CSGO, a method co-authored by one of this paper's authors, but that encoder is averaged with the external CSD metric and is not used as a ground-truth reference, so it does not constitute a derivation-level circularity. Overall, the benchmark has substantial independent content, but its highlighted reasoning dimension reduces by construction to agreement with one of the evaluated models, warranting a partial circularity score of 6.
Assumptions & free parameters
free parameters (5)
- phi_EN =
100
- phi_ZH =
50
- short/medium/long word thresholds =
30 and 60 words
- Beta distribution parameters =
2.37, 2.86
- number of reference images per style (l) =
3
assumptions (6)
- domain assumption GPT-4o-generated reasoning answers are a valid reference for evaluating image reasoning.
- domain assumption LLM2CLIP cosine similarity captures reasoning quality in images.
- domain assumption The DSG question graph generated by GPT-4o provides a reliable semantic-alignment evaluation.
- domain assumption The style reference images and the two encoders (CSD and OneIG style encoder) adequately capture the target style.
- domain assumption The prompt collection and LLM-rewriting pipeline yields diverse, unbiased prompts.
- ad hoc to paper The word-length ratio 1:2:1 and the chosen thresholds are appropriate for evaluating models at different prompt complexities.
invented entities (1)
-
OneIG style image encoder
Cite this review
Pith. "Pith review of OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation." pith.science (2026). https://pith.science/paper/STWCDIS6
@misc{pith2026250607977,
author = {Pith},
title = {Pith review of: OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/STWCDIS6}},
note = {Machine review of arXiv:2506.07977}
}
read the original abstract
Text-to-image (T2I) models have garnered significant attention for generating high-quality images aligned with text prompts. However, rapid T2I model advancements reveal limitations in early benchmarks, lacking comprehensive evaluations, for example, the evaluation on reasoning, text rendering and style. Notably, recent state-of-the-art models, with their rich knowledge modeling capabilities, show promising results on the image generation problems requiring strong reasoning ability, yet existing evaluation systems have not adequately addressed this frontier. To systematically address these gaps, we introduce OneIG-Bench, a meticulously designed comprehensive benchmark framework for fine-grained evaluation of T2I models across multiple dimensions, including prompt-image alignment, text rendering precision, reasoning-generated content, stylization, and diversity. By structuring the evaluation, this benchmark enables in-depth analysis of model performance, helping researchers and practitioners pinpoint strengths and bottlenecks in the full pipeline of image generation. Specifically, OneIG-Bench enables flexible evaluation by allowing users to focus on a particular evaluation subset. Instead of generating images for the entire set of prompts, users can generate images only for the prompts associated with the selected dimension and complete the corresponding evaluation accordingly. Our codebase and dataset are now publicly available to facilitate reproducible evaluation studies and cross-model comparisons within the T2I research community.
Figures
Figures from the paper (9 more)
Forward citations
Cited by 5 Pith papers
-
Parallel Decoding Distillation for Fast Image and Video Generation
A trajectory-based distillation method trains a student to predict multiple mean velocities per network evaluation, enabling 4-8 step generation with competitive quality and improved diversity.
-
Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer
Z-Image is an efficient 6B-parameter foundation model for image generation that rivals larger commercial systems in photorealism and bilingual text rendering through a new single-stream diffusion transformer and strea...
-
Interleaving Reasoning for Better Text-to-Image Generation
A text-image-text-image multi-turn pipeline improves text-to-image generation, gaining up to 8 points over the base model on several benchmarks while adding an image-conditioned reflection step.
-
X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again
GRPO reinforcement learning applied to a discrete autoregressive image generator with a diffusion decoder improves instruction following, image quality, and long-text rendering in a unified multimodal model.
-
Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budget
An open-source image-generation family shows that agentic prompt rewriting and a stronger text encoder can lift quality to near closed-source levels with only 208.62M images and about $400K of training compute.
Reference graph
Works this paper leans on
-
[1]
Vqa: Visual question answering
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015
2015
-
[2]
Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023
arXiv 2023
-
[4]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...
arXiv 2025
-
[5]
Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models
Eslam Mohamed Bakr, Pengzhan Sun, Xiaoqian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny. Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20041–20053, 2023
work page 2023
-
[6]
The official api of flux-1.dev
black-forest labs. The official api of flux-1.dev. https://api.us1.bfl.ai/scalar#tag/tasks/POST /v1/flux-dev, 2024
work page 2024
-
[7]
Attend-and-excite: Attention- based semantic guidance for text-to-image diffusion models
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention- based semantic guidance for text-to-image diffusion models. ACM transactions on Graphics (TOG) , 42(4):1–10, 2023
work page 2023
-
[8]
Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset
Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset. arXiv preprint arXiv:2505.09568, 2025
arXiv 2025
-
[9]
Pixart-sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision, pages 74–91. Springer, 2024
work page 2024
Show all 79 references
-
[10]
Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023
-
[11]
Mmdetection: Open mmlab detection toolbox and benchmark
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019
1906 arXiv
-
[12]
Generative pretraining from pixels
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691–1703. PMLR, 2020
2020
-
[13]
Janus-pro: Unified multimodal understanding and generation with data and model scaling
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811, 2025
2025 arXiv
-
[14]
Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vis...
2024
-
[15]
Mask2former for video instance segmentation
Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexander G Schwing. Mask2former for video instance segmentation. arXiv preprint arXiv:2112.10764, 2021
2021 arXiv
-
[16]
Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation
Jaemin Cho, Yushi Hu, Jason Baldridge, Roopal Garg, Peter Anderson, Ranjay Krishna, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. In ICLR, 2024
2024
-
[17]
Google deepmind Imagen4 team. Imagen4. https://storage.googleapis.com/deepmind-media /Model-Cards/Imagen-4-Model-Card.pdf , 2025
2025
-
[18]
Emerging properties in unified multimodal pretraining
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683, 2025
2025 arXiv
-
[19]
Scaling rectified flow transformers for high-resolution image synthesis
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine ...
2024
-
[20]
Training-free structured diffusion guidance for compositional text-to-image synthesis
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05032, 2022
2022 arXiv
-
[21]
Dreamsim: Learning new dimensions of human visual similarity using synthetic data, 2023
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data, 2023
2023
-
[22]
Commonsense-t2i challenge: Can text-to-image generation models understand commonsense? arXiv preprint arXiv:2406.07546, 2024
Xingyu Fu, Muyu He, Yujie Lu, William Yang Wang, and Dan Roth. Commonsense-t2i challenge: Can text-to-image generation models understand commonsense? arXiv preprint arXiv:2406.07546, 2024
2024 arXiv
-
[23]
Distilling diversity and control in diffusion models
Rohit Gandikota and David Bau. Distilling diversity and control in diffusion models. arXiv preprint arXiv:2503.10637, 2025
2025
-
[24]
Seedream 3.0 technical report
Yu Gao, Lixue Gong, Qiushan Guo, Xiaoxia Hou, Zhichao Lai, Fanshi Li, Liang Li, Xiaochen Lian, Chao Liao, Liyang Liu, et al. Seedream 3.0 technical report. arXiv preprint arXiv:2504.11346, 2025
2025 arXiv
-
[25]
Geneval: An object-focused framework for evaluating text-to-image alignment
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132–52152, 2023
2023
-
[26]
Generative adversarial networks.Communications of the ACM, 63(11):139– 144, 2020
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks.Communications of the ACM, 63(11):139– 144, 2020
2020
-
[27]
Evalmuse-40k: A reliable and fine-grained benchmark with comprehensive human annotations for text-to-image generation model evaluation, 2024
Shuhao Han, Haotian Fan, Jiachen Fu, Liang Li, Tao Li, Junhui Cui, Yunqiu Wang, Yang Tai, Jingwei Sun, Chunle Guo, and Chongyi Li. Evalmuse-40k: A reliable and fine-grained benchmark with comprehensive human annotations for text-to-image generation model evaluation, 2024
2024
-
[28]
Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017
2017
-
[29]
Hidream-i1
HiDream-ai. Hidream-i1. https://github.com/HiDream-ai/HiDream-I1 , 2025
2025
-
[30]
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020
2020
-
[31]
Ella: Equip diffusion models with llm for enhanced semantic alignment
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135, 2024
2024 arXiv
-
[32]
Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages ...
2023
-
[33]
T2i-compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation
Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025
2025
-
[34]
T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723–78747, 2023
2023
-
[35]
Llm2clip: Powerful language model unlock richer visual representation
Weiquan Huang, Aoqi Wu, Yifan Yang, Xufang Luo, Yuqing Yang, Liang Hu, Qi Dai, Xiyang Dai, Dongdong Chen, Chong Luo, et al. Llm2clip: Powerful language model unlock richer visual representation. arXiv preprint arXiv:2411.04997, 2024
2024
-
[36]
Imagen 3, 2024
Imagen-Team-Google. Imagen 3, 2024
2024
-
[37]
Pick-a- pic: An open dataset of user preferences for text-to-image generation
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a- pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652–36663, 2023
2023
-
[38]
Black Forest Labs. Flux. https://github.com/black-forest-labs/flux , 2024
2024
-
[39]
Genai-bench: Evaluating and improving compositional text-to-visual generation
Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, et al. Genai-bench: Evaluating and improving compositional text-to-visual generation. arXiv preprint arXiv:2406.13743, 2024
2024 arXiv
-
[40]
Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation, 2024
Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation, 2024
2024
-
[41]
Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR, 2022
2022
-
[42]
Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pages 23–40. Springer, 2024. 14
2024
-
[43]
Wise: A world knowledge-informed semantic evaluation for text-to-image generation
Yuwei Niu, Munan Ning, Mengren Zheng, Bin Lin, Peng Jin, Jiaqi Liao, Kunpeng Ning, Bin Zhu, and Li Yuan. Wise: A world knowledge-informed semantic evaluation for text-to-image generation. arXiv preprint arXiv:2503.07265, 2025
2025 arXiv
-
[44]
Gpt-4o system card
OpenAI. Gpt-4o system card. https://openai.com/index/gpt-4o-system-card/ , 2024
2024
-
[45]
Introducing 4o image generation
OpenAI. Introducing 4o image generation. https://openai.com/index/introducing-4o-image -generation/, 2025
2025
-
[46]
Sdxl: Improving latent diffusion models for high-resolution image synthesis
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023
2023 arXiv
-
[47]
Lumina-image 2.0: A unified and efficient image generative framework, 2025
Qi Qin, Le Zhuo, Yi Xin, Ruoyi Du, Zhen Li, Bin Fu, Yiting Lu, Xinyue Li, Dongyang Liu, Xiangyang Zhu, Will Beddow, Erwann Millon, Wenhai Wang Victor Perez, Yu Qiao, Bo Zhang, Xiaohong Liu, Hongsheng Li, Chang Xu, and Peng Gao. Lumina-image 2.0: A unified and efficient image g...
2025
-
[48]
Phybench: Holistic evaluation of physical perception and reasoning in large language models
Shi Qiu, Shaoyang Guo, Zhuo-Yang Song, Yunbo Sun, Zeyu Cai, Jiashen Wei, Tianyu Luo, Yixuan Yin, Haoxu Zhang, Yi Hu, et al. Phybench: Holistic evaluation of physical perception and reasoning in large language models. arXiv preprint arXiv:2504.16074, 2025
2025 arXiv
-
[50]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...
2021
-
[51]
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019
2019
-
[52]
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020
2020
-
[53]
Hierarchical text-conditional image generation with clip latents
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022
2022 arXiv
-
[54]
Zero-shot text-to-image generation
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning , pages 8821–8831. Pmlr, 2021
2021
-
[55]
High-resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, June 2022
2022
-
[56]
Photorealistic text-to- image diffusion models with deep language understanding
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in neural informatio...
2022
-
[57]
Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. In International conference on machine learning, pages 30105–30118. PMLR, 2023
2023
-
[58]
Measuring style similarity in diffusion models
Gowthami Somepalli, Anubhav Gupta, Kamal Gupta, Shramay Palta, Micah Goldblum, Jonas Geiping, Abhinav Shrivastava, and Tom Goldstein. Measuring style similarity in diffusion models. arXiv preprint arXiv:2404.01292, 2024
2024 arXiv
-
[59]
Denoising diffusion implicit models
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020
2010 arXiv
-
[60]
stable-diffusion-3.5-large
Stability-AI. stable-diffusion-3.5-large. https://github.com/Stability-AI/sd3.5, 2024
2024
-
[61]
Autore- gressive model beats diffusion: Llama for scalable image generation
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autore- gressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024
2024 arXiv
-
[62]
Evalalign: Evaluating text-to-image models through precision alignment of multimodal large models with supervised fine-tuning to human annotations
Zhiyu Tan, Xiaomeng Yang, Luozheng Qin, Mengping Yang, Cheng Zhang, and Hao Li. Evalalign: Evaluating text-to-image models through precision alignment of multimodal large models with supervised fine-tuning to human annotations. arXiv e-prints, pages arXiv–2406, 2024. 15
2024
-
[63]
Chameleon: Mixed-modal early-fusion foundation models
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024
2024 arXiv
-
[64]
Kolors2.0
Kuaishou Kolors team. Kolors2.0. https://app.klingai.com/cn/, 2025
2025
-
[65]
How to create sota image generation with text recrafts ml team insights
Recraft team. How to create sota image generation with text recrafts ml team insights. https://www.re craft.ai/blog/how-to-create-sota-image-generation-with-text-recrafts-ml-team-i nsights, 2024
2024
-
[66]
Recraft v3
Recraft team. Recraft v3. https://www.recraft.ai/blog/recraft-introduces-a-revolutio nary-ai-model-that-thinks-in-design-language?utm_source=ai-bot.cn , 2024
2024
-
[67]
Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024
2024 arXiv
-
[68]
Emu3: Next-token prediction is all you need
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024
2024 arXiv
-
[69]
Image quality assessment: from error visibility to structural similarity
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004
2004
-
[70]
Revisiting text-to-image evaluation with gecko: On metrics, prompts, and human ratings
Olivia Wiles, Chuhan Zhang, Isabela Albuquerque, Ivana Kaji´c, Su Wang, Emanuele Bugliarello, Yasumasa Onoe, Pinelopi Papalampidi, Ira Ktena, Chris Knutsen, et al. Revisiting text-to-image evaluation with gecko: On metrics, prompts, and human ratings. arXiv preprint arXiv:2404...
2024 arXiv
-
[71]
Omnigen2: Exploration to advanced multimodal generation
Chenyuan Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang, Wanli Li, Xiyan Jiang, Yexin Liu, Junjie Zhou, Ze Liu, Ziyi Xia, Chaofan Li, Haoge Deng, Jiahao Wang, Kun Luo, Bo Zhang, Defu Lian, Xinlong Wang, Zhongyuan Wang, Tiejun Huang, and Zheng Liu. Omnigen2: Ex...
2025 arXiv
-
[72]
Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341, 2023
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341, 2023
2023 arXiv
-
[73]
Sana 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer, 2025
Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, Han Cai, et al. Sana 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer, 2025
2025
-
[74]
Show-o: One single transformer to unify multimodal understanding and generation
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024
2024 arXiv
-
[75]
Show-o2: Improved native unified multimodal models
Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. Show-o2: Improved native unified multimodal models. arXiv preprint arXiv:2506.15564, 2025
2025 arXiv
-
[76]
Csgo: Content-style composition in text-to-image generation
Peng Xing, Haofan Wang, Yanpeng Sun, Qixun Wang, Xu Bai, Hao Ai, Renyuan Huang, and Zechao Li. Csgo: Content-style composition in text-to-image generation. arXiv preprint arXiv:2408.16766, 2024
2024 arXiv
-
[77]
Scaling autoregressive models for content-rich text-to-image generation
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2(3):5, 2022
2022 arXiv
-
[78]
Cogview4
THUKEG Z.ai. Cogview4. https://github.com/THUDM/CogView4, 2025
2025
-
[79]
Sigmoid loss for language image pre-training
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023
2023
-
[80]
Worldgenbench: A world-knowledge-integrated benchmark for reasoning-driven text-to-image generation, 2025
Daoan Zhang, Che Jiang, Ruoshi Xu, Biaoxiang Chen, Zijian Jin, Yutian Lu, Jianguo Zhang, Liang Yong, Jiebo Luo, and Shengda Luo. Worldgenbench: A world-knowledge-integrated benchmark for reasoning-driven text-to-image generation, 2025. 16 A Appendix A.1 Word Count Distribution...
2025
-
[81]
K & R" is the abbreviation for
"K & R" is the abbreviation for "Knowledge & Reasoning". Category Avg Short Middle Long Portrait 56.4 0.184 0.443 0.373 General Object 46.5 0.330 0.422 0.248 Anime & Stylization 50.6 0.212 0.522 0.265 Text Rendering 51.2 0.275 0.475 0.250 Knowledge & Reasoning 20.5 0.960 0.018...
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.