Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One benchmark now scores text-to-image and image-customization models on a single, interpretable scale.

desk verdict A useful unified benchmark dataset for multimodal image generation, but the new AMS metric's human-alignment claim rests on only 12 model-level points and needs stronger validation. read the letter →

arxiv 2505.19415 v2 pith:PCD5HDAM submitted 2025-05-26 cs.CV

Hang Hua , Ziyun Zeng , Yizhi Song , Yunlong Tang , Liu He , Daniel Aliaga , Wei Xiong , Jiebo Luo This is my paper
classification cs.CV
keywords multi-modalimagegenerationbenchmarkAspectMatchingScoreVQA-basedevaluationtext-to-imagecustomizedprompt-imagealignmenthuman
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MMIG-Bench claims to be the first benchmark that treats multi-modal image generation as a single task rather than two separate ones, pairing 4,850 densely annotated text prompts with 1,750 multi-view reference images of 380 subjects so that text-to-image, image-conditioned customization, and hybrids can be scored side by side. The paper's central discovery is a three-level evaluation framework whose middle layer, the Aspect Matching Score (AMS), turns each prompt into a set of object, relation, attribute, and counting questions, uses a vision-language model to answer them from the generated image, and scores alignment as the fraction of exact matches. Across 32k human ratings on 17 models, AMS is reported to correlate with human judgment at Spearman rho = 0.699, higher than existing VQA-based metrics. This matters because it would give the field one shared, interpretable yardstick for both prompt following and identity preservation, replacing disjoint toolkits.

What carries the argument

The load-bearing mechanism is the QA-pair pipeline behind AMS: prompts are parsed into aspect graphs by GPT-4o, then GPT-4o with in-context examples generates one or more question-answer pairs per aspect, and human annotators filter and correct them. At scoring time, a single vision-language model answers each question from the generated image, and an exact string match against the gold answer counts as correct. This converts prompt-image alignment into an answerable, per-aspect checklist, which is what makes the score fine-grained and interpretable. The three-level wrapper (low-level artifact and identity metrics, mid-level AMS, high-level aesthetics and preference) is the benchmark's second piece of machinery, letting one suite cover both visual quality and semantic fidelity.

What would settle it

Take the benchmark's prompts, feed the question-answering stage only the text prompt with no generated image, and compute AMS; if it remains high or keeps its 0.699 correlation with human ratings, the metric is capturing prompt priors rather than image content. More directly, swap a generated image with one that contains none of the queried objects: an AMS that still answers yes to presence questions would fail the check.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that a VQA-based, aspect-level metric can serve as a unified mid-level evaluation for both text-only and image-plus-text generation. The authors build MMIG-Bench, a dataset of 4,850 richly annotated prompts paired with 1,750 multi-view reference images across 380 subjects, and define AMS(I, P) = 1/n * sum_i 1(Ans_i = Ans_i), where each QA pair is generated by GPT-4o and human-validated for the semantics of objects, relations, attributes, and counting, and answered by Qwen-VL2.5-72B. They report that AMS achieves the highest Spearman correlation with human ratings (0.699) among BLIPVQA, VQ2, and DSG on 12 text-to-image models, and that per-aspect correlations reach 0.909 for relations and 0.839 for counting. They also benchmark 17 models and report findings such as HiDream-I1 leading most text-to-image metrics, DreamBooth leading identity preservation, and GPT-4o leading semantic alignment in customization while losing identity.

Load-bearing premise

AMS assumes that a question-answer pair generated by GPT-4o and validated by humans, answered exactly by one vision-language model, correctly captures whether an aspect of the prompt appears in the image; if the question can be answered from the prompt alone, or the answering model is inaccurate, the score measures something other than image-text alignment.

Editorial extensions

If this is right

  • A single benchmark can rank text-to-image models, customization models, and hybrids on the same scale, making architecture and data choices directly comparable.
  • AMS separates failures by aspect: relations and counting are the most discriminating dimensions, so practitioners can see whether a model loses spatial logic, object identity, or attribute detail.
  • The leaderboard results indicate that scaling and rectified-flow/diffusion-transformer architectures currently give the strongest text-to-image fidelity, while identity preservation and composition remain a trade-off, as seen in DreamBooth versus GPT-4o.
  • The 32k human ratings provide a reusable validation set for future automated metrics, not just an evaluation of current models.
  • By covering text-only, image-conditioned, and mixed-input tasks, the benchmark supports side-by-side evaluation of every hybrid in between, which existing T2I and customization benchmarks do not offer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the QA-pair generation step is itself a text-to-question model; if future benchmarks automate it without human validation, AMS will inherit hallucinated or redundant questions, so the human-verification stage is the part most worth preserving and auditing.
  • Editorial inference: because AMS is exact-match and aspect-based, it could be applied to video generation by sampling frames, and to editing tasks by checking unchanged regions, though the paper does not claim this.
  • Editorial inference: the reported relation and counting correlations suggest that the hardest dimension of these benchmarks is compositional logic; a model that scores well on objects and attributes but poorly on relations would be a candidate for targeted training data rather than a full architecture change.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces MMIG-Bench, a benchmark and evaluation framework for multimodal image generation. It pairs 1,750 multi-view reference images (in roughly 380–386 groups) with 4,850 text prompts that are densely annotated with aspect phrases and QA pairs, and proposes a three-level evaluation suite: low-level artifact/identity metrics, a mid-level VQA-based Aspect Matching Score (AMS), and high-level aesthetics/preference metrics. The authors benchmark 17 T2I and customization models and report a human evaluation study with 32k ratings. The central claim is that AMS provides a fine-grained, human-aligned prompt-image alignment score, supported by a Spearman correlation of rho=0.699 against human ratings (Table 3), which is higher than DSG (0.692), VQ2 (0.399), and BLIPVQA (0.147).

Significance. If the metric-validation claim held, MMIG-Bench would be a valuable unified benchmark: the dataset size (4,850 prompts, 1,750 reference images, 28,668 QA pairs), the breadth of covered models (17), and the 32k human ratings are substantial assets. The paper also makes a useful step by treating T2I and subject-customized generation under one evaluation protocol. However, the distinctive contribution—the AMS metric as a fine-grained, human-aligned score—is not yet established at the claimed level of precision. The validation rests on 12 model-level data points with no uncertainty quantification, the margin over DSG is small, and the per-image/per-prompt behavior that would justify the word 'fine-grained' is not reported. The benchmark resource itself is still valuable, and the central claim is likely repairable with additional validation experiments.

major comments (3)
  1. [Sec. 5.2 / Table 3] The claim that AMS achieves the highest human correlation (rho=0.699) and is 'a reliable metric' is not supported at the reported granularity. The Spearman correlation is computed over only 12 model-level points, and the paper reports no confidence interval, bootstrap, or significance test for the difference with DSG (rho=0.692). With n=12, the estimate has a very wide interval and the 0.007 difference is far smaller than the plausible uncertainty. Please report bootstrap/permutation confidence intervals for rho and for the AMS-minus-DSG difference, and state explicitly the number of independent units used in each correlation.
  2. [Sec. 4.2.1 Eq. (1) / Sec. 5.2] AMS is defined per image in Eq. (1), but the validation in Table 3 and Table 4 is at the level of model means. Aggregation over thousands of prompts can conceal large per-item errors: a metric can rank models correctly while mis-scoring individual images. To substantiate the 'fine-grained' claim, report per-prompt or per-image agreement with human ratings (e.g., correlation on individual prompt-image pairs, or a mixed-effects analysis with prompt and model as random effects). For Table 4, clarify whether the aspect-level correlations are also model-level, and if so report the corresponding n and confidence intervals.
  3. [Sec. 3.3.2 / Sec. 4.2.1] The construct validity of AMS depends on whether the QA pairs can be answered only by inspecting the generated image, rather than inferred from language priors or the prompt text. The human verification in Sec. 3.4 checks 'logical coherence, accuracy, and relevance to the original aspect phrase,' but it does not verify that the question is answerable from the image alone or that the ground-truth answer is visually discriminative. Since Eq. (1) uses exact-match against answers produced by a single VLM (Qwen-VL2.5-72B), any yes-bias or language-prior shortcut in that VLM would be inherited by AMS. Please add negative controls (e.g., images where the target aspect is deliberately absent, or questions with 'No' as the correct answer) and report per-control accuracy; additionally, report agreement with a second VLM judge or at least per-question error analysis.
minor comments (6)
  1. [Sec. 1] There is a typo in 'opaen-source models'; it should read 'open-source models'.
  2. [Table 2 caption] The caption contains garbled wording: 'across imagees' and a duplicated clause about resolution. Please rewrite the caption for clarity.
  3. [Sec. 3.1 / Sec. 3.2 / Abstract] The number of groups is inconsistent: the abstract says 380 subjects, while Sec. 3.2 and Appendix A.6 say 386 groups. Please harmonize these numbers.
  4. [Sec. 6] The stated limitation that human ratings do not cover visual quality is appropriately acknowledged; however, the abstract's phrase 'strong correlation with human judgments' could be read more broadly. Please specify in the abstract or Sec. 5 that the human judgments are for semantic alignment only.
  5. [Appendix A.6] The prompt templates contain several typos, e.g., 'extrace' and 'Spatila'; please proofread the appendix.
  6. [Table 4] The appendix reports aspect-level correlations (Objects 0.469, Relations 0.909, Attributes 0.601, Counting 0.839) without stating the number of models/prompts used or any uncertainty. Please add this information so readers can assess the reliability of those numbers.

Circularity Check

0 steps flagged · score 2.0 of 10

No construction-level circularity: AMS is an empirical VQA accuracy score validated against external human ratings; self-citations are non-load-bearing.

full rationale

MMIG-Bench's central functional claim is that AMS correlates with human judgments (Table 3, rho=0.699). This is an empirical measurement over 12 T2I models with 32.4k MTurk ratings, not a derivation from the benchmark's inputs. Eq. (1) defines AMS as the exact-match proportion of VLM answers to GPT-4o-generated, human-validated QA pairs; there are no fitted constants, no parameters calibrated to the human scores, and no leaderboard quantity that is reused as an input to the validity claim. The QA pairs are generated from prompts, which creates a real validity risk (a VLM could exploit prompt answerability or yes-bias), and the n=12 correlation has wide uncertainty, but these are correctness/robustness concerns, not circularity. The paper does cite several works by its own authors: FineMatch [17] supplies the aspect parsing taxonomy and dense labels, and PromptCap/MMComposition/VidComposition appear among VQA precedents. These citations are design choices or prior published tooling rather than load-bearing justifications for AMS's validity; the metric's correlation claim is independently evidenced by the MTurk study in this paper. No equation reduces to another by construction, no fitted input is renamed as a prediction, and no uniqueness theorem from the authors' prior work is invoked to force the design. Accordingly, no circular step is established; the score reflects only minor non-load-bearing self-citation.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims are not derived from physical theory; they rest on dataset-construction choices and on the reliability of GPT-4o, Qwen-VL2.5-72B, and MTurk raters. No fitted constants are used, but several hand-set thresholds shape the data and the evaluation.

free parameters (4)
  • Entity frequency cutoff = top 207 entities
    Retained the 207 most frequent entities from over 2,000 extracted from existing T2I benchmarks; this cutoff defines prompt and image coverage (Section 3.1).
  • Reference images per group = 3 to 5
    Annotation guideline for multi-view reference groups; affects identity-preservation difficulty (Section 3.2).
  • Counting aspect minimum = 2
    Instruction template ignores counting aspects below 2, shaping the Counting dimension of AMS (Appendix A.6).
  • Human evaluation sample size = 150 per aspect; 300 for general
    Sample-size choice for MTurk studies; limits precision of correlation estimates in Section 5.1.
assumptions (5)
  • domain assumption GPT-4o-generated prompts and QA pairs, after human verification, correctly represent prompt and image semantics.
    Sections 3.1.1, 3.3.2, and 3.4 rely on generated content plus human correction; errors here propagate to every AMS score.
  • domain assumption Qwen-VL2.5-72B answers the QA pairs correctly, so exact-match scoring is a valid measure of image-text alignment.
    Section 4.2.1 Eq. (1) uses exact match without calibrating for VLM errors or paraphrases.
  • domain assumption Aggregated MTurk 1-5 ratings are reliable ground truth for semantic alignment.
    Section 5.1 uses three ratings per item; no inter-annotator agreement is reported and ratings exclude visual quality.
  • domain assumption The 207 most frequent entities from prior T2I benchmarks are representative of real image generation use.
    Section 3.1 selects entities from GenAI-Bench, HEIM, and Gecko; the benchmark's coverage claim depends on this.
  • domain assumption Images generated at different native resolutions can be compared without resolution bias.
    Tables 1 and 2 compare models at their default resolutions; no resolution normalization or sensitivity analysis is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models." pith.science (2026). https://pith.science/paper/PCD5HDAM

@misc{pith2026250519415,
  author       = {Pith},
  title        = {Pith review of: MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PCD5HDAM}},
  note         = {Machine review of arXiv:2505.19415}
}
read the original abstract

Recent multimodal image generators such as GPT-4o, Gemini 2.0 Flash, and Gemini 2.5 Pro excel at following complex instructions, editing images and maintaining concept consistency. However, they are still evaluated by disjoint toolkits: text-to-image (T2I) benchmarks that lacks multi-modal conditioning, and customized image generation benchmarks that overlook compositional semantics and common knowledge. We propose MMIG-Bench, a comprehensive Multi-Modal Image Generation Benchmark that unifies these tasks by pairing 4,850 richly annotated text prompts with 1,750 multi-view reference images across 380 subjects, spanning humans, animals, objects, and artistic styles. MMIG-Bench is equipped with a three-level evaluation framework: (1) low-level metrics for visual artifacts and identity preservation of objects; (2) novel Aspect Matching Score (AMS): a VQA-based mid-level metric that delivers fine-grained prompt-image alignment and shows strong correlation with human judgments; and (3) high-level metrics for aesthetics and human preference. Using MMIG-Bench, we benchmark 17 state-of-the-art models, including Gemini 2.5 Pro, FLUX, DreamBooth, and IP-Adapter, and validate our metrics with 32k human ratings, yielding in-depth insights into architecture and data design.

Figures

Figures reproduced from arXiv: 2505.19415 by the authors.

Figure 1
Figure 1. Overview of MMIG-Bench. We present a unified multi-modal benchmark which contains [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Statistics of the tags in MMIG-Bench. Top-left: Data distribution of compositional categories and high-level categories for text in T2I task. Bottom-left: Data distribution of text prompts in customization task. Right: Statistics of classes for the reference images. 3.1.1 Prompting GPT for Text Prompt Generation To enable scalable and diverse prompt generation, we use GPT-4o with several predefined instruction templ… view at source ↗
Figure 3
Figure 3. Our data curation pipeline for multi-modal image generation benchmarking. We begin by [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: A qualitative study of text-only (top) and text-image-conditioned (bottom) generation [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Word clouds of text prompts for the text-only generation (T2I) task (left) and the multimodal [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Aspect Distribution of the QA pairs of AMS [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: The AMS of different models on the text-only generation (T2I) task (left) and the multimodal [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Overview of MMIG-Bench. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: More qualitative results of text-only generation methods on MMIG-Bench. [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: More qualitative results of text-only generation methods on MMIG-Bench. [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: More qualitative results of text-image-conditioned generation methods on MMIG-Bench. [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: The interface of user study for general prompt following. [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: The interface of user study for prompt following on [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: The interface of user study for prompt following on [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: The interface of user study for prompt following on [PITH_FULL_IMAGE:figures/full_fig_p028_15.png]
Figure 16
Figure 16. Figure 16: The interface of user study for prompt following on [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    None of ten tested video-generation models reliably remembers objects after occlusion in dynamic scenes; static-camera videos inflate consistency scores.

Reference graph

Works this paper leans on

90 extracted references · 39 canonical work pages · cited by 1 Pith paper

  1. [1]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  2. [2]

    Pixart- σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision ,

  3. [3]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. ArXiv, abs/2501.17811, 2025. URL https://api.semanticscholar.org/ CorpusID:275954151

    arXiv 2025

  4. [4]

    Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation

    Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. ArXiv, abs/2310.18235, 2023. URL https://api.semanticscholar.org/CorpusID:264555374

    arXiv

  5. [5]

    Dreambench, 2022

    dreambench. Dreambench, 2022. https://github.com/nousr/dream-bench

    2022

  6. [6]

    Patrick Esser, Sumith Kulal, A. Blattmann, Rahim Entezari, Jonas Muller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. ArXiv, abs/2403.03206, 2024. URL https://...

    arXiv 2024

  7. [7]

    Personalize anything for free with diffusion transformer

    Haoran Feng, Zehuan Huang, Lin Li, Hairong Lv, and Lu Sheng. Personalize anything for free with diffusion transformer. arXiv preprint arXiv:2503.12590, 2025

    arXiv 2025

  8. [8]

    Commonsense-t2i challenge: Can text-to-image generation models understand commonsense? arXiv preprint arXiv:2406.07546, 2024

    Xingyu Fu, Muyu He, Yujie Lu, William Yang Wang, and Dan Roth. Commonsense-t2i challenge: Can text-to-image generation models understand commonsense? arXiv preprint arXiv:2406.07546, 2024

    arXiv 2024

Show all 90 references
  1. [9]

    An image is worth one word: Personalizing text-to-image generation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022. 10

    2022 arXiv

  2. [10]

    Geneval: An object-focused framework for evaluating text-to-image alignment

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132–52152, 2023

    2023

  3. [11]

    Gemini 2.0 flash, 2025

    Google. Gemini 2.0 flash, 2025. https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024/#gemini-2-0-flash

    2025

  4. [12]

    Evalmuse-40k: A reliable and fine-grained benchmark with comprehensive human annotations for text-to-image generation model evalua- tion

    Shuhao Han, Haotian Fan, Jiachen Fu, Liang Li, Tao Li, Junhui Cui, Yunqiu Wang, Yang Tai, Jingwei Sun, Chunle Guo, and Chongyi Li. Evalmuse-40k: A reliable and fine-grained benchmark with comprehensive human annotations for text-to-image generation model evalua- tion. ArXiv, a...

    2024 arXiv

  5. [13]

    Ella: Equip diffusion models with llm for enhanced semantic alignment

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135, 2024

    2024 arXiv

  6. [14]

    Promptcap: Prompt-guided task-aware image captioning

    Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. Promptcap: Prompt-guided task-aware image captioning. arXiv preprint arXiv:2211.09699, 2022

    2022 arXiv

  7. [15]

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with ques- tion answering. 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 20349–...

    2023

  8. [16]

    Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages ...

    2023

  9. [17]

    Finematch: Aspect-based fine-grained image and text mismatch detection and correction

    Hang Hua, Jing Shi, Kushal Kafle, Simon Jenni, Daoan Zhang, John Collomosse, Scott Cohen, and Jiebo Luo. Finematch: Aspect-based fine-grained image and text mismatch detection and correction. In European Conference on Computer Vision, pages 474–491. Springer, 2024

    2024

  10. [18]

    Mmcomposition: Revisiting the compositionality of pre-trained vision-language models

    Hang Hua, Yunlong Tang, Ziyun Zeng, Liangliang Cao, Zhengyuan Yang, Hangfeng He, Chenliang Xu, and Jiebo Luo. Mmcomposition: Revisiting the compositionality of pre-trained vision-language models. arXiv preprint arXiv:2410.09733, 2024

    arXiv

  11. [19]

    T2i- compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation

    Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i- compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 47:3563–3579,

  12. [20]

    T2i-factualbench: Benchmarking the factuality of text-to-image models with knowledge-intensive concepts

    Ziwei Huang, Wanggui He, Quanyu Long, Yandi Wang, Haoyuan Li, Zhelun Yu, Fangxun Shu, Long Chan, Hao Jiang, Leilei Gan, et al. T2i-factualbench: Benchmarking the factuality of text-to-image models with knowledge-intensive concepts. arXiv preprint arXiv:2412.04300, 2024

    2024 arXiv

  13. [21]

    Imagic: Text-based real image editing with diffusion models

    Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6007–6017, 2023

    2023

  14. [22]

    Profashion: Prototype-guided fashion video generation with multiple reference images

    Xianghao Kong, Qiaosong Qi, Yuanbin Wang, Anyi Rao, Biaolong Chen, Aixi Zhang, Si Liu, and Hao Jiang. Profashion: Prototype-guided fashion video generation with multiple reference images. arXiv preprint arXiv:2505.06537, 2025

    2025

  15. [23]

    Klemen Kotar, Stephen Tian, Hong-Xing Yu, Daniel L. K. Yamins, and Jiajun Wu. Are these the same apple? comparing images based on object intrinsics. ArXiv, abs/2311.00750, 2023. URL https://api.semanticscholar.org/CorpusID:264935263

    2023 arXiv

  16. [24]

    Multi- concept customization of text-to-image diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi- concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1931–1941, 2023. 11

    1931

  17. [25]

    Flux.1, 2024

    Black Forest Labs. Flux.1, 2024. https://bfl.ai/announcements/24-08-01-bfl

    2024

  18. [26]

    Black Forest Labs. Flux. https://github.com/black-forest-labs/flux , 2024

    2024

  19. [27]

    Holistic evaluation of text-to-image models

    Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al. Holistic evaluation of text-to-image models. Advances in Neural Information Processing Systems, 36:69981–70011, 2023

    2023

  20. [28]

    Genai-bench: Evaluating and improving compositional text-to-visual generation

    Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, and Deva Ramanan. Genai-bench: Evaluating and improving compositional text-to-visual generation. ArXiv, abs/2406.13743, 2024. URL https: //api.semantics...

    2024 arXiv

  21. [29]

    Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing

    Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Processing Systems, 36:30146–30166, 2023

    2023

  22. [30]

    Unimo-g: Unified image generation through multimodal conditional diffusion

    Wei Li, Xue Xu, Jiachen Liu, and Xinyan Xiao. Unimo-g: Unified image generation through multimodal conditional diffusion. arXiv preprint arXiv:2401.13388, 2024

    2024 arXiv

  23. [31]

    Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, Dayou Chen, Jiajun He, Jiahao Li, Wenyue Li, Chen Zhang, Rongwei Quan, Jianxiang Lu, Jiabin Huang, Xiaoyan Yuan, Xiao-Ting Zheng, Yixuan Li, J...

    2024 arXiv

  24. [32]

    Evaluating text-to-visual generation with image-to-text generation

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pages 366–384. Springer, 2024

    2024

  25. [33]

    Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation

    Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai Yu, Liang Zhao, Yisong Wang, Jiaying Liu, and Chong Ruan. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and ge...

    2024 arXiv

  26. [34]

    Dreamo: A unified framework for image customization

    Chong Mou, Yanze Wu, Wenxu Wu, Zinan Guo, Pengze Zhang, Yufeng Cheng, Yiming Luo, Fei Ding, Shiwen Zhang, Xinghui Li, et al. Dreamo: A unified framework for image customization. arXiv preprint arXiv:2504.16915, 2025

    2025

  27. [35]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023. URL https: //arxiv.org/abs/2303.08774

    2023 arXiv

  28. [36]

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Q. V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russ Howes, Po-Yao (Bernie) Huang, Shang-Wen Li, Ishan Misra, Mic...

    2023 arXiv

  29. [37]

    Kosmos-g: Generating images in context with multimodal large language models

    Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos-g: Generating images in context with multimodal large language models. ArXiv, abs/2310.02992,

    arXiv

  30. [38]

    Dreambench++: A human-aligned bench- mark for personalized image generation

    Yuang Peng, Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, and Shu-Tao Xia. Dreambench++: A human-aligned bench- mark for personalized image generation. ArXiv, abs/2406.16855, 2024. URL https: //api.semanticscholar.org/CorpusID:...

    2024 arXiv

  31. [39]

    Pexels, 2014

    Pexels. Pexels, 2014. https://www.pexels.com/

    2014

  32. [40]

    URL https://api.semanticscholar.org/CorpusID:263620748

  33. [41]

    Blattmann, Tim Dockhorn, Jonas Muller, Joe Penna, and Robin Rombach

    Dustin Podell, Zion English, Kyle Lacey, A. Blattmann, Tim Dockhorn, Jonas Muller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. ArXiv, abs/2307.01952, 2023. URL https://api.semanticscholar.org/ CorpusID:259341735

    2023 arXiv

  34. [42]

    Lumina-image 2.0: A unified and efficient image generative framework

    Qi Qin, Le Zhuo, Yi Xin, Ruoyi Du, Zhen Li, Bin Fu, Yiting Lu, Jiakang Yuan, Xinyue Li, Dongyang Liu, Xiangyang Zhu, Manyuan Zhang, Will Beddow, Erwann Millon, Victor Perez, Wen-Hao Wang, Conghui He, Bo Zhang, Xiaohong Liu, Hongsheng Li, Yu-Hao Qiao, Chang Xu, and Peng Gao. Lu...

    2025 arXiv

  35. [43]

    Photon-v1

    Photon78. Photon-v1. https://civitai.com/models/84728/photon78, 2023. Accessed: 2025-05-06

    2023

  36. [44]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. 2022

    2022

  37. [45]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2250...

    2023

  38. [46]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional...

    2021

  39. [47]

    Instantbooth: Personalized text-to- image generation without test-time finetuning

    Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to- image generation without test-time finetuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8543–8552, 2024

    2024

  40. [48]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages...

    2024

  41. [49]

    Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models, 2023

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wad- hwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models, 2023

    2023

  42. [50]

    Hidream-i1: A 17b parameter open chinese text-to-image generation model

    HiDream-AI Team. Hidream-i1: A 17b parameter open chinese text-to-image generation model. https://github.com/HiDream-ai/HiDream-I1 , 2024. Accessed: 2025-05-14

    2024

  43. [51]

    Lmm4lmm: Benchmarking and evaluating large-multimodal image generation with lmms

    Jiarui Wang, Huiyu Duan, Yu Zhao, Juntong Wang, Guangtao Zhai, and Xiongkuo Min. Lmm4lmm: Benchmarking and evaluating large-multimodal image generation with lmms. 2025. URL https://api.semanticscholar.org/CorpusID:277741112

    2025

  44. [52]

    Vidcomposition: Can mllms analyze compositions in compiled videos? arXiv preprint arXiv:2411.10979, 2024

    Yunlong Tang, Junjia Guo, Hang Hua, Susan Liang, Mingqian Feng, Xinyang Li, Rui Mao, Chao Huang, Jing Bi, Zeliang Zhang, et al. Vidcomposition: Can mllms analyze compositions in compiled videos? arXiv preprint arXiv:2411.10979, 2024

    2024

  45. [53]

    Instantid: Zero-shot identity-preserving generation in seconds

    Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, Anthony Chen, Huaxia Li, Xu Tang, and Yao Hu. Instantid: Zero-shot identity-preserving generation in seconds. arXiv preprint arXiv:2401.07519, 2024. 13

    2024 arXiv

  46. [54]

    Ms-diffusion: Multi- subject zero-shot image personalization with layout guidance

    Xierui Wang, Siming Fu, Qihan Huang, Wanggui He, and Hao Jiang. Ms-diffusion: Multi- subject zero-shot image personalization with layout guidance. arXiv preprint arXiv:2406.07209, 2024

    2024 arXiv

  47. [55]

    Pf-lrm: Pose-free large reconstruction model for joint pose and shape prediction

    Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. Pf-lrm: Pose-free large reconstruction model for joint pose and shape prediction. arXiv preprint arXiv:2311.12024, 2023

    2023 arXiv

  48. [56]

    Personalized image generation with deep generative models: A decade survey

    Yuxiang Wei, Yiheng Zheng, Yabo Zhang, Ming Liu, Zhilong Ji, Lei Zhang, and Wangmeng Zuo. Personalized image generation with deep generative models: A decade survey. arXiv preprint arXiv:2502.13081, 2025

    2025 arXiv

  49. [57]

    Revisiting text-to-image evaluation with gecko: On metrics, prompts, and human ratings

    Olivia Wiles, Chuhan Zhang, Isabela Albuquerque, Ivana Kaji´c, Su Wang, Emanuele Bugliarello, Yasumasa Onoe, Chris Knutsen, Cyrus Rashtchian, Jordi Pont-Tuset, and Aida Nematzadeh. Revisiting text-to-image evaluation with gecko: On metrics, prompts, and human ratings. arXiv pr...

    2024 arXiv

  50. [58]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Lian zi Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tieju...

    2024 arXiv

  51. [59]

    Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran. O. Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation. ArXiv, abs/2305.10400, 2023. URL https://api.semanticscholar. org/CorpusID:258740893

    2023 arXiv

  52. [60]

    Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721, 2023

    2023 arXiv

  53. [61]

    Grounding- booth: Grounding text-to-image customization

    Zhexiao Xiong, Wei Xiong, Jing Shi, He Zhang, Yizhi Song, and Nathan Jacobs. Grounding- booth: Grounding text-to-image customization. arXiv preprint arXiv:2409.08520, 2024

    2024 arXiv

  54. [62]

    Perceptual artifacts localization for image synthesis tasks

    Lingzhi Zhang, Zhengjie Xu, Connelly Barnes, Yuqian Zhou, Qing Liu, He Zhang, Sohrab Amirghodsi, Zhe Lin, Eli Shechtman, and Jianbo Shi. Perceptual artifacts localization for image synthesis tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICC...

    2023

  55. [63]

    Q-eval-100k: Evaluating visual quality and alignment level for text-to-vision content

    Zicheng Zhang, Tengchuan Kou, Shushi Wang, Chunyi Li, Wei Sun, Wei Wang, Xiaoyu Li, Zongyu Wang, Xuezhi Cao, Xiongkuo Min, Xiaohong Liu, and Guangtao Zhai. Q-eval-100k: Evaluating visual quality and alignment level for text-to-vision content. ArXiv, abs/2503.02357,

    arXiv

  56. [64]

    Scaling autoregressive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-ric...

    2022

  57. [65]

    Objects” dominate with 38.3%, while “Attributes

    Zhuofan Zong, Dongzhi Jiang, Bingqi Ma, Guanglu Song, Hao Shao, Dazhong Shen, Yu Liu, and Hongsheng Li. Easyref: Omni-generalized group image reference for diffusion models via multimodal llm. arXiv preprint arXiv:2412.09618, 2024. 14 A Appendix A.1 Qualitative Results of MMIG...

    2024 arXiv

  58. [68]

    Cogview3: Finer and faster text-to-image generation via relay diffusion

    Wendi Zheng, Jiayan Teng, Zhuoyi Yang, Weihan Wang, Jidong Chen, Xiaotao Gu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogview3: Finer and faster text-to-image generation via relay diffusion. ArXiv, abs/2403.05121, 2024. URL https://api.semanticscholar.org/ CorpusID:268297194

    2024 arXiv

  59. [70]

    entity" should be common objects; e.g., chair, dog, car, lamp, etc

    "entity" should be common objects; e.g., chair, dog, car, lamp, etc. "entity2" is optional. Use "{entity}" as entity1 here

  60. [72]

    interaction

    "interaction" should describe the relationship between "entity1" and "entity2". "spatial interaction" can be "on the left of / on the right of / on / on top of / on the bottom of / beneath / on the side of / neighboring / next to / touching / in front of / behind / with / etc....

  61. [75]

    scene description

    The order of these terms should not be fixed, as long as the sentence still looks natural. E.g., "scene description" can be put at the end. Instruction Template for T2I Prompts Generation (natural) Please generate prompts in a NATURAL format. It should contain one or more "ent...

  62. [77]

    A robot and a dolphin dancing under the ocean, surrounded by swirling schools of fish

    The description should be imaginative. If imaginative, an example: "A robot and a dolphin dancing under the ocean, surrounded by swirling schools of fish"

  63. [78]

    minimalism meets hygge vibes / editorial photoshoot style / baroque detail / etc

    Avoid repeating sentences you’ve already generated. A.6 Text-Image-Conditioned Dataset Overview An overview of our comprehensive MMIG-Bench is shown in Fig. 8. Based on the 207 common entities we curated, we collect 386 reference image groups, each containing 3–5 multi-view, o...

  64. [80]

    negation

    It should include the logic of "negation", such as the examples below: "The girl with glasses is drawing, and the girl without glasses is singing.", "In the supermarket, a man with glasses pays a man without glasses.", "The larger person wears a yellow hat and the smaller pers...

  65. [81]

    Randomly ignore one or more items from [attributes, interactions, background]

  66. [82]

    comparison

    It should have the logic of "comparison", such as the examples below: "In a magnificent castle, a red dragon sits and a green dragon flies.", "A magician holds two books; the left one is open, the right one is closed.", "One cat is sleeping on the table and the other is playin...

  67. [83]

    four dogs played with two toys

  68. [84]

    two chickens, four pens and one lemon

  69. [85]

    Five cylindrical mugs beside two rectangular napkins

  70. [86]

    three helicopters buzzed over two pillows

  71. [87]

    Three cookies on a plate

  72. [88]

    [scene description (op- tional)] + [number][attribute][entity1] + [interaction (spatial or action)] + [number (optional)][attribute][entity2]

    A group of sheep being led by two shepherds across a green field. Avoid repeating sentences you’ve already generated. 18 Instruction Template for T2I Prompts Generation (numeracy in fixed structure) Please generate natural sentences following a format of "[scene description (o...

  73. [89]

    entity" should be common objects; e.g., chair, dog, car, lamp, etc

    "entity" should be common objects; e.g., chair, dog, car, lamp, etc. "entity2" is optional. Use "entity" as entity1 here

  74. [90]

    attribute

    "attribute" should be an adjective that describes "shape / color / material / size / condition / etc."

  75. [91]

    number" should be

    "number" should be "two/three/four/..." before the attribute, indicating the number of entities. It is optional for entity2

  76. [92]

    interaction

    "interaction" should describes the relationship between "entity1" and "entity2". "spatial interaction" can be "on the left of / on the right of / on / on top of / on the bottom of / beneath / on the side of / neighboring / next to / touching / in front of / behind / with / and...

  77. [93]

    scene description

    "scene description" is the background where the entities appear. It can contain other objects. It is optional

  78. [94]

    interaction action

    The "interaction action" can be either in active or passive voice

  79. [95]

    scene description

    The order of these terms should not be fixed, as long as the sentence still looks natural. E.g., "scene description" can be put at the end. Prompt Template for Text Prompts Aspect Extraction You need to analyze the query to a aspect graph that matches all the objects, relation...

  80. [2023]

    URL https://api.semanticscholar.org/CorpusID:259847295

  81. [2024]

    URL https://api.semanticscholar.org/CorpusID:268264262

  82. [2025]

    URL https://api.semanticscholar.org/CorpusID:276775486

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.