Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

TIIF-Bench: How Does Your T2I Model Follow Your Instructions?

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TIIF-Bench claims a fine-grained, length-robust way to rank text-to-image models, with rankings nearly identical to human preference.

desk verdict Useful new T2I evaluation benchmark with a sensible text-rendering metric, but the GPT-4o self-loop (question writer, answer setter, scorer) and a tiny user study leave the human-alignment claim underdetermined. read the letter →

arxiv 2506.02161 v3 pith:AHPI6N2T submitted 2025-06-02 cs.CV

Xinyu Wei , Jinrui Zhang , Zeqing Wang , Hongyang Wei , Zhen Guo , Bairui Li , Lei Zhang This is my paper
classification cs.CV
keywords text-to-imagegenerationinstructionfollowingbenchmarkconstructionvision-languagemodelevaluationpromptlengthrobustnesstextrenderingstylecontroleditdistancemetric
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TIIF-Bench is a benchmark built to measure how faithfully text-to-image (T2I) models follow written instructions, a capability the paper argues existing benchmarks cannot accurately assess because their prompts are short, templated, and repetitive and because CLIP-based or single-score evaluation is too coarse. It contains 5,000 prompts across three difficulty tiers, each supplied in a short and a long version that preserve the same core semantics, so that sensitivity to prompt length can be scored for the first time. Evaluation replaces coarse caption-matching with attribute-specific yes/no questions that a large vision-language model answers per image, yielding a fine-grained compliance score, plus a new GNED metric for text rendering. The paper reports that TIIF-Bench's model rankings align with human preference rankings nearly perfectly across every dimension, with Spearman $\rho$ values up to 1.00, while CompBench++ and GenAI-Bench show only weak alignment. If these claims hold, the benchmark gives the field a sharper yardstick for instruction following and supports concrete conclusions, such as GPT-4o's lead, the universal weakness on spatial and logical instructions, and a positive link between instruction comprehension and generation quality.

What carries the argument

The load-bearing mechanism is the attribute-specific yes/no evaluation protocol. For each prompt, core concepts are extracted into concept pools, an LLM such as GPT-4o generates a yes/no question with a ground-truth answer for each concept, and the generated image is presented together with the question list to a VLM (GPT-4o or Qwen2.5-VL-72B); the model's score is the fraction of questions answered correctly, and the original prompt is never shown to the evaluator. The second mechanism is GNED (Global Normalized Edit Distance), computed as the minimal character-level normalized edit distance between target words and OCR-extracted words under optimal bipartite matching, normalized by the longer word list, with a penalty for unmatched words so that both omission and hallucinated text lower the score. Together the two mechanisms are what let the benchmark separate instruction comprehension from image quality and rank models in a way that tracks human preference.

What would settle it

Feed the evaluator a deliberately non-compliant image: generate or hand-pick an image that violates one or more of a prompt's yes/no questions, and count how often the VLM still answers 'yes'; if the evaluator cannot reliably reject clearly wrong images, the reported alignment with human preference cannot be trusted. Alternatively, re-run the user study with more than ten participants and far more than three prompts per dimension, and check whether the Spearman correlation with TIIF-Bench rankings stays near 1.00, since any large drop would falsify the benchmark's central claim. A third check is to regenerate all ground-truth answers with a different LLM than GPT-4o and observe whether the model rankings change materially.

Watch

Extended reading notes

Core claim

The central claim is that fine-grained instruction following in T2I generation is measurable by decomposition: every prompt's core concepts are turned into yes/no questions with known answers, and a VLM scores the generated image by the fraction of those questions it answers correctly. Because the scoring questions never include the original prompt, the design is meant to suppress the VLM hallucination that plagues caption-based judging, where the judge can answer from language rather than from the image. For text rendering, the paper introduces GNED, a metric bounded in $[0,1]$ that matches the prompt's target words to OCR-extracted words from the image via the Hungarian algorithm and penalizes both missing and extra words. On this protocol the paper reports near-perfect Spearman rank correlations with human ranking, finds that stronger instruction-followers are also more robust to prompt length, and observes that autoregressive models such as Janus-Pro match diffusion models in instruction following despite lower visual fidelity.

Load-bearing premise

The load-bearing assumption is that a self-consistent GPT-4o pipeline, which composes the prompts, writes the yes/no questions and expected answers, and then evaluates the images by answering those same questions, measures genuine instruction following; the 10-participant user study correlates the resulting rankings with human preference but does not independently verify the correctness of the question-answer ground truth.

Editorial extensions

If this is right

  • Prompt length becomes a standard evaluation axis: short and long versions of the same instruction rank models differently, so models must be tested at both extremes.
  • Spatial relations and logical reasoning, not simple attributes like color or texture, are the binding constraint for current T2I models, with GPT-4o the only model that holds up across these dimensions.
  • Instruction-comprehension ability is positively correlated with generation quality, since top-ranked models are also the most robust to prompt-length variation.
  • Autoregressive architectures, jointly trained on understanding and generation, reach instruction-following parity with diffusion models despite lower image fidelity, pointing to a promising direction for the next generation.
  • The benchmark's own stated limits are that its prompts cover mainly common objects, are English-only, and ignore stylistic register, so its rankings say nothing about rare-object, multilingual, or register-sensitive instruction following.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's ground truth is generated and scored by the same model family: GPT-4o writes the prompts, writes the questions and expected answers, and judges the images, so TIIF-Bench may in part measure that model's own interpretation of 'following instructions'; sourcing expected answers from human annotation or from a different LLM would test whether the rankings hold.
  • GNED is not specific to T2I evaluation: the same Hungarian-matched edit-distance scoring could apply wherever target text must appear in a generated artifact, such as logo design, document-image synthesis, or glyph rendering in video generation.
  • If the reported correlation between instruction-following quality and prompt-length robustness is real, labs could screen models cheaply on long prompts alone, treating length robustness as a fast proxy for overall instruction comprehension.
  • The headline rankings for closed-source models rest on a 10% testmini subset of the prompts, not the full 5,000, so small score gaps between commercial models should be read with that sample size in mind.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces TIIF-Bench, a text-to-image instruction-following benchmark containing 5,000 prompts organized into six compositional dimensions, with short and long variants of each prompt to test length robustness. Evaluation is performed by decomposing each prompt into attribute-specific yes/no questions generated by an LLM, then asking a VLM to answer those questions on the generated image; text rendering is scored with a proposed GNED edit-distance metric. The authors benchmark a wide set of open- and closed-source T2I models, compare TIIF-Bench rankings with CompBench++ and GenAI-Bench, and report a user study claiming high Spearman correlation between TIIF-Bench scores and human preferences across dimensions.

Significance. If the evaluation protocol is valid, TIIF-Bench would be a useful resource: the prompt set is substantially larger and more diverse than predecessor benchmarks, the short/long prompt design addresses a real evaluation gap, and the yes/no decomposition is a sensible way to obtain fine-grained signal from VLMs. The paper also ships unusually broad model coverage, a second VLM evaluator (Qwen2.5-VL-72B) as a stability check, and a concrete new metric for text rendering. The load-bearing weakness is that the questions, ground-truth answers, and the primary scorer all come from the same model family, so the benchmark measures GPT-4o's internal consistency more directly than it measures human-verifiable instruction following. The user study is too small and too coarse to validate per-prompt ground truth. These issues are fixable in a revision, and the benchmark infrastructure itself appears reusable.

major comments (3)
  1. [Section 3.2, Section 3.3, A1.1] There is a self-referential evaluation loop: GPT-4o composes the prompts (Section 3.2), expands them into long versions (A1.1), generates the yes/no questions and their ground-truth answers (Section 3.3), and then acts as the VLM evaluator that answers those questions. Any systematic bias in GPT-4o's question interpretation, such as a yes-bias or tolerance for a particular failure mode, is therefore shared by the answer set and the scorer. The Qwen2.5-VL-72B cross-check in Table 9 shows ranking stability across two judges, but both judges are scored against the same unvalidated answer set, so it does not break the loop. The central claim of the paper requires independent validation of the auto-generated ground-truth answers; without it, the scores may reflect GPT-4o's self-consistency rather than true instruction following.
  2. [Section 4.3, Table 7] The user study has 10 participants, samples only 3 prompts per dimension, and computes Spearman rho over 8 model-level ranks. With n=8, the reported rho values of 0.81-1.00 have very wide confidence intervals and are sensitive to a single swapped rank. More importantly, the study asks participants to rank whole images by quality, not to answer the per-attribute yes/no questions, so it cannot validate the correctness of the auto-generated ground-truth answers and cannot detect yes-bias or other evaluator errors. The paper should report confidence intervals or significance tests, increase the number of participants and prompts, and add per-question human labels on a sample of the benchmark.
  3. [Section 4.3, Table 7] The claim that TIIF-Bench results "exhibit a high degree of alignment with human preferences across all dimensions" is quantitatively underdetermined by the evidence presented. For dimensions such as Attribute-short (rho=0.81) and Style-short (rho=0.81), the correlation is not distinguishable from moderate or weak alignment given the tiny sample, and the user-study rankings are averaged over only three prompts. The authors should either provide a substantially larger human study or soften the headline claim to a preliminary indication of alignment.
minor comments (5)
  1. [Section 3.1, Figure 1] The text says "As shown in Fig. 1 (right)" for semantic redundancy, but the right panel of Figure 1 shows prompt length; the redundancy discussion refers to the left panel. Please correct the cross-reference.
  2. [Section 3.3, Figure 6] The figure caption refers to the "GEND metric" but the metric is named GNED elsewhere in the paper; please unify the spelling.
  3. [Table 7] The table header says "VLM Eval/PNED" even though the text states that GNED is used for text rendering. Please clarify the relationship between PNED and GNED in this table.
  4. [Section 4.1.5] "achieves" is misspelled as "achiees" in the discussion of Flux.1 Pro versus Flux.1 Dev.
  5. [Figure 8 caption] The caption says the evaluation set is from COMPBENCH++, but Section 4.3 describes the user study as measuring TIIF-Bench alignment. Please clarify which benchmark the illustrated set belongs to.

Circularity Check

1 steps flagged · score 7.0 of 10

GPT-4o writes the prompts, writes the yes/no questions, sets the ground-truth answers, and serves as the primary scorer, so TIIF-Bench scores primarily measure GPT-4o's self-consistency rather than an externally anchored instruction-following ground truth; the small user study does not break the loop.

  1. self definitional [Section 3.3 (Evaluation Method), Eq. (1); Appendix A3.1 (Tab. 9)]
    "For each concept c_i, we employ an LLM (e.g., GPT-4o) to generate a corresponding yes/no question q_i, resulting in a set of evaluation questions Q={q_i} and the associated ground-truth answers A={â_i}. The generated image, along with the questions Q, is then input into a VLM (e.g., GPT-4o, Qwen2.5-VL-72B), which produces N predicted answers. The final evaluation score s for the generated image is computed as: s = 1/N Σ I[a_i = â_i]."

    Here the same model (GPT-4o) creates the answer key â_i and, in the primary evaluation, supplies the predicted answers a_i. The benchmark score is therefore the agreement of GPT-4o-the-scorer with GPT-4o-the-question-writer; the 'ground-truth' label is not an independent specification of what the prompt requires. A T2I model is scored by whether GPT-4o's visual answer matches GPT-4o's textual expectation, so the measured quantity is GPT-4o's self-consistency about its own prompts. The Qwen2.5-VL-72B cross-check (Tab. 9) uses the same GPT-4o-authored answer key, so it only shows inter-VLM agreement against that key and cannot validate correctness.

full rationale

TIIF-Bench's prompt construction is not itself circular: the prompts, length variants, GNED metric, and comparisons against CompBench++/GenAI-Bench involve external benchmarks and are independently checkable. No load-bearing self-citation or imported uniqueness theorem appears; the authors' prior works are cited only as related model papers. The central circularity is in the evaluation protocol: GPT-4o generates the yes/no questions and the ground-truth answers (Sec. 3.3), and GPT-4o is the primary VLM that produces the predicted answers scored against that same key. Thus the reported per-model scores are agreement rates between two GPT-4o behaviors (question writer/answer setter and visual answerer), not between an externally specified correct answer and a model's output. The Qwen2.5-VL-72B cross-check (Appendix A3.1) uses the same GPT-4o-authored answer key and therefore only demonstrates inter-judge ranking consistency, not correctness of the key. The user study (Sec. 4.3) samples 3 prompts per dimension and 8 models per prompt; with n=8 ranks, Spearman ρ values from 0.81 to 1.00 have wide confidence intervals, and the study does not score per-question accuracy, so it cannot independently validate the generated ground truth. These facts make the headline claim of 'high degree of alignment with human preferences across all dimensions' underdetermined by the evidence and make the benchmark's core metric self-referential. The circularity is methodological and load-bearing: the benchmark's central scores reduce, by construction, to agreement with GPT-4o's own answer key.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central evaluation loop is built on the assumption that GPT-4o's self-generated questions and answers constitute valid ground truth. No new physical entities are introduced. The only hand-chosen parameter is the 0.85 deduplication threshold, which is peripheral to the main scoring.

free parameters (1)
  • semantic deduplication cosine threshold = 0.85
    Used in Section 3.1 to determine prompt uniqueness; hand-chosen, though it only affects the diversity statistics, not the central evaluation scores.
assumptions (3)
  • domain assumption GPT-4o-generated yes/no questions and their ground-truth answers are correct and unbiased
    Section 3.3 relies on LLM-generated questions and answers as the benchmark's ground truth; there is no independent human verification of these questions.
  • domain assumption VLM yes/no answers reliably reflect true image-instruction alignment
    The evaluation score in Section 3.3 is the agreement rate between the evaluator VLM's answers and the LLM-generated ground truth; this assumes the VLM answers are accurate.
  • domain assumption Human rankings in the user study reflect instruction-following ability
    Section 4.3 asks participants to consider both instruction following and aesthetic quality, so the human signal conflates the two.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TIIF-Bench: How Does Your T2I Model Follow Your Instructions?." pith.science (2026). https://pith.science/paper/AHPI6N2T

@misc{pith2026250602161,
  author       = {Pith},
  title        = {Pith review of: TIIF-Bench: How Does Your T2I Model Follow Your Instructions?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AHPI6N2T}},
  note         = {Machine review of arXiv:2506.02161}
}
read the original abstract

The rapid advancements of Text-to-Image (T2I) models have ushered in a new phase of AI-generated content, marked by their growing ability to interpret and follow user instructions. However, existing T2I model evaluation benchmarks fall short in limited prompt diversity and complexity, as well as coarse evaluation metrics, making it difficult to evaluate the fine-grained alignment performance between textual instructions and generated images. In this paper, we present TIIF-Bench Text-to-Image Instruction Following Benchmark), aiming to systematically assess T2I models' ability in interpreting and following intricate textual instructions. TIIF-Bench comprises 5,000 prompts organized along multiple dimensions and categorized into three levels of difficulty and complexity. To rigorously evaluate robustness to prompt length, each prompt is provided in both short and long versions with identical core semantics. We further propose a novel Global Normalized Edit Distance (GNED) metric for text rendering and provide aspect-ratio-diverse reference images for each prompt to assess style control. In addition, we collect 100 high-quality designer-level prompts covering diverse scenarios for comprehensive evaluation. To enable scalable and fine-grained evaluation, we explore the best paradigm for leveraging the world knowledge encoded in large Vision-Language Models (VLMs) as automated binary evaluators. Through extensive ablations, we develop a fully reproducible evaluator that provides interpretable reasoning and reliable verification, enabling our benchmark to discern subtle variations in T2I model outputs. Through comprehensive benchmarking of mainstream T2I models on TIIF-Bench, we analyze the strengths and weaknesses of current T2I systems and reveal the limitations of existing evaluation benchmarks. Project Page: https://a113n-w3i.github.io/TIIF_Bench/.

Figures

Figures reproduced from arXiv: 2506.02161 by the authors.

Figure 1
Figure 1. Prompt diversity/complexity of TIIF-Bench compared to prior benchmarks. (Left) Semantic uniqueness after de-duplication using a cosine similarity threshold of 0.85: less than 30% and 60% prompts in COMPBENCH++ and GENAI BENCH are unique, while more than 90% prompts in TIIF-Bench are unique. (Middle) t-SNE visualization of CLIP text embeddings shows that TIIF-Bench spans a much broader semantic space than existing be… view at source ↗
Figure 2
Figure 2. Prompt-length sensitivity on the Numeracy attribute (“more birds than fish”) from TIIF Bench. Short prompt: “The birds are more numerous than the fish.” Long prompt: “The birds, with their feathers catching the gentle light of dawn, vastly outnumber their aquatic counterparts, the fish, which glide silently beneath the rippling surface of the water, their sleek forms moving like shadows in the depths below.” We obse… view at source ↗
Figure 3
Figure 3. Illustration of the limitations of expert scorers widely used in T2I benchmarks such as [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Failure cases of current T2I evaluation methods under coarse prompt queries. Since the [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: (Top): Prompt-construction and evaluation pipeline of TIIF-Bench. The example depicts a “Color + 3D Perspective” pairing, one of the 36 distinct attribute-pool combinations defined in our framework. (Bottom): In addition to systematically reusing prompts from existing …
Figure 6
Figure 6. Figure 6: Visualization examples for Recall and GNED. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Although AR-based models typically produce images with lower visual fidelity, their [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Illustration of the user study procedure. The figure shows one evaluation set from a specific [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Failure cases of closed-source models on prompts involving spatial relations. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Style Control evaluates a T2I model’s ability to manage global image quality, content comprehension, and overall aesthetic coherence. We provide illustrative examples for eight represen￾tative T2I models on this dimension. DALLE3 Prompt: A picture of a charming illust…
Figure 11
Figure 11. Figure 11: Text Rendering is introduced as a novel evaluation dimension for assessing a model’s ability to generate complex, non-natural textures—embedded human language text. We present illustrative examples for eight representative T2I models on this dimension. 21 [PITH_FULL_…
Figure 12
Figure 12. Figure 12: Designer-level prompts comprise the densest and most diverse set of requirements, offering the most comprehensive test of a model’s instruction-following capabilities. We provide illustrative examples showing how eight representative T2I models perform on this dimensi…

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DynEval: Holistic Evaluations of T2I Generative Models in the Wild

    cs.CV 2026-07 conditional novelty 6.5 of 10

    DynEval distills a 235B teacher VLM into 2B/4B evaluators via 250K synthetic instruction triplets, yielding higher human correlation than existing T2I metrics while enabling open-set dynamic QA and scene-graph quality checks.

  2. Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer

    cs.CV 2025-11 unverdicted novelty 6.0 of 10

    Z-Image is an efficient 6B-parameter foundation model for image generation that rivals larger commercial systems in photorealism and bilingual text rendering through a new single-stream diffusion transformer and strea...

  3. Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generation

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A curated GPT-4o synthetic image dataset improves open-source generation models on instruction-following, surreal scenes, and multi-reference synthesis, plus two new benchmarks to measure those skills.

  4. Mage-Flow: An Efficient Native-Resolution Foundation Model for Image Generation and Editing

    cs.CV 2026-07 conditional novelty 5.0 of 10

    A compact 4B image generation/editing system with a fast one-step VAE, native-resolution packing, RL alignment, and 4-step distillation reports competitive benchmarks against 6B–80B open models.

Reference graph

Works this paper leans on

47 extracted references · 38 canonical work pages · cited by 4 Pith papers

  1. [1]

    High-Resolution Image Synthesis With Latent Diffusion Models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-Resolution Image Synthesis With Latent Diffusion Models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022

    work page 2022

  2. [2]

    Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. InForty-First International Conference on Machine Learning, June 2024

    work page 2024

  3. [3]

    PixArt- Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation, March 2024

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. PixArt- Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation, March 2024

    work page 2024

  4. [4]

    PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models, January 2024

    Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models, January 2024

    work page 2024

  5. [5]

    PixArt-$α$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis, December 2023

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. PixArt-$α$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis, December 2023

    work page 2023

  6. [6]

    Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation, February 2024

    Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation, February 2024

    work page 2024

  7. [7]

    Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Models, October 2024

    Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Chase Lambert, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Models, October 2024

    work page 2024

  8. [8]

    Flux.https://github.com/black-forest-labs/flux, 2024

    Black Forest Labs. Flux.https://github.com/black-forest-labs/flux, 2024

    2024

Show all 47 references
  1. [9]

    SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillation, March 2025

    Junsong Chen, Shuchen Xue, Yuyang Zhao, Jincheng Yu, Sayak Paul, Junyu Chen, Han Cai, Enze Xie, and Song Han. SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillation, March 2025

    2025

  2. [10]

    SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer, March 2025

    Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Chengyue Wu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, Han Cai, Bingchen Liu, Daquan Zhou, and Song Han. SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer,...

    2025

  3. [11]

    SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers, October 2024

    Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, and Song Han. SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers, October 2024

    2024

  4. [12]

    Improving Image Generation with Better Captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh. Improving Image Generation with Better Captions

  5. [13]

    Imagen 3, December 2024

    Imagen-Team-Google, Jason Baldridge, Jakob Bauer, Mukul Bhutani, and Others. Imagen 3, December 2024. 15

    2024

  6. [14]

    Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT, June 2024

    Le Zhuo, Ruoyi Du, Han Xiao, Yangguang Li, Dongyang Liu, Rongjie Huang, Wenze Liu, Lirui Zhao, Fu-Yun Wang, Zhanyu Ma, Xu Luo, Zehan Wang, Kaipeng Zhang, Xiangyang Zhu, Si Liu, Xiangyu Yue, Dingning Liu, Wanli Ouyang, Ziwei Liu, Yu Qiao, Hongsheng Li, and Peng Gao. Lumina-Next...

    2024

  7. [15]

    Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding, May 2024

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, Dayou Chen, Jiajun He, Jiahao Li, Wenyue Li, Chen Zhang, Rongwei Quan, Jianxiang Lu, Jiabin Huang, Xiaoyan Yuan, Xiaoxiao Zheng, Yixuan Li, Ji...

    2024

  8. [16]

    Autore- gressive Model Beats Diffusion: Llama for Scalable Image Generation, June 2024

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autore- gressive Model Beats Diffusion: Llama for Scalable Image Generation, June 2024

    2024

  9. [17]

    Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling

  10. [18]

    Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation, October 2024

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation, October 2024

    2024

  11. [19]

    Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis, 2024

    Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis, 2024

    2024

  12. [20]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction, 2024

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction, 2024

    2024

  13. [21]

    Lumina-mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal generative pretraining, 2025

    Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yi Xin, Xinyue Li, Qi Qin, Yu Qiao, Hongsheng Li, and Peng Gao. Lumina-mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal generative pretraining, 2025

    2025

  14. [22]

    T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot.arXiv preprint arXiv:2505.00703, 2025

    Dongzhi Jiang, Ziyu Guo, Renrui Zhang, Zhuofan Zong, Hao Li, Le Zhuo, Shilin Yan, Pheng-Ann Heng, and Hongsheng Li. T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot.arXiv preprint arXiv:2505.00703, 2025

    2025 arXiv

  15. [23]

    Can we generate images with cot? let’s verify and reinforce image generation step by step.arXiv preprint arXiv:2501.13926, 2025

    Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng-Ann Heng. Can we generate images with cot? let’s verify and reinforce image generation step by step.arXiv preprint arXiv:2501.13926, 2025

    2025 arXiv

  16. [24]

    Delving into rl for image generation with cot: A study on dpo vs

    Chengzhuo Tong, Ziyu Guo, Renrui Zhang, Wenyu Shan, Xinyu Wei, Zhenghao Xing, Hongsheng Li, and Pheng-Ann Heng. Delving into rl for image generation with cot: A study on dpo vs. grpo, 2025

    2025

  17. [25]

    Mavis: Mathematical visual instruction tuning with an automatic data engine, 2024

    Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Ziyu Guo, Shicheng Li, Yichi Zhang, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, Peng Gao, Chunyuan Li, and Hongsheng Li. Mavis: Mathematical visual instruction tuning with an automatic data engine, 2024

    2024

  18. [26]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  19. [27]

    Imagen 3.arXiv preprint arXiv:2408.07009, 2024

    Jason Baldridge, Jakob Bauer, Mukul Bhutani, Nicole Brichtova, Andrew Bunner, Lluis Castrejon, Kelvin Chan, Yichang Chen, Sander Dieleman, Yuqing Du, et al. Imagen 3.arXiv preprint arXiv:2408.07009, 2024

    2024

  20. [28]

    Midjourney.https://www.midjourney.com/, 2025

    Midjourney Team. Midjourney.https://www.midjourney.com/, 2025

    2025

  21. [29]

    Evaluating text-to-visual generation with image-to-text generation, 2024

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation, 2024

    2024

  22. [30]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis, 2023

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis, 2023

    2023

  23. [31]

    Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation, 2025

    Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, Jiayan Teng, Zhuoyi Yang, Wendi Zheng, Xiao Liu, Ming Ding, Xiaohan Zhang, Xiaotao Gu, Shiyu Huang, Minlie Huang, Jie Tang, and Yuxiao Dong. Visionreward: Fin...

    2025

  24. [32]

    T2i-compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation, 2025

    Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation, 2025

    2025

  25. [33]

    Geneval: An object-focused framework for evaluating text-to-image alignment, 2023

    Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment, 2023

    2023

  26. [34]

    Genai-bench: Evaluating and improving compositional text-to-visual generation, 2024

    Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, and Deva Ramanan. Genai-bench: Evaluating and improving compositional text-to-visual generation, 2024. 16

    2024

  27. [35]

    Improving image generation with better captions.Computer Science

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions.Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023

    2023

  28. [36]

    LightGen: Efficient Image Generation through Knowledge Distillation and Direct Preference Optimization, March 2025

    Xianfeng Wu, Yajing Bai, Haoze Zheng, Harold Haodong Chen, Yexin Liu, Zihao Wang, Xuran Ma, Wen-Jie Shu, Xianzu Wu, Harry Yang, and Ser-Nam Lim. LightGen: Efficient Image Generation through Knowledge Distillation and Direct Preference Optimization, March 2025

    2025

  29. [37]

    Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens, October 2024

    Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens, October 2024

    2024

  30. [38]

    CLIPScore: A Reference-free Evaluation Metric for Image Captioning, March 2022

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A Reference-free Evaluation Metric for Image Captioning, March 2022

    2022

  31. [39]

    Imagine-e: Image generation intelligence evaluation of state-of-the-art text-to-image models, 2025

    Jiayi Lei, Renrui Zhang, Xiangfei Hu, Weifeng Lin, Zhen Li, Wenjian Sun, Ruoyi Du, Le Zhuo, Zhongyu Li, Xinyue Li, Shitian Zhao, Ziyu Guo, Yiting Lu, Peng Gao, and Hongsheng Li. Imagine-e: Image generation intelligence evaluation of state-of-the-art text-to-image models, 2025

    2025

  32. [40]

    Lex-art: Rethinking text generation via scalable high-quality data synthesis, 2025

    Shitian Zhao, Qilong Wu, Xinyue Li, Bo Zhang, Ming Li, Qi Qin, Dongyang Liu, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Peng Gao, Bin Fu, and Zhen Li. Lex-art: Rethinking text generation via scalable high-quality data synthesis, 2025

    2025

  33. [41]

    Show-o: One Single Transformer to Unify Multimodal Understanding and Generation, October 2024

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One Single Transformer to Unify Multimodal Understanding and Generation, October 2024. 17 Appendix A1 Meta Prompts A1.1...

    2024

  34. [42]

    yes" or

    Answer each question on a separate line, starting with "yes" or "no", followed by a brief reason

  35. [43]

    Maintain the exact order of the questions in your answers

  36. [44]

    Provide only one answer per question

  37. [45]

    Return only the answers—no additional commentary

  38. [46]

    Each answer must be on its own line

  39. [47]

    drink",

    Ensure the number of answers matches the number of questions. A2 Additional Visualizations A2.1 Failure Cases of Strong Closed-source Models in the Relation Dimension Through extensive experiments on TIIF BENCH, we observe that most models exhibit strong instruction-following ...

    1940

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.