R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation

Jiaxin Zhang; Joy Rimchala; Kaijie Chen; Lifu Huang; Ying Shen; Yuguang Yao; Zhiyang Xu; Zihao Lin

REVIEW 4 major objections 5 minor 50 references

R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read R2I-Bench shows that even the strongest open text-to-image models cannot reliably turn reasoning prompts into correct images, with mathematical reasoning scores near zero across the board.

desk verdict A genuinely useful reasoning-T2I benchmark whose main finding is plausible but whose numeric rankings rest on a thinly validated GPT-4o judge; deserves review, and the path to acceptance is artifact release plus stronger metric validation. read the letter →

arxiv 2505.23493 v1 pith:4TYWJA3X submitted 2025-05-29 cs.CV cs.CL

Kaijie Chen , Zihao Lin , Zhiyang Xu , Ying Shen , Yuguang Yao , Joy Rimchala , Jiaxin Zhang , Lifu Huang This is my paper

classification cs.CVcs.CL

keywords text-to-imagegenerationreasoningbenchmarkR2I-Scoremathematicalvision-languageevaluationlogicalcompositionalconceptmixing

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that current text-to-image models, even the best open ones, cannot handle prompts that require genuine reasoning. It builds a 3,068-prompt benchmark spanning seven reasoning categories and a QA-style metric, R2I-Score, that asks a vision-language model to answer hand-written, instance-specific questions about the generated image. Across 16 models, every open-source model scores below 0.45 overall, mathematical reasoning sits near zero (0.07 to 0.19), and chain-of-thought or reinforcement-learned variants improve only marginally. Only the closed-source gpt-image-1 exceeds 0.7, and even a pipeline that rewrites prompts with GPT-4o before generating still fails on compositional, numerical, and mathematical cases. If these measurements hold, reasoning is the main bottleneck separating current text-to-image systems from human-like prompt understanding.

What carries the argument

The load-bearing object is R2I-Score, a QA-style metric. For each of the 3,068 prompts, the authors generate and human-validate a set of instance-specific evaluation questions plus scoring rubrics, grouped into three dimensions: text-image alignment, reasoning accuracy, and image quality. A vision-language model (GPT-4o) answers each question for the generated image, and the scores are aggregated as the weighted average $\text{R2I-Score} = \frac{\sum_i w_i s_i}{\sum_i w_i}$, with weights deliberately set so reasoning questions (7 to 10) dominate alignment (4 to 6) and quality (1 to 3). The benchmark also carries an instance-specific reference caption and explanation description, which are used to derive the evaluation questions and to make failures interpretable.

What would settle it

Recruit independent human annotators to rank image pairs from all 16 evaluated models rather than only Lumina-Image 2.0 and Sana-1.5, and compare the resulting model order to the R2I-Score order; material disagreement would show the benchmark's headline numbers are an artifact of the GPT-4o judge, not a property of the models.

Watch

Extended reading notes

Core claim

The paper's central claim is that reasoning-driven text-to-image generation is a distinct and largely unmet capability: models that render explicit descriptions well tend to treat reasoning prompts as bags of words, rendering objects literally and ignoring logical connectives, quantifiers, conditionals, or quantities. The benchmark documents this failure mode across commonsense, compositional, logical, mathematical, causal, numerical, and concept-mixing prompts, with reasoning errors accounting for more than 80 percent of failures in the error analysis. The paper further claims that R2I-Score, built on per-instance evaluation questions with fixed weight ranges (reasoning 7 to 10, alignment 4 to 6, quality 1 to 3), agrees with human judgments better than existing alignment metrics, and that decoupling reasoning (GPT-4o) from rendering (SD3-medium) helps commonsense and causal categories but barely moves mathematical reasoning.

Load-bearing premise

The benchmark's numbers stand or fall on whether GPT-4o, answering hand-written instance-specific questions, judges reasoning in generated images the way people would; the human validation covers only two similar models.

Editorial extensions

If this is right

Open-source text-to-image models, whatever their visual quality, are not yet reliable at following prompts that require inference: all scored below 0.45 on R2I-Score.
Mathematical prompts are the weakest point, with open models scoring 0.07 to 0.19, which points to missing training data and architectural support for visual mathematical reasoning.
Chain-of-thought, outcome-reward, DPO, and PARM style reasoning enhancements give only small gains over the base Show-o model, so prompt-side reasoning is not enough to fix image-side reasoning.
A pipeline that lets GPT-4o rewrite the prompt before SD3-medium renders it improves commonsense and causal scores substantially, but helps little for compositional, numerical, and mathematical prompts.
Reasoning errors dominate the failure distribution, over 80 percent across representative models, making reasoning rather than rendering quality the primary bottleneck.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the R2I-Score ranking reflects true capability, benchmark-style per-instance questions could double as a reward signal: fine-tuning text-to-image models to maximize R2I-Score might transfer better than current outcome-reward methods.
The 'bag of words' failure pattern fits a CLIP-text-encoder account, as prompts like 'either a spoon or a bowl' are encoded as if the objects were co-present; a direct test would compare models with LLM-based text encoders against CLIP-conditioned ones on the same disjunctive prompts.
Because the scoring vision-language model is also GPT-4o, part of the closed-versus-open gap may reflect judge familiarity with proprietary outputs; a useful check is to score all models with an independent open-weight vision-language judge.
The benchmark structure, category definitions plus per-instance evaluation questions, could be adapted to video, audio, and 3D generation merely by changing the rendering target, a direction the paper itself mentions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: R2I-Bench is the broadest reasoning-driven T2I benchmark I've seen, and the core finding—open models are bad at reasoning, especially math—is probably right. But the R2I-Score numbers should not be treated as firm until the evaluation is better validated and the artifacts ship.

What's new: 3,068 prompts spanning 7 reasoning categories and 32 subcategories, with per-instance QA questions and human-in-the-loop curation. That's a real step beyond Commonsense-T2I, GeckoNum, ConceptMix, and GenEval, which each cover one slice. The pipeline baseline (GPT-4o rewrite + SD3) is a sensible upper-bound probe, and the error analysis—models rendering "either a spoon or a bowl" as both objects, math prompts producing mecha robots or abstract art—is genuinely informative. The authors also include a limitations section that names real constraints, which I appreciate.

Soft spots: R2I-Score is a GPT-4o judge answering hand-crafted questions, with reasoning questions weighted 7–10. The only human validation compares two similar open diffusion models and reports 0.713 pairwise accuracy, with no inter-annotator agreement and no repeated scoring. That is thin for a metric carrying 16-model rankings. The stress-test concern is fair: GPT-4o's known weak spot is exactly counting and precise geometry, so the near-zero math scores (0.07–0.19) could be partly a judge artifact. I don't think it flips the paper's qualitative conclusion—the failure images shown are clearly wrong—but the exact numbers and the 0.45 threshold could shift if the judge were replaced or recalibrated. Also, no code or data are in the preprint; for a benchmark paper that is a genuine problem, since the lasting value is the artifact.

Bottom line: this is a solid, useful paper that deserves a serious referee. The path to acceptance is clear: release the dataset and scoring code, expand the human validation to more models with variance and agreement stats, and ideally show that a different VLM judge produces similar rankings. The core claim will likely survive; the precision it currently claims will not.

Referee Report

4 major / 5 minor

Summary. R2I-Bench introduces a benchmark of 3,068 text-to-image prompts spanning seven reasoning categories (commonsense, compositional, logical, mathematical, causal, numerical, and concept mixing) with 32 subcategories, together with a QA-style metric, R2I-Score, that aggregates GPT-4o judgments over instance-specific human-refined evaluation questions weighted by reasoning accuracy (7–10), text-image alignment (4–6), and image quality (1–3). The paper evaluates 16 models, including diffusion, autoregressive, reasoning-enhanced, and closed-source models, plus a prompt-rewriting pipeline (GPT-4o + SD3-medium). Its central findings are that open-source models all score below 0.45 overall, mathematical reasoning scores are near zero (0.07–0.19), reasoning-enhanced variants give only marginal gains, and closed-source models, especially gpt-image-1, dominate. The paper also reports a human study comparing R2I-Score with other metrics on pairwise preferences for two open models.

Significance. If the findings hold, R2I-Bench would be a valuable resource: it is larger and broader than prior T2I reasoning benchmarks, spans 32 fine-grained subcategories, and the per-instance QA protocol makes evaluation more interpretable than a single similarity score. The paper also contributes a sizable evaluation of 16 current models, a prompt-rewriting pipeline baseline, and a qualitative error taxonomy. However, the significance is currently conditional on the validity of R2I-Score, and the evidence for that validity is thin: the human validation covers only one pair of models, no judge-variance or error-bar analysis is reported, and the scoring judge is the same model family used to draft prompts and questions. The benchmark itself and the qualitative findings are likely useful, but the quantitative headline claims need stronger metric validation.

major comments (4)

[Sec. 4.3, Table 3] The human validation of R2I-Score is too narrow to support the paper's absolute-score claims. The study compares only two models (Lumina-Image 2.0 vs. Sana-1.5) via pairwise preference, yet the abstract and Sec. 4.2 make absolute claims such as 'all open-source models achieve less than 45%' and 'mathematical reasoning remains near zero.' Pairwise accuracy on one model pair does not establish absolute calibration of the 0–1 scores, nor does it validate the 16-model ordering in Table 2. The section reports no number of annotators, number of image pairs, or inter-annotator agreement, and the 0.713 pairwise accuracy is only modestly above the 0.694 of VIEScore. Without a broader human study (e.g., per-question human scoring on a stratified sample across categories and models), the metric's reliability for the headline rankings is not established.
[Eq. (1), Sec. 3, Appendix D.1.3, Fig. 16] The scoring protocol uses GPT-4o as the judge, and GPT-4o is also used to generate the prompts, reference captions, and candidate evaluation questions. No analysis quantifies how sensitive the R2I-Score is to the choice of judge or to repeated scoring at the stated temperature of 0.1. Since reasoning questions carry weights 7–10 in Eq. (1), systematic judge errors on dense counting or geometric transformations directly inflate or deflate the overall scores. The paper itself acknowledges numerical limitations of VLMs in the Figure 16 caption ('Due to the current limitations of the best visual language models in numerical tasks, the numbers in Exact Number Generation are restricted to values within 30'), but provides no evidence that GPT-4o can reliably verify counts up to 30 or judge geometrical transformations such as rotations and scaling. This is load-bearing for the claim in Sec. 4.2 that mathematical reasoning is near zero (0.07–0.19). A concrete fix is to re-score a stratified subset with a different VLM (e.g., Gemini or Claude) and with human per-question judgments, and to report agreement and score differences.
[Table 2 and Appendix C] All model scores are reported as point estimates without confidence intervals, repeated sampling, or significance tests. This is a problem for the comparative claims in Sec. 4.2, such as the assertion that reasoning-enhanced models give only 'marginal improvements' (Show-o+PARM 0.38 vs. base Show-o 0.36) or that EMU3 and Sana-1.5 essentially tie at 0.41. Given that each score is a mean over roughly 3,068 instances with per-instance questions scored by a stochastic VLM judge, the absence of error bars makes it impossible to know whether the reported differences are meaningful. The authors should provide confidence intervals (e.g., bootstrap over instances and scoring repeats) and, where relevant, pairwise significance tests.
[Sec. 3, R2I-Score weight design] The weight ranges [7,10], [4,6], and [1,3] in Eq. (1) are motivated by the goal of emphasizing reasoning, but the paper does not test how sensitive the model ranking is to these weights. Because the weights are hand-set and the per-question weights are also generated by GPT-4o with human refinement, the final scores are partly a product of these choices. An ablation that varies the weight ranges (e.g., equal weights, or 5/3/1) would show whether the main conclusions—open models below 0.45, math near zero—are robust or artifacts of the weighting scheme. This is especially relevant given the thin human validation of the metric.

minor comments (5)

[Appendix C, Tables 13–17] The appendix tables inconsistently label the closed-source image model as 'GPT-4o' instead of 'gpt-image-1' used in the main Table 2, and the table headers contain other typos ('OminGen', 'DALLE', 'gpt-iamge-1', 'Showo-o+PARM'). This makes it hard to reconcile subcategory results with the main results and should be fixed.
[Sec. 2, Table 1] The 'Human Annotation' column in Table 1 is ambiguous: R2I-Bench involves human annotation of prompts and evaluation criteria, but not human annotation of model outputs. The column heading should be clarified to avoid implying that the benchmark itself is annotated with ground-truth images.
[Sec. 4.4, Figure 5] The methodology for the error analysis is not described: the paper does not state who or what classified errors into reasoning, visual element, and image quality failures, nor whether any inter-annotator agreement was computed. This limits the reproducibility of the error distribution.
[Sec. 3, Evaluation Criteria Generation] The sentence beginning 'For efficient' contains a typo and should read 'For efficiency.'
[Limitations section] The Limitations section discusses generality of the evaluation method but does not explicitly mention the metric's dependence on a single VLM judge or the absence of judge-variance analysis; this should be acknowledged in the main text rather than only in the Figure 16 caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; R2I-Score is an external GPT-4o judge with human-refined criteria, not a fitted or self-defined target.

full rationale

The paper is a benchmark and evaluation study rather than a derived theoretical result, and I found no circular step that reduces a prediction to its inputs by construction. R2I-Score is defined in Eq. (1) as a weighted average of per-question scores si assigned by GPT-4o according to evaluation criteria that were generated by GPT-4o but then manually validated and refined by human experts. The overall accuracy numbers (e.g., open-source models below 0.45, mathematical reasoning near zero) are empirical measurements produced by applying this fixed scoring protocol to generated images, not quantities fitted from the data and then renamed as predictions. The weight ranges for reasoning, alignment, and quality dimensions (7-10, 4-6, 1-3) are stated design choices, not fitted parameters. There is no load-bearing self-citation: the cited prior benchmarks and methods are external works, and the paper does not invoke a uniqueness theorem from the authors' own prior results. The most plausible circularity concern is evaluator-related: GPT-4o drafts the evaluation questions/criteria and also serves as the judge, and the pipeline-based framework uses GPT-4o as a prompt rewriter. This overlap could bias the metric or favor certain outputs, and the human validation in Table 3 covers only two models, which is a legitimate validity and robustness weakness. However, the criteria are human-refined and the reference captions are human-validated, so the measurement is not defined solely as GPT-4o's self-agreement. The paper's Limitations section explicitly acknowledges that the evaluation method is constrained by the specific benchmark and that the granularity is coarse, further indicating an acknowledged scope limitation rather than a hidden circular derivation. Therefore, while the metric's reliability for ranking 16 models deserves scrutiny, the paper's central quantitative claims are not circular in the sense of being equivalent to their inputs by construction.

Assumptions & free parameters 3 free parameters · 3 assumptions · 2 invented entities

The paper's central claim relies on the benchmark dataset, the R2I-Score metric, and the specific evaluation protocol. The metric weight scheme is the main free choice; the human alignment study is the main supporting evidence. No entities are invented to explain phenomena, so the ledger is short.

free parameters (3)

R2I-Score question weights = weights 7-10, 4-6, 1-3 per dimension, assigned by GPT-4o and human-refined
The weighted aggregation is the core of the new metric; the particular weights are set by prompt instruction and human validation, not derived from first principles, and they directly affect all reported model scores.
Number of instances per subcategory = approximately 100 per subcategory, 3,068 total
The dataset balance across categories and subcategories is the result of a filtering and augmentation process; the chosen balance affects aggregate scores.
GPT-4o temperature 0.1 for scoring = 0.1
The VLM scoring protocol uses temperature 0.1 to ensure stability; this is an implementation choice that influences all numeric results and is not reported with variance.

assumptions (3)

domain assumption GPT-4o can reliably generate and validate reasoning prompts with human expert supervision
Data collection and evaluation criteria generation depend on GPT-4o in the loop; the paper asserts expert validation but does not quantify inter-annotator agreement or measure GPT-4o's own error rate.
domain assumption Text-to-image models are evaluated in a zero-shot setting without inference-time reasoning
The evaluation protocol fixes a zero-shot setting (Appendix B), which may understate what models can do with CoT or few-shot prompting; the plot choice is a valid design decision but limits the generality of the 'limited reasoning' conclusion.
domain assumption R2I-Score is a faithful proxy for human judgment
The metric is validated on only two models, and the VLM judging may itself fail on the same reasoning challenges the benchmark targets; the paper even shows VQAScore failures on similar tasks.

invented entities (2)

R2I-Score
purpose: QA-style metric for evaluating reasoning-driven T2I generation across three dimensions
The metric is a new evaluation procedure; it has no independent falsifiable handle outside the paper, and its validity rests entirely on the human-alignment study reported here.
R2I-Bench dataset independent evidence
purpose: Broad reasoning-driven T2I benchmark with 3,068 instances
The dataset is a public artifact (with a project page) that other labs can use to measure models, providing an independent falsifiable handle: any model's score on R2I-Bench can be checked by others.

how reviews work

0 comments

Cite this review

Pith. "Pith review of R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation." pith.science (2026). https://pith.science/paper/4TYWJA3X

@misc{pith2026250523493,
  author       = {Pith},
  title        = {Pith review of: R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4TYWJA3X}},
  note         = {Machine review of arXiv:2505.23493}
}

read the original abstract

Reasoning is a fundamental capability often required in real-world text-to-image (T2I) generation, e.g., generating ``a bitten apple that has been left in the air for more than a week`` necessitates understanding temporal decay and commonsense concepts. While recent T2I models have made impressive progress in producing photorealistic images, their reasoning capability remains underdeveloped and insufficiently evaluated. To bridge this gap, we introduce R2I-Bench, a comprehensive benchmark specifically designed to rigorously assess reasoning-driven T2I generation. R2I-Bench comprises meticulously curated data instances, spanning core reasoning categories, including commonsense, mathematical, logical, compositional, numerical, causal, and concept mixing. To facilitate fine-grained evaluation, we design R2IScore, a QA-style metric based on instance-specific, reasoning-oriented evaluation questions that assess three critical dimensions: text-image alignment, reasoning accuracy, and image quality. Extensive experiments with 16 representative T2I models, including a strong pipeline-based framework that decouples reasoning and generation using the state-of-the-art language and image generation models, demonstrate consistently limited reasoning performance, highlighting the need for more robust, reasoning-aware architectures in the next generation of T2I systems. Project Page: https://r2i-bench.github.io

Figures

Figures reproduced from arXiv: 2505.23493 by the authors.

**Figure 1.** We introduce R2I-Bench, a comprehensive benchmark designed to assess the reasoning capabilities of text-to-image (T2I) generation models. It encompasses 7 primary reasoning categories, which are further subdivided into 32 fine-grained subcategories. Abstract Reasoning is a fundamental capability often required in real-world text-to-image (T2I) generation, e.g., generating “a bitten apple that has been left in the a… view at source ↗

**Figure 2.** Example Illustration of R2I-Bench and R2I-Score. T2I models. (2) To enable fine-grained evaluation of reasoning-driven T2I generation, we design a new metric, R2I-Score, built on human-validated evaluation questions and scoring criteria tailored to each data instance in R2I-Bench. R2I-Score assesses model performance across three critical dimensions, including text-image alignment, reasoning accuracy, and image qu… view at source ↗

**Figure 3.** [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (15 more)

**Figure 4.** Figure 4: Failure Cases of the Pipeline-based Framework on Compositional/Numerical/Mathematical Reasoning. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Distribution of Errors of Emu3, SD3-medium, Show-o+PARM, gpt-image-1. 4.4 Error Analysis To better understand the limitations of current T2I models, we categorize and accordingly define three failure types: basic element errors, reasoning errors, and visual quality i…

**Figure 6.** Figure 6: Benchmark Curation Pipeline. The pipeline starts with data collection, followed by data filtering, evaluation criteria generation, and ultimately results in R2I-Bench. To ensure data quality, human verification is performed at each key stage to eliminate low-quality da…

**Figure 7.** Figure 7: Failure Cases of VQAScore [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

**Figure 8.** Figure 8: Examples of Seven Subfields in Commonsense Reasoning, spanning Affordance, Attribute, Color, Emotion Intention Commonsense, Social Cultural Knowledge Object and Scene and Temporal Understanding. We showcase the Text-lite version [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

**Figure 9.** Figure 9: Examples of Seven Subfields in Logical Reasoning, spanning Abductive, Categorical, conjunctive, Deductive, Hypothetical, Sufficient Conditional [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

**Figure 10.** Figure 10: Examples of Eight Subfields in Mathematical Reasoning, spanning Combinatorial, Crypographic Encoding, Geometrical Transformations, Mathematical Function,spatial reasoning,et Theory, Spatial Reasoning and Vector Matrix Visualizations [PITH_FULL_IMAGE:figures/full_fig_…

**Figure 11.** Figure 11: Examples of Two Subfields in Concept Mixing, including Functional Mixing and Literal Mixing [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

**Figure 12.** Figure 12: Examples of Three Subfields in Compositional Reasoning, including Creative Compositional, Inferential Spatial, Color, Prescriptive Spatial. Prompt Generate an image to explain the cause of following scene: a trail of spilled milk leading to an empty glass on the kitch…

**Figure 13.** Figure 13: Examples of two Subfields in Causal Reasoning, including Cause to Effect Reasoning and Cause to Effect Reasoning [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]

**Figure 14.** Figure 14: Examples of Three Subfields in Numerical Reasoning, including Approximate Number Generation, Conceptual Quantitative, Exact Number Generation. Disjunctive 49.51% Conjunctive 50.49% Disjunctive &Conjunctive Approximate Quantities 45.03% Comparative Relationships 36.26%…

**Figure 15.** Figure 15: Distribution of Quantifiers and Operations in Categorical, Approximate Number Generation, Disjunctive Reasoning, and Conjunctive Reasoning [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]

**Figure 16.** Figure 16: Distribution of Numbers in Exact Number Generation and Conceptual Quantitative Reasoning. Due to the current limitations of the best visual language models in numerical tasks, the numbers in Exact Number Generation are restricted to values within 30 [PITH_FULL_IMAGE:…

**Figure 20.** Figure 20: Prompt for evaluation criteria generation [PITH_FULL_IMAGE:figures/full_fig_p023_20.png]

**Figure 21.** Figure 21: Detailed Performance Comparison: Standard T2I Model vs. Pipeline-based Framework. We denote the results of standard T2I models in blue pillars and highlight the increase and decrease magnitude with the pipeline-based framework by green and red colors, respectively […

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 13 canonical work pages

[1]

Liang Chen, Sinan Tan, Zefan Cai, Weichu Xie, Haozhe Zhao, Yichi Zhang, Junyang Lin, Jinze Bai, Tianyu Liu, and Baobao Chang. 2024. A spark of vision-language intelligence: 2-dimensional autoregressive transformer for efficient finegrained image generation. In The Thirteenth International Conference on Learning Representations

work page 2024
[2]

Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. 2025. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811

arXiv 2025
[3]

Mathieu Chevalley, Yusuf Roohani, Arash Mehrjou, Jure Leskovec, and Patrick Schwab. 2022. Causalbench: A large-scale benchmark for network inference from single-cell perturbation data. arXiv preprint arXiv:2210.17283

work page Pith review arXiv 2022
[4]

Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. 2023. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. arXiv preprint arXiv:2310.18235

arXiv 2023
[5]

Daniel Deutsch, George Foster, and Markus Freitag. 2023. Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. arXiv preprint arXiv:2305.14324

arXiv 2023
[6]

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M \"u ller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. 2024 a . Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning

work page 2024
[7]

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. 2024 b . http://arxiv.org/abs/2403.03206 Scaling rectified flow transformers for high-resolution image synthesis

arXiv 2024
[8]

Xingyu Fu, Muyu He, Yujie Lu, William Yang Wang, and Dan Roth. 2024. Commonsense-t2i challenge: Can text-to-image generation models understand commonsense? arXiv preprint arXiv:2406.07546

arXiv 2024

Show all 50 references

[9]

Peng Gao, Le Zhuo, Dongyang Liu, Ruoyi Du, Xu Luo, Longtian Qiu, Yuhang Zhang, Chen Lin, Rongjie Huang, Shijie Geng, et al. 2024. Lumina-t2x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arXiv:2405.05945

2024 arXiv
[10]

Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. 2023. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132--52152

2023
[11]

Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng-Ann Heng. 2025. Can we generate images with cot? let's verify and reinforce image generation step by step. arXiv preprint arXiv:2501.13926

2025 arXiv
[12]

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718

2021 arXiv
[13]

Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. 2024. Ella: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135

2024 arXiv
[14]

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

2024 arXiv
[15]

Sneha Jadhav and Shuangge Ma. 2019. Kendall's tau for functional data analysis. arXiv preprint arXiv:1912.03725

2019 arXiv
[16]

Dongzhi Jiang, Ziyu Guo, Renrui Zhang, Zhuofan Zong, Hao Li, Le Zhuo, Shilin Yan, Pheng-Ann Heng, and Hongsheng Li. 2025. T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot. arXiv preprint arXiv:2505.00703

2025 arXiv
[17]

Ivana Kaji \'c , Olivia Wiles, Isabela Albuquerque, Matthias Bauer, Su Wang, Jordi Pont-Tuset, and Aida Nematzadeh. 2024. Evaluating numerical reasoning in text-to-image models. Advances in Neural Information Processing Systems, 37:42211--42224

2024
[19]

Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. 2024. http://arxiv.org/abs/2312.14867 Viescore: Towards explainable metrics for conditional image synthesis evaluation

2024 arXiv
[20]

Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al. 2023. Holistic evaluation of text-to-image models. Advances in Neural Information Processing Systems, 36:69981--70011

2023
[21]

Jiayi Lei, Renrui Zhang, Xiangfei Hu, Weifeng Lin, Zhen Li, Wenjian Sun, Ruoyi Du, Le Zhuo, Zhongyu Li, Xinyue Li, et al. 2025. Imagine-e: Image generation intelligence evaluation of state-of-the-art text-to-image models. arXiv preprint arXiv:2501.13920

2025 arXiv
[22]

Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, et al. 2024. Genai-bench: Evaluating and improving compositional text-to-visual generation. arXiv preprint arXiv:2406.13743

2024 arXiv
[23]

Jiaqi Liao, Zhengyuan Yang, Linjie Li, Dianqi Li, Kevin Lin, Yu Cheng, and Lijuan Wang. 2025. Imagegen-cot: Enhancing text-to-image in-context learning with chain-of-thought reasoning. arXiv preprint arXiv:2503.19312

2025 arXiv
[24]

Jun Hao Liew, Hanshu Yan, Daquan Zhou, and Jiashi Feng. 2022. Magicmix: Semantic mixing with diffusion models. arXiv preprint arXiv:2210.16056

2022 arXiv
[25]

Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. 2024. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pages 366--384. Springer

2024
[26]

Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124

2020 arXiv
[27]

Mushui Liu, Yuhang Ma, Zhen Yang, Jun Dan, Yunlong Yu, Zeng Zhao, Zhipeng Hu, Bai Liu, and Changjie Fan. 2025. Llm4gen: Leveraging semantic representation of llms for text-to-image generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 5...

2025
[28]

Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, and William Yang Wang. 2023. Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation. Advances in Neural Information Processing Systems, 36:23075--23093

2023
[29]

Shuailei Ma, Kecheng Zheng, Ying Wei, Wei Wu, Fan Lu, Yifei Zhang, Chen-Wei Xie, Biao Gong, Jiapeng Zhu, and Yujun Shen. 2024. Learning visual generative priors without text. arXiv preprint arXiv:2412.07767

2024 arXiv
[30]

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195--3204

2019
[31]

Yuwei Niu, Munan Ning, Mengren Zheng, Bin Lin, Peng Jin, Jiaqi Liao, Kunpeng Ning, Bin Zhu, and Li Yuan. 2025. Wise: A world knowledge-informed semantic evaluation for text-to-image generation. arXiv preprint arXiv:2503.07265

2025 arXiv
[33]

Qi Qin, Le Zhuo, Yi Xin, Ruoyi Du, Zhen Li, Bin Fu, Yiting Lu, Jiakang Yuan, Xinyue Li, Dongyang Liu, et al. 2025 b . Lumina-image 2.0: A unified and efficient image generative framework. arXiv preprint arXiv:2503.21758

2025 arXiv
[34]

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. 2022. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684--10695

2022
[35]

Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. 2024 a . Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525

2024 arXiv
[36]

Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. 2024 b . http://arxiv.org/abs/2312.13286 Generative multimodal models are in-context learners

2024 arXiv
[37]

Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. 2023. Emu: Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222

2023 arXiv
[38]

Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

2022
[39]

Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. 2024. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412.14164

2024 arXiv
[40]

Shengxin Tu, Chun Li, and Bryan E Shepherd. 2025. Between-and within-cluster spearman rank correlations. Statistics in Medicine, 44(3-4):e10326

2025
[41]

Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. 2024. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869

2024 arXiv
[42]

Xindi Wu, Dingli Yu, Yangsibo Huang, Olga Russakovsky, and Sanjeev Arora. 2024. Conceptmix: A compositional image generation benchmark with controllable difficulty. arXiv preprint arXiv:2408.14339

2024 arXiv
[43]

Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Chaofan Li, Shuting Wang, Tiejun Huang, and Zheng Liu. 2024. Omnigen: Unified image generation. arXiv preprint arXiv:2409.11340

2024 arXiv
[45]

Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Chengyue Wu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, et al. 2025 b . Sana 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer. arXiv preprint arXiv:2501.18427

2025 arXiv
[46]

Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. 2024. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528

2024 arXiv
[47]

Ling Yang, Jingwei Liu, Shenda Hong, Zhilong Zhang, Zhilin Huang, Zheming Cai, Wentao Zhang, and Bin Cui. 2024. http://arxiv.org/abs/2401.02015 Improving diffusion-based image synthesis with context prediction

2024 arXiv
[48]

Qian Zhang, Xiangzi Dai, Ninghua Yang, Xiang An, Ziyong Feng, and Xingyu Ren. 2024. Var-clip: Text-to-image generator with visual auto-regressive modeling. arXiv preprint arXiv:2408.01181

2024 arXiv
[49]

Xiangyu Zhao, Peiyuan Zhang, Kexian Tang, Hao Li, Zicheng Zhang, Guangtao Zhai, Junchi Yan, Hua Yang, Xue Yang, and Haodong Duan. 2025. Envisioning beyond the pixels: Benchmarking reasoning-informed visual editing. arXiv preprint arXiv:2504.02826

2025 arXiv
[50]

Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. 2024. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039

2024 arXiv
[51]

Xiangru Zhu, Penglei Sun, Yaoxian Song, Yanghua Xiao, Zhixu Li, Chengyu Wang, Jun Huang, Bei Yang, and Xiaoxiao Xu. 2024. Evaluating semantic variation in text-to-image synthesis: A causal perspective. arXiv preprint arXiv:2410.10291

2024 arXiv
[52]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
[53]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Liang Chen, Sinan Tan, Zefan Cai, Weichu Xie, Haozhe Zhao, Yichi Zhang, Junyang Lin, Jinze Bai, Tianyu Liu, and Baobao Chang. 2024. A spark of vision-language intelligence: 2-dimensional autoregressive transformer for efficient finegrained image generation. In The Thirteenth International Conference on Learning Representations

work page 2024

[2] [2]

Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. 2025. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811

arXiv 2025

[3] [3]

Mathieu Chevalley, Yusuf Roohani, Arash Mehrjou, Jure Leskovec, and Patrick Schwab. 2022. Causalbench: A large-scale benchmark for network inference from single-cell perturbation data. arXiv preprint arXiv:2210.17283

work page Pith review arXiv 2022

[4] [4]

Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. 2023. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. arXiv preprint arXiv:2310.18235

arXiv 2023

[5] [5]

Daniel Deutsch, George Foster, and Markus Freitag. 2023. Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. arXiv preprint arXiv:2305.14324

arXiv 2023

[6] [6]

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M \"u ller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. 2024 a . Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning

work page 2024

[7] [7]

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. 2024 b . http://arxiv.org/abs/2403.03206 Scaling rectified flow transformers for high-resolution image synthesis

arXiv 2024

[8] [8]

Xingyu Fu, Muyu He, Yujie Lu, William Yang Wang, and Dan Roth. 2024. Commonsense-t2i challenge: Can text-to-image generation models understand commonsense? arXiv preprint arXiv:2406.07546

arXiv 2024