Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

EvalGIM: A Library for Evaluating Generative Image Models

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EvalGIM unifies text-to-image benchmarks and exposes ranking blind spots

desk verdict A genuinely useful evaluation library whose cross-dataset Prompt Types exercise overstates what equal-size subsampling buys you; worth publishing with a fix. read the letter →

arxiv 2412.10604 v2 pith:KNSUACMF submitted 2024-12-13 cs.CV

Melissa Hall , Oscar Mañas , Reyhane Askari-Hemmat , Mark Ibrahim , Candace Ross , Pietro Astolfi , Tariq Berrada Ifriqi , Marton Havasi
show 9 more authors
Yohann Benchetrit Karen Ullrich Carolina Braga Abhishek Charnalia Maeve Ryan Mike Rabbat Michal Drozdzal Jakob Verbeek Adriana Romero-Soriano
This is my paper · ORCID
classification cs.CV
keywords text-to-imageevaluationgenerativeimagemodelslibraryFIDconsistencymetricsrankingrobustnessprompttypesgroupdisparities
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces EvalGIM, a library for evaluating text-to-image generative models that brings together many datasets, metrics, and visualizations under one interface. Its core aim is to make evaluation 'actionable' by packaging analyses into Evaluation Exercises, two of which are new: Ranking Robustness, which checks whether model rankings hold across metrics and datasets, and Prompt Types, which compares models on balanced prompt distributions. Applied as demonstrations, these exercises yield concrete insights: consistency plateaus near 450,000 training iterations while quality and diversity fluctuate, FID can hide whether a model's edge is quality or diversity, and mixing original with re-captioned training data improves both diversity and consistency. The paper argues that such a library lets researchers move beyond scattered metric tables to interpretable conclusions.

What carries the argument

The central object is the EvalGIM library itself, built around a plug-and-play architecture of dataset classes, metric wrappers, and visualization scripts, topped by four Evaluation Exercises. Each exercise bundles a specific evaluation question with chosen datasets, metrics, and visualizations: Trade-offs uses Pareto Fronts on COCO with precision, coverage, and VQAScore; Group Representation uses radar plots on GeoDE with precision, coverage, and CLIPScore; Ranking Robustness builds a color-coded ranking table across ImageNet, COCO, GeoDE, and CC12M; and Prompt Types uses balanced scatterplots across subsampled datasets. The library's design—inheritable dataset interfaces, metric update/compute functions, and reusable visualization code—is what lets new datasets and metrics be added without rewriting the evaluation pipeline.

What would settle it

A study that presents the same set of model pairs to both the library's metrics and human raters (for example, ranking LDM-1.5 versus LDM-3 on consistency with VQAScore and with human preference) would show whether the exercise's conclusions track human judgment; if automatic and human rankings diverge on a substantial sample, the library's 'actionable takeaways' would need human auditing.

Watch

Extended reading notes

Core claim

The central claim is that EvalGIM closes a gap in text-to-image evaluation by unifying datasets, metrics, and visualizations and by translating them into repeatable Evaluation Exercises. The exercises reproduce two existing analysis methods—Pareto-front trade-off analysis and disaggregated geographic group measurement—and introduce two novel ones: Ranking Robustness, which tests whether comparative claims about models are consistent across metrics and datasets, and Prompt Types, which enables fair distributional comparisons across datasets of different prompt styles by subsampling to a common size. In preliminary studies, the library's exercises produced three findings: consistency (measured by VQAScore) increases then plateaus around 450,000 training iterations; FID can rank a model worst while precision ranks it best, showing that quality and diversity are separable; and training with a mix of original and re-captioned captions improves diversity and consistency compared with either alone.

Load-bearing premise

The automatic metrics bundled in the library—FID, precision and coverage, CLIPScore, and VQAScore—are treated as faithful proxies for human-judged quality, diversity, and consistency, so conclusions like the 450,000-iteration plateau are only as strong as that proxy.

Editorial extensions

If this is right

  • Consistency, as measured by VQAScore, plateaus around 450,000 training steps, so training beyond that point should be monitored primarily through quality and diversity metrics.
  • Reporting FID alone can mislead: a model can rank worst on FID yet best on precision, so FID should be supplemented with precision and coverage to separate quality from diversity.
  • Consistency rankings are metric-dependent: LDM-1.5 appears worst by VQAScore while LDM-3 appears worst by CLIPScore, implying that conclusions about 'best' consistency require multiple metrics.
  • Training with both original and re-captioned captions improves diversity and consistency relative to using only original or only re-captioned data, supporting caption-enrichment as a training intervention.
  • Model rankings are not stable across datasets, particularly for diversity, so benchmark results should be contextualized with geographic and prompt-style variation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Ranking Robustness exercise were applied to other model families, the observed metric–dataset sensitivity suggests that single-number leaderboards systematically understate the uncertainty about which model is best.
  • The balanced-subsampling method for Prompt Types could be ported to FID and other distributional metrics, but the equal-size assumption may break when the smallest dataset is not representative of real deployment conditions.
  • The 450,000-iteration plateau is an early-training observation; extending the trade-off analysis to late training could reveal whether consistency resumes improving or eventually degrades.
  • A direct test of actionability would be whether practitioners using the Evaluation Exercises make different model choices than those reading a standard FID table; if choices do not differ, the exercises would be descriptive rather than decision-changing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces EvalGIM, an open-source library for evaluating text-to-image generative models, with support for multiple real-image and prompt datasets, marginal and conditional metrics, grouped/disaggregated evaluations, and visualization tools. The library is organized around four Evaluation Exercises: Trade-offs via consistency-diversity-realism Pareto Fronts, Group Representation via regional disaggregation, Ranking Robustness (a new analysis of whether model rankings are stable across metrics and datasets), and Prompt Types (a new analysis of balanced cross-dataset comparisons across prompt styles). The authors demonstrate each Exercise with preliminary studies, including training-progress analysis of a flow-matching model, comparison of successive latent diffusion models, ranking tables across four models and four datasets, and a re-captioning study on CC12M. The central claim is that EvalGIM provides unified, flexible, and actionable evaluations, as stated in the abstract.

Significance. If the library works as described, it is a useful community resource: it bundles several datasets and modern metrics (including VQAScore and DSG), provides inheritable interfaces for adding datasets and metrics, supports distributed evaluation, and ships reproducible notebooks and visualizations. The code is released on GitHub, which is a concrete strength. The two new Evaluation Exercises (Ranking Robustness and Prompt Types) are potentially valuable analysis patterns. However, the paper's empirical demonstrations are preliminary and currently carry two load-bearing weak points: the Prompt Types exercise claims cross-dataset comparability of distributional metrics after equal-size subsampling, which is not justified, and the ranking-based conclusions are presented without error bars or statistical tests. These issues affect the strength of the paper's actionable-insight claims, though they are fixable by reframing or adding calibration/uncertainty quantification.

major comments (3)
  1. [§3.4.1] The Prompt Types Evaluation Exercise asserts that uniformly subsampling ImageNet, GeoDE, COCO, and CC12M to roughly 15,000 images makes the datasets 'comparable due to being balanced' and uses this to compare FID, precision, and coverage across prompt styles. This is not supported: FID, precision, and coverage are defined relative to each dataset's reference feature manifold (Appendix B.1, InceptionV3, k=3), so equalizing the number of real images does not place the metrics on a common scale. A lower FID or higher precision on ImageNet than on COCO only indicates a better match to that particular dataset's reference distribution, not better generation quality in an absolute sense. Consequently, the cross-dataset findings in §3.4.2 (e.g., re-captioning yields 'stronger performance improvements' on ImageNet and re-captioned CC12M, and excluding original captions has a 'larger negative effect' on GeoDE and ImageNet) compare incomparable quantities. The library remains valid for within-dataset comparisons, but the Promise of 'balanced evaluations across different prompt styles' requires either a calibration mechanism, a common reference distribution, or an explicit reframing that the metrics measure per-dataset distributional match rather than cross-dataset quality.
  2. [§3.3.2] The Ranking Robustness Exercise reports model rankings (e.g., 'LDM-3 ranks worst on FID across all datasets but highest for quality', and 'LDM-1.5 shows the worst consistency when evaluated with VQAScore while LDM-3 is worst when using CLIPScore') without specifying the inference settings used for each model (guidance scale, number of samples per prompt, seeds, batch size) and without any uncertainty estimate or statistical test. Because the central claim of this Exercise is about the stability of orderings, a ranking difference that lies within measurement noise is not an actionable finding. The authors should either provide error bars or significance tests over repeated sampling, or clearly label the exercise as a qualitative demonstration rather than an empirical result.
  3. [§3.1.2] The Trade-offs Exercise concludes that 'consistency increases steadily then plateaus at around 450,000 iterations' and that quality and diversity fluctuate, but this is based on a single training run with no repeated seeds or confidence intervals. Since the Exercise is intended to showcase actionable takeaways, the plateau claim in particular needs stronger support; otherwise it should be described as an illustrative observation from one run.
minor comments (5)
  1. [§3.4.1] There is a typo 'largest possible shared sized' that should read 'largest possible shared size'.
  2. [Appendix A.1] The description 'alt-text captions with up to 256 works per images' should read 'up to 256 words per image'.
  3. [§3.2.2] The phrase 'a base model to that generates latents' should read 'a base model that generates latents'.
  4. [Figure 7] The figure caption says re-captioning training data 'can help improve diversity and consistency,' but §3.4.2 reports improvements in precision (quality) and coverage (diversity) and does not clearly report an improvement in consistency, so the caption should be aligned with the reported metrics.
  5. [§2.3.1] The paper cites two variants of 'Improved precision and recall metric for assessing generative models' in the references (Kynkäänniemi et al., 2019 appears twice); one reference should be removed or disambiguated.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: EvalGIM is a library paper whose empirical demonstrations are benchmarked against external datasets, metrics, and public models; the self-citations reproduce prior methods rather than define the results.

full rationale

The paper's central claim is the utility of EvalGIM as a unified evaluation library, supported by software artifacts, external metrics (FID, precision/coverage with InceptionV3 features, CLIPScore, VQAScore), public datasets (COCO, ImageNet, CC12M, GeoDE), and public model checkpoints. The Evaluation Exercises are demonstrations, not derived predictions. The Pareto Front and Group Representation exercises cite the authors' prior work (Astolfi et al. 2024; Hall et al. 2024b) as the source of the methods, but the paper explicitly says it "reproduce[s] evaluation methods of prior works" (Sec. 1) and applies them to new model families and training runs; the conclusions (e.g., consistency plateauing around 450,000 iterations, geographic disparities across LDM versions, recaptioning effects) are empirical measurements against those external benchmarks, not consequences of the definitions of the metrics. No fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work to force a choice. The Prompt Types Exercise's claim that equal-size subsampling makes datasets "comparable due to being balanced" (Sec. 3.4.1) is a methodological assumption about cross-dataset comparability of distributional metrics, not a circular reduction; it is contestable because each marginal metric is computed against a dataset-specific reference manifold (Appendix B.1), but the paper's own Section 4 acknowledges limitations of automatic metrics and notes EvalGIM is not a replacement for human evaluation. Thus the closest issue is an unsupported comparability assumption, which is a validity concern rather than circularity. The self-citations are numerous but not load-bearing: they identify prior method definitions, while the library's central value is judged externally. Score 1 reflects the presence of self-citation without any circular derivation.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central contribution is a software library, so the ledger captures the analysis choices and domain assumptions that the empirical demonstrations depend on rather than fitted model parameters. The free parameters are hand-chosen evaluation settings; the axioms are validity assumptions about metrics and dataset comparability that the paper partially acknowledges.

free parameters (4)
  • k (nearest neighbors for precision/coverage manifolds)
    Set to 3 by hand for manifold construction; this choice affects precision/coverage values and hence the trade-off and group results (Appendix B.1).
  • Classifier guidance scale
    Fixed at 7.5 for the Trade-offs training-progress study; not swept across models, so model comparisons in other exercises may not be tuned per model (Section 3.1.2).
  • Balanced GeoDE subset size = 180 images per region, 27 objects, 29,160 total
    The balanced version omits under-sampled objects and tree-tagged images to control manifold size effects, which changes the diversity/quality measurements (Appendix A.1).
  • Balanced subsample size for Prompt Types = approximately 15,000 images
    Datasets are subsampled to the size of the smallest dataset (CC12M) to enable distributional metric comparison; this discards data and assumes size balancing is sufficient for comparability (Section 3.4.1).
assumptions (5)
  • domain assumption Automatic metrics (FID, precision, coverage, CLIPScore, VQAScore) are valid and reliable proxies for human-perceived quality, diversity, and consistency.
    The library's recommendations and the exercises' conclusions rest on this; the paper explicitly notes known limitations of automatic metrics in Section 4, including texture bias and stereotypical representation rewards.
  • domain assumption VQAScore correlates with human preference for consistency more strongly than CLIPScore.
    Used to justify choosing VQAScore in the Trade-offs exercise (Section 3.1.1), based on Lin et al. (2024) rather than an independent verification in this paper.
  • domain assumption Subsampling datasets to equal size makes distributional metrics such as precision and coverage comparable across datasets.
    The Prompt Types exercise (Section 3.4.1) assumes that equal-size subsamples yield fair cross-dataset comparisons; this is not proved and may confound dataset difficulty with dataset size.
  • domain assumption The compared models (LDM-1.5, LDM-2.1, LDM-3, LDM-XL) are evaluated under fair and representative configurations.
    The paper does not specify per-model sampling steps, guidance scales, or prompt preprocessing, so ranking conclusions rely on unspecified parity of setup (Sections 3.2.2 and 3.3.2).
  • domain assumption The selected datasets (COCO, ImageNet, GeoDE, CC12M, PartiPrompts, T2I-Compbench, DrawBench) cover the relevant distribution of text-to-image use cases.
    The library's actionability and the generalization of findings depend on dataset coverage; Section 4 acknowledges coverage limitations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EvalGIM: A Library for Evaluating Generative Image Models." pith.science (2026). https://pith.science/paper/KNSUACMF

@misc{pith2026241210604,
  author       = {Pith},
  title        = {Pith review of: EvalGIM: A Library for Evaluating Generative Image Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KNSUACMF}},
  note         = {Machine review of arXiv:2412.10604}
}
read the original abstract

As the use of text-to-image generative models increases, so does the adoption of automatic benchmarking methods used in their evaluation. However, while metrics and datasets abound, there are few unified benchmarking libraries that provide a framework for performing evaluations across many datasets and metrics. Furthermore, the rapid introduction of increasingly robust benchmarking methods requires that evaluation libraries remain flexible to new datasets and metrics. Finally, there remains a gap in synthesizing evaluations in order to deliver actionable takeaways about model performance. To enable unified, flexible, and actionable evaluations, we introduce EvalGIM (pronounced ''EvalGym''), a library for evaluating generative image models. EvalGIM contains broad support for datasets and metrics used to measure quality, diversity, and consistency of text-to-image generative models. In addition, EvalGIM is designed with flexibility for user customization as a top priority and contains a structure that allows plug-and-play additions of new datasets and metrics. To enable actionable evaluation insights, we introduce ''Evaluation Exercises'' that highlight takeaways for specific evaluation questions. The Evaluation Exercises contain easy-to-use and reproducible implementations of two state-of-the-art evaluation methods of text-to-image generative models: consistency-diversity-realism Pareto Fronts and disaggregated measurements of performance disparities across groups. EvalGIM also contains Evaluation Exercises that introduce two new analysis methods for text-to-image generative models: robustness analyses of model rankings and balanced evaluations across different prompt styles. We encourage text-to-image model exploration with EvalGIM and invite contributions at https://github.com/facebookresearch/EvalGIM/.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Classifier-Free Guidance: From High-Dimensional Analysis to Generalized Guidance Forms

    cs.LG 2025-02 conditional novelty 7.0 of 10

    CFG's distortion of the target distribution vanishes as data dimension grows, and a power-law generalization improves fidelity and diversity in high-dimensional generative models.

Reference graph

Works this paper leans on

61 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Evaluating clip: Towards characterization of broader capabilities and downstream implications, 2021

    Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. Evaluating clip: Towards characterization of broader capabilities and downstream implications, 2021. https://arxiv.org/abs/2108.02818

    arXiv 2021

  3. [3]

    Building normalizing flows with stochastic interpolants

    Michael S Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. arXiv preprint arXiv:2209.15571, 2022

    arXiv 2022

  4. [4]

    Consistency-diversity-realism pareto fronts of conditional image generative models, 2024

    Pietro Astolfi, Marlene Careil, Melissa Hall, Oscar Mañas, Matthew Muckley, Jakob Verbeek, Adriana Romero Soriano, and Michal Drozdzal. Consistency-diversity-realism pareto fronts of conditional image generative models, 2024. https://arxiv.org/abs/2406.10429

    arXiv 2024

  5. [5]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2 0 (3): 0 8, 2023

    2023

  6. [6]

    Paligemma: A versatile 3b vlm for transfer, 2024

    Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bau...

    arXiv 2024

  7. [7]

    Announcing black forest labs

    BlackForestLabs. Announcing black forest labs. https://blackforestlabs.ai/announcing-black-forest-labs/, 2024. [Online; accessed 10-December-2024]

    work page 2024

  8. [8]

    Large scale gan training for high fidelity natural image synthesis

    Andrew Brock. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018

    arXiv 2018

Show all 61 references
  1. [9]

    Conceptual 12M : Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M : Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021

    2021

  2. [10]

    Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-image generation

    Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-image generation. arXiv preprint arXiv:2310.18235, 2023

    arXiv

  3. [11]

    Visual programming for step-by-step text-to-image generation and evaluation

    Jaemin Cho, Abhay Zala, and Mohit Bansal. Visual programming for step-by-step text-to-image generation and evaluation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  4. [12]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248--255, 2009. doi:10.1109/CVPR.2009.5206848

    2009

  5. [13]

    Torchmetrics-measuring reproducibility in pytorch

    Nicki Skafte Detlefsen, Jiri Borovec, Justus Schock, Ananya Harsh Jha, Teddy Koker, Luca Di Liello, Daniel Stancl, Changsheng Quan, Maxim Grechkin, and William Falcon. Torchmetrics-measuring reproducibility in pytorch. Journal of Open Source Software, 7 0 (70): 0 4101, 2022

    2022

  6. [14]

    Does object recognition work for everyone?, 2019

    Terrance DeVries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. Does object recognition work for everyone?, 2019. https://arxiv.org/abs/1906.02659

    2019 arXiv

  7. [15]

    Scaling rectified flow transformers for high-resolution image synthesis, 2024

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transf...

    2024 arXiv

  8. [16]

    Commonsense-t2i challenge: Can text-to-image generation models understand commonsense? arXiv preprint arXiv:2406.07546, 2024

    Xingyu Fu, Muyu He, Yujie Lu, William Yang Wang, and Dan Roth. Commonsense-t2i challenge: Can text-to-image generation models understand commonsense? arXiv preprint arXiv:2406.07546, 2024

    2024 arXiv

  9. [17]

    Masked diffusion transformer is a strong image synthesizer

    Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is a strong image synthesizer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23164--23173, 2023

    2023

  10. [18]

    Wichmann, and Wieland Brendel

    Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. CoRR, abs/1811.12231, 2018. http://arxiv.org/abs/1811.12231

    2018 arXiv

  11. [19]

    Facet: Fairness in computer vision evaluation benchmark

    Laura Gustafson, Chloe Rolland, Nikhila Ravi, Quentin Duval, Aaron Adcock, Cheng-Yang Fu, Melissa Hall, and Candace Ross. Facet: Fairness in computer vision evaluation benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20370--20...

    2023

  12. [20]

    Bell, Candace Ross, Adina Williams, Michal Drozdzal, and Adriana Romero Soriano

    Melissa Hall, Samuel J. Bell, Candace Ross, Adina Williams, Michal Drozdzal, and Adriana Romero Soriano. Towards geographic inclusion in the evaluation of text-to-image models. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT '24, ...

    2024

  13. [21]

    Dig in: Evaluating disparities in image generations with indicators for geographic diversity, 2024 b

    Melissa Hall, Candace Ross, Adina Williams, Nicolas Carion, Michal Drozdzal, and Adriana Romero Soriano. Dig in: Evaluating disparities in image generations with indicators for geographic diversity, 2024 b . https://arxiv.org/abs/2308.06198

    2024 arXiv

  14. [22]

    Clipscore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021

    2021 arXiv

  15. [23]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium, 2018

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium, 2018. https://arxiv.org/abs/1706.08500

    2018 arXiv

  16. [24]

    Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages ...

    2023

  17. [25]

    T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36: 0 78723--78747, 2023 a

    2023

  18. [26]

    T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation, 2023 b

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation, 2023 b . https://arxiv.org/abs/2307.06350

    2023 arXiv

  19. [27]

    On improved conditioning mechanisms and pre-training strategies for diffusion models, 2024

    Tariq Berrada Ifriqi, Pietro Astolfi, Melissa Hall, Reyhane Askari-Hemmat, Yohann Benchetrit, Marton Havasi, Matthew Muckley, Karteek Alahari, Adriana Romero-Soriano, Jakob Verbeek, and Michal Drozdzal. On improved conditioning mechanisms and pre-training strategies for diffus...

    2024 arXiv

  20. [28]

    Openclip, July 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. https://doi.org/10.5281/zenodo.5143773. If yo...

    2021 doi

  21. [29]

    Genai arena: An open evaluation platform for generative models

    Dongfu Jiang, Max Ku, Tianle Li, Yuansheng Ni, Shizhuo Sun, Rongqi Fan, and Wenhu Chen. Genai arena: An open evaluation platform for generative models. arXiv preprint arXiv:2406.04485, 2024

    2024 arXiv

  22. [30]

    Viescore: Towards explainable metrics for conditional image synthesis evaluation

    Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explainable metrics for conditional image synthesis evaluation. arXiv preprint arXiv:2312.14867, 2023

    2023 arXiv

  23. [31]

    Improved precision and recall metric for assessing generative models

    Tuomas Kynk \"a \"a nniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019

    2019

  24. [32]

    Improved precision and recall metric for assessing generative models, 2019

    Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models, 2019

    2019

  25. [33]

    Holistic evaluation of text-to-image models, 2023

    Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Benita Teufel, Marco Bellagente, Minguk Kang, Taesung Park, Jure Leskovec, Jun-Yan Zhu, Li Fei-Fei, Jiajun Wu, Stefano Ermon, and Percy Liang. Holistic ev...

    2023 arXiv

  26. [34]

    Genai-bench: Evaluating and improving compositional text-to-visual generation

    Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, et al. Genai-bench: Evaluating and improving compositional text-to-visual generation. arXiv preprint arXiv:2406.13743, 2024 a

    2024 arXiv

  27. [35]

    K-sort arena: Efficient and reliable benchmarking for generative models via k-wise human preferences

    Zhikai Li, Xuewen Liu, Dongrong Fu, Jianquan Li, Qingyi Gu, Kurt Keutzer, and Zhen Dong. K-sort arena: Efficient and reliable benchmarking for generative models via k-wise human preferences. arXiv preprint arXiv:2408.14468, 2024 b

    2024 arXiv

  28. [36]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Aco...

    2023

  29. [37]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Procee...

    2014

  30. [38]

    Evaluating text-to-visual generation with image-to-text generation

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. arXiv preprint arXiv:2404.01291, 2024

    2024 arXiv

  31. [39]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2023. https://arxiv.org/abs/2210.02747

    2023 arXiv

  32. [40]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022

    2022 arXiv

  33. [41]

    Reliable fidelity and diversity metrics for generative models

    Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pages 7176--7185. PMLR, 2020

    2020

  34. [42]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195--4205, 2023

    2023

  35. [43]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. https://arxiv.org/abs/2307.01952

    2023 arXiv

  36. [44]

    Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever

    Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021

    2021

  37. [45]

    Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron B

    Vikram V. Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron B. Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. Geode: a geographically diverse evaluation dataset for object recognition, 2023. https://arxiv.org/abs/2301.02560

    2023 arXiv

  38. [46]

    Dalle-2 is seeing double: flaws in word-to-concept mapping in text2image models

    Royi Rassin, Shauli Ravfogel, and Yoav Goldberg. Dalle-2 is seeing double: flaws in word-to-concept mapping in text2image models. arXiv preprint arXiv:2210.10606, 2022

    2022 arXiv

  39. [47]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj\"orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684--10695, June 2022 a

    2022

  40. [48]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \"o rn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684--10695, 2022 b

    2022

  41. [49]

    What makes a good metric? evaluating automatic metrics for text-to-image consistency

    Candace Ross, Melissa Hall, Adriana Romero-Soriano, and Adina Williams. What makes a good metric? evaluating automatic metrics for text-to-image consistency. In First Conference on Language Modeling, 2024

    2024

  42. [50]

    Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion mod...

    2022 arXiv

  43. [51]

    Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lu c i \'c , Olivier Bousquet, and Sylvain Gelly. Assessing Generative Models via Precision and Recall . In Advances in Neural Information Processing Systems (NeurIPS) , 2018 a

    2018

  44. [52]

    Assessing generative models via precision and recall

    Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in neural information processing systems, 31, 2018 b

    2018

  45. [53]

    Who evaluates the evaluations? objectively scoring text-to-image prompt coherence metrics with t2iscorescore (ts2)

    Michael Saxon, Fatima Jahara, Mahsa Khoshnoodi, Yujie Lu, Aditya Sharma, and William Yang Wang. Who evaluates the evaluations? objectively scoring text-to-image prompt coherence metrics with t2iscorescore (ts2). arXiv preprint arXiv:2404.04251, 2024

    2024 arXiv

  46. [54]

    Decomposed evaluations of geographic disparities in text-to-image models, 2024

    Abhishek Sureddy, Dishant Padalia, Nandhinee Periyakaruppa, Oindrila Saha, Adina Williams, Adriana Romero-Soriano, Megan Richards, Polina Kirichenko, and Melissa Hall. Decomposed evaluations of geographic disparities in text-to-image models, 2024. https://arxiv.org/abs/2406.11988

    2024 arXiv

  47. [55]

    Szegedy, V

    C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016

    2016

  48. [56]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  49. [57]

    Superglue: A stickier benchmark for general-purpose language understanding systems

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019

    2019

  50. [58]

    Conceptmix: A compositional image generation benchmark with controllable difficulty

    Xindi Wu, Dingli Yu, Yangsibo Huang, Olga Russakovsky, and Sanjeev Arora. Conceptmix: A compositional image generation benchmark with controllable difficulty. arXiv preprint arXiv:2408.14339, 2024

    2024 arXiv

  51. [59]

    Florence-2: Advancing a unified representation for a variety of vision tasks, 2023

    Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks, 2023. https://arxiv.org/abs/2311.06242

    2023 arXiv

  52. [60]

    Scaling autoregressive models for content-rich text-to-image generation, 2022

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-ric...

    2022 arXiv

  53. [61]

    A contrastive compositional benchmark for text-to-image synthesis: A study with unified text-to-image fidelity metrics

    Xiangru Zhu, Penglei Sun, Chengyu Wang, Jingping Liu, Zhixu Li, Yanghua Xiao, and Jun Huang. A contrastive compositional benchmark for text-to-image synthesis: A study with unified text-to-image fidelity metrics. arXiv preprint arXiv:2312.02338, 2023

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.