Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

What makes a good metric? Evaluating automatic metrics for text-to-image consistency

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper asks what makes a good text-image consistency metric, proposes a minimal checklist, and finds that CLIPScore, TIFA, VPEval, and DSG all fail parts of it, mostly because they lean on prompt statistics and answer bias rather than…

desk verdict A solid meta-evaluation that earns its central cautionary conclusion, though its visual-properties evidence is weaker than the paper suggests. read the letter →

arxiv 2412.13989 v1 pith:GKV5YRCQ submitted 2024-12-18 cs.CL

Candace Ross , Melissa Hall , Adriana Romero Soriano , Adina Williams This is my paper · ORCID
classification cs.CL
keywords text-to-imageconsistencyautomaticevaluationmetricsconstructvalidityvisualquestionansweringCLIPScoreTIFAVPEvalDavidsonianSceneGraph
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Text-to-image generators are often judged by automatic metrics that chain a language model to a visual question-answering (VQA) system. This paper asks whether those metrics measure what they claim, and proposes a concrete checklist of properties a good metric should have. Applying that checklist to CLIPScore, TIFA, VPEval, and DSG, the authors find that none of the four satisfies all requirements: the metrics track linguistic properties of the prompt, barely track visual properties, and the three VQA-based metrics correlate strongly with each other while correlating weakly with CLIPScore. The paper also shows that the VQA-based metrics rest on skewed question distributions in which 'yes' or the first answer is almost always correct, so a trivial program could score well without looking at the image. The practical upshot is that current automatic consistency scores overstate how much image content they verify.

What carries the argument

The load-bearing tool is a construct-validity checklist: a good text-image consistency metric should be human interpretable, sensitive to text properties, sensitive to image properties, robust to known shortcuts, and additive beyond existing baselines. The authors operationalize each criterion with Spearman correlations against linguistic properties (Flesch-Kincaid grade level, Yngve syntactic complexity, word count) and visual properties (imageability, concreteness, overlap with ImageNet-21k object classes), with inter-metric correlations, and with four ablations: shuffled images, shuffled text, CLIP as a pseudo-VQA model, and text-only question answering. The question-statistics analysis, which counts how often the LM-generated ground-truth answer is 'yes' or the first multiple-choice option, is the machinery that exposes the shortcut reliance.

What would settle it

Take a single caption and generate one correctly matched image and one image that changes only a visible attribute (for example, turn a purple dog green or swap the order of two objects), then measure whether TIFA, VPEval, and DSG scores shift as much under the image change as under a matched word change in the caption. If image-side changes move scores as much as text-side changes, the central claim fails; if scores barely move, the claim is confirmed.

Watch

Extended reading notes

Core claim

The central claim is that the construct validity of these metrics fails: a text-image consistency metric should be sensitive to both text and image, robust to known shortcuts, and add information beyond existing baselines, and none of the four tested metrics meets those desiderata. The evidence has four parts: metric scores correlate moderately to strongly with readability, syntactic complexity, and prompt length; they correlate near zero with caption-derived imageability, concreteness, and ImageNet-21k lexical overlap; the VQA-based metrics correlate strongly among themselves and weakly with CLIPScore; and replacing the VQA model with text-only question answering changes scores only slightly. The paper concludes that existing automatic metrics are more language-driven than vision-driven, and that their VQA components can be mimicked by answer priors such as a default 'yes'.

Load-bearing premise

The conclusion that current metrics are insensitive to visual content rests on treating caption-derived word norms (imageability, concreteness, and ImageNet-21k lexical overlap) as proxies for what is actually visible in the image; if those word norms do not track image content, the visual-sensitivity result does not directly test the image.

Editorial extensions

If this is right

  • Published rankings of text-to-image models that rely on these metrics may reflect prompt difficulty and answer priors more than actual image faithfulness.
  • The strong negative correlation between the number of LM-generated questions and metric scores implies that the question-generation stage alone accounts for much of what these metrics measure.
  • Because text-only QA nearly matches the VQA-based scores, the visual component contributes little signal, and a cheaper text-only pipeline could approximate the same numbers.
  • The high inter-correlation among TIFA, VPEval, and DSG means a new VQA-based metric of the same family adds little unless it changes the question-answer distribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The visual-sensitivity conclusion should be probed with direct image-content perturbations, such as changing a visible attribute (color, count, spatial relation) while holding the caption fixed; that test would separate true image sensitivity from word-level proxy sensitivity.
  • Inference: The yes-bias critique likely extends to newer LM-plus-VQA evaluators built on the same question-generation templates, so reporting a majority-class baseline and a balanced yes/no question set would be a quick admission test for any candidate metric.
  • Inference: A practical design consequence the authors leave implicit is that future metrics should generate questions with balanced ground-truth answers and should be validated against their own majority-class baselines before being used for model ranking.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a minimal set of desiderata for automatic text-to-image (T2I) consistency metrics and applies them to four metrics: CLIPScore, TIFA, VPEval, and DSG. It reports correlations between metric scores and linguistic prompt properties, correlations between metric scores and three word-level 'visual properties' (imageability, concreteness, ImageNet-21k overlap), inter-metric correlations, statistics of the LM-generated question distributions, and four ablations (shuffled images, shuffled text, CLIP replacing VQA, text-only QA). The main conclusions are that none of the four metrics satisfies all desiderata; the VQA-based metrics are highly correlated with each other and contribute limited new information over one another; and the VQA-based metrics likely exploit yes-bias and first-answer bias, so their high scores may not reflect genuine text-image consistency.

Significance. If the conclusions hold, the paper is a valuable construct-validity audit of widely used metrics. Its desiderata framework is a useful starting point, and the Table 6 question-distribution statistics are a concrete, reproducible warning about answer-skew artifacts. The ablations, especially the text-only QA condition, provide a direct test of whether the image is necessary. The paper is transparent about several caveats. However, as detailed below, the central visual-sensitivity claim is not supported by the Section 3.2 experiment as designed, and the shortcut claim would be materially strengthened by a majority-class baseline. With those revisions, the paper would make a solid contribution.

major comments (3)
  1. [3.2, Table 3b] The three 'visual properties'—imageability, concreteness, and ImageNet-21k caption overlap—are computed from the words of the prompt, not from the generated images. Weak correlations with these lexical norms therefore do not establish that the metrics are insensitive to visual properties; they only show insensitivity to these particular prompt-level features. The conclusion in the abstract and Table 1 that the metrics are 'more language-related than vision-related' is not supported by this experiment. Furthermore, Ablation 1 (Section 4) shows a large drop when images are shuffled, which is direct evidence of sensitivity to gross image correspondence. Please either replace this experiment with image-derived descriptors or controlled image perturbations, or restrict the claim to the tested lexical norms and rely on Ablation 4 for the visual-insensitivity conclusion.
  2. [3.4, Table 6; Table 2] The yes-bias and first-answer bias argument is suggestive but incomplete. Table 6 reports the distribution of ground-truth answers in the LM-generated questions, not the answer distribution of the VQA model; Table 2's random-chance baseline is acknowledged to be uninformative, but the paper does not provide the required majority-class baseline. The assertion that a program printing 'yes' or the first answer 'could replace the VQA component entirely' requires knowing the VQA model's actual output distribution. Please add a majority-class baseline computed with the actual BLIP2-Flan T5 XL predictions, and report the model's yes-rate and first-answer rate. The text-only QA ablation (Section 4, Ablation 4) partially fills this gap, but the shortcut claim would be much stronger with the direct baseline.
  3. [Section 4, Figure 2] The ablation results are reported only as bars in figures, with no numerical values or confidence intervals. This matters because the paper claims that 'not all model components are strictly necessary' and that VQA is 'basically sufficient' replaced by text-only QA; the reader cannot assess the magnitude of the performance gap from the figure. Please provide a table with exact scores for each metric, model, and ablation condition, together with error bars or confidence intervals.
minor comments (6)
  1. [2.2] The sentence 'Where TIFA questions are in natural language The visual programs are executed...' has a grammatical error; a period or semicolon is needed.
  2. [Appendix A.1] The caption of Table 5 contains the typo 'simialr'; it should be 'similar'.
  3. [3.3, Figures 1 and 3] The pairwise Spearman correlations are not reported numerically in the text; since redundancy among VQA-based metrics is a core finding, please include a supplementary table with the correlation values and significance levels.
  4. [3.2, footnote 3] The robustness checks for missing word norms are described in the footnote but their results are not shown; please report them in an appendix or table.
  5. [Table 1] CLIPScore's 'Human Interpretable' is marked with ✗, but the rubric for this column is not defined in Section 2; please add a sentence clarifying what 'human interpretable' means and why CLIPScore fails this criterion.
  6. [3.4, Table 4] The Winoground question-count correlations are mostly not statistically significant, likely due to the small sample size; the text should explicitly note this lower power when interpreting the Winoground results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the audit is self-contained, empirical, and benchmarked externally; the word-norm proxy for visual properties is a validity limitation, not a by-construction reduction.

full rationale

The paper is an empirical meta-evaluation, not a derivation: it defines desiderata, measures correlations between existing metrics and external text/image properties, and tests ablations against random-chance and CLIP baselines. Its central claims—that VQA-based metrics are redundant with each other and likely exploit yes-bias—are supported by direct statistics on LM-generated question distributions (Table 6: roughly 99% of yes/no questions have yes as the correct answer) and by prior VQA bias literature, not by fitting any parameter to the metrics' own scores. The 'more language-related than vision-related' conclusion is weakened by the fact that the 'visual properties' (imageability, concreteness, ImageNet-21k overlap) are computed from prompt words rather than image content (Section 3.2), but that is a construct-validity or proxy limitation, not circularity: the paper does not define metric sensitivity in terms of those norms, nor does it re-derive its conclusion from the same quantity it measures. Self-citations (Winoground as a prompt dataset, Hall et al. 2023 for diversity desiderata) are external or auxiliary; no uniqueness theorem or ansatz is imported from the authors' prior work to force the conclusion. The audit is therefore self-contained and benchmarked externally.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The ledger is short because the paper is an empirical audit rather than a derivation. It does not fit parameters or introduce entities. The main assumptions are about the adequacy of the desiderata, the validity of lexical proxies for visual properties, and the representativeness of the specific LM and VQA models used to instantiate the metrics.

assumptions (3)
  • domain assumption The list of desiderata in Table 1 is an adequate minimal operationalization of construct validity for T2I consistency metrics.
    The paper asserts these criteria without external justification; if the desiderata are inappropriate, the conclusion that no metric satisfies all of them is less meaningful. Section 2.1.
  • ad hoc to paper Lexical norms (imageability, concreteness) and ImageNet-21k overlap are valid proxies for sensitivity to visual properties.
    These are text-derived measures, not image features; the paper does not justify them as measures of visual information use. Section 3.2.
  • domain assumption Llama-2-Chat 70B and BLIP2-Flan T5 XL are representative instantiations of the LM and VQA components underlying TIFA, VPEval, and DSG.
    The metrics are pipelines; results may change with different component models. Section 3, evaluation setup.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What makes a good metric? Evaluating automatic metrics for text-to-image consistency." pith.science (2026). https://pith.science/paper/GKV5YRCQ

@misc{pith2026241213989,
  author       = {Pith},
  title        = {Pith review of: What makes a good metric? Evaluating automatic metrics for text-to-image consistency},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GKV5YRCQ}},
  note         = {Machine review of arXiv:2412.13989}
}
read the original abstract

Language models are increasingly being incorporated as components in larger AI systems for various purposes, from prompt optimization to automatic evaluation. In this work, we analyze the construct validity of four recent, commonly used methods for measuring text-to-image consistency - CLIPScore, TIFA, VPEval, and DSG - which rely on language models and/or VQA models as components. We define construct validity for text-image consistency metrics as a set of desiderata that text-image consistency metrics should have, and find that no tested metric satisfies all of them. We find that metrics lack sufficient sensitivity to language and visual properties. Next, we find that TIFA, VPEval and DSG contribute novel information above and beyond CLIPScore, but also that they correlate highly with each other. We also ablate different aspects of the text-image consistency metrics and find that not all model components are strictly necessary, also a symptom of insufficient sensitivity to visual information. Finally, we show that all three VQA-based metrics likely rely on familiar text shortcuts (such as yes-bias in QA) that call their aptitude as quantitative evaluations of model performance into question.

Figures

Figures reproduced from arXiv: 2412.13989 by the authors.

Figure 1
Figure 1. Correlation between each pair of text-image consistency metrics for COCO. The [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Ablation results for COCO. The bars refer to the original, unmodified metrics [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Correlation between each pair of text-image consistency metrics – CLIPScore, TIFA, [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Ablation results for Winoground dataset. The bars refer to the original, unmodified [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: For the TIFA, VPEval and DSG text-image consistency metrics, we use ablate the [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Maestro: Self-Improving Text-to-Image Generation via Agent Orchestration

    cs.AI 2025-09 conditional novelty 6.0 of 10

    A multi-agent prompt-refinement system using pairwise AI judging and targeted edit signals outperforms prior automated methods on complex text-to-image tasks.

  2. DIMCIM: A Quantitative Evaluation Framework for Default-mode Diversity and Generalization in Text-to-Image Generative Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A new evaluation framework, DIMCIM, measures default-mode diversity and prompted generalization in text-to-image models, finding a scale trade-off and a 0.85 correlation between default diversity and training data diversity.

Reference graph

Works this paper leans on

42 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [4]

    doi: 10.18653/v1/2021.acl-long.295

    Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.295. URL https://aclanthology.org/ 2021.acl-long.295. Jon Gauthier and Roger Levy. Linking artificial and human neural representations of language. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on...

    doi:10.18653/v1/2021.acl-long.295 2021

  2. [5]

    doi: 10.18653/v1/D19-1050

    Association for Computational Linguistics. doi: 10.18653/v1/D19-1050. URL https://aclanthology.org/D19-1050. Robert Geirhos, J¨orn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673,

    work page doi:10.18653/v1/d19-1050

  3. [10]

    doi: 10.18653/ v1/N18-2017

    Association for Computational Linguistics. doi: 10.18653/ v1/N18-2017. URL https://aclanthology.org/N18-2017. Melissa Hall, Candace Ross, Adina Williams, Nicolas Carion, Michal Drozdzal, and Adri- ana Romero Soriano. Dig in: Evaluating disparities in image generations with indicators for geographic diversity. arXiv preprint arXiv:2308.06198,

    arXiv 2017

  4. [12]

    doi: 10.18653/v1/2023.eacl-main.38

    Association for Computational Linguistics. doi: 10.18653/v1/2023.eacl-main.38. URL https:// aclanthology.org/2023.eacl-main.38. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718,

    arXiv 2023

  5. [13]

    URL https://aclanthology.org/2020.inlg-1.23

    Association for Computational Linguistics. URL https://aclanthology.org/2020.inlg-1.23. Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint arXiv:2303.11897,

    arXiv 2020

  6. [14]

    org/10.5281/zenodo.5143773

    URL https://doi. org/10.5281/zenodo.5143773. If you use this software, please cite it as below. Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. Revisiting visual question answering baselines. In European Conference on Computer Vision, pp. 727–739. Springer,

    doi:10.5281/zenodo.5143773

  7. [16]

    Adversarial examples for evaluating reading comprehension systems

    Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2021–2031, Copenhagen, Denmark, September

    work page 2017

  8. [17]

    doi: 10.18653/v1/D17-1215

    Association for Com- putational Linguistics. doi: 10.18653/v1/D17-1215. URL https://aclanthology.org/ D17-1215. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423– 438,

    doi:10.18653/v1/d17-1215

Show all 42 references
  1. [18]

    URL https://aclanthology.org/2020.tacl-1.28

    doi: 10.1162/tacl a 00324. URL https://aclanthology.org/2020.tacl-1.28. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethinking benchmarking in nlp. arXiv ...

    2020 arXiv

  2. [19]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597,

    arXiv

  3. [20]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  4. [22]

    doi: 10.18653/v1/2023

    Association for Computational Linguistics. doi: 10.18653/v1/2023. acl-long.736. URL https://aclanthology.org/2023.acl-long.736. Edward Loper and Steven Bird. NLTK: The natural language toolkit. arXiv preprint cs/0205028,

    2023 arXiv

  5. [24]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741,

    arXiv

  6. [25]

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen

    URL https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/ 2021/file/084b6fbb10729ed4da8c3d3f5a3ae7c9-Paper-round2.pdf. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierar- chical text-conditional image generation with clip latents. arxiv

    2021

  7. [27]

    Sunny and dark outside?! improving answer consistency in vqa through entailed question generation

    Arijit Ray, Karan Sikka, Ajay Divakaran, Stefan Lee, and Giedrius Burachas. Sunny and dark outside?! improving answer consistency in vqa through entailed question generation. arXiv preprint arXiv:1909.04696,

    1909 arXiv

  8. [28]

    Imagenet-21k pretrain- ing for the masses

    Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretrain- ing for the masses. arXiv preprint arXiv:2104.10972,

    arXiv

  9. [29]

    Who evaluates the evaluations? objectively scoring text-to-image prompt coherence metrics with t2iscorescore (ts2)

    Michael Saxon, Fatima Jahara, Mahsa Khoshnoodi, Yujie Lu, Aditya Sharma, and William Yang Wang. Who evaluates the evaluations? objectively scoring text-to-image prompt coherence metrics with t2iscorescore (ts2). arXiv preprint arXiv:2404.04251,

    arXiv

  10. [30]

    you are grounded!

    Vered Shwartz, Rachel Rudinger, and Oyvind Tafjord. “you are grounded!”: Latent name artifacts in pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6850–6861, Online, November

    2020

  11. [31]

    doi: 10.18653/v1/2020.emnlp-main.556

    Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.556. URL https://aclanthology.org/2020.emnlp-main.556. Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language modeling and the distributional hypothes...

    2020 arXiv

  12. [32]

    doi: 10.18653/v1/2023.acl-long.333

    Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.333. URL https:// aclanthology.org/2023.acl-long.333. Sanchit Sinha, Hanjie Chen, Arshdeep Sekhon, Yangfeng Ji, and Yanjun Qi. Perturbing inputs for fragile interpretations in deep natural language proce...

    2023 arXiv

  13. [33]

    doi: 10.18653/v1/P19-1355

    Association for Computational Linguistics. doi: 10.18653/v1/P19-1355. URL https://aclanthology. org/P19-1355. Kaiser Sun, Adina Williams, and Dieuwke Hupkes. The validity of evaluation results: Assessing concurrence across compositionality benchmarks. In Jing Jiang, David Reit...

    doi

  14. [34]

    doi: 10.18653/v1/2023.conll-1.19

    Associ- ation for Computational Linguistics. doi: 10.18653/v1/2023.conll-1.19. URL https: //aclanthology.org/2023.conll-1.19. Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models...

    2023 doi

  15. [35]

    URL https: //aclanthology.org/2020.tacl-1.40

    doi: 10.1162/tacl a 00335. URL https: //aclanthology.org/2020.tacl-1.40. Tianlu Wang, Rohit Sridhar, Diyi Yang, and Xuezhi Wang. Identifying and mitigating spuri- ous correlations for improving robustness in NLP models. In Findings of the Association for Computational Linguist...

    2020 doi

  16. [36]

    doi: 10.18653/v1/2022.findings-naacl.130

    Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-naacl.130. URL https://aclanthology.org/2022.findings-naacl.130. Chris Welty, Praveen Paritosh, and Lora Aroyo. Metrology for ai: From benchmarks to instruments. arXiv preprint arXiv:1911.01875,

    2022 arXiv

  17. [37]

    Vera Liao

    Ziang Xiao, Susu Zhang, Vivian Lai, and Q. Vera Liao. Evaluating evaluation metrics: A framework for analyzing NLG evaluation metrics using measurement theory. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natur...

    2023

  18. [38]

    doi: 10.18653/v1/2023.emnlp-main.676

    Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.676. URL https://aclanthology.org/2023.emnlp-main.676. Victor H Yngve. A model and an hypothesis for language structure. Proceedings of the American philosophical society, 104(5):444–466,

    2023 doi

  19. [40]

    Similar to the findings for COCO, we observe moderate to strong correlation for some of linguistic properties and essentially no correlation for the visual properties

    15 Published as a conference paper at COLM 2024 A Text-image consistency metric Correlations for Winoground A.1 Correlations between Linguistic and Visual Properties We present the results for the correlation between different linguistic and visual properties for text-image co...

    2024

  20. [41]

    bear” and answers them with “yes

    16 Published as a conference paper at COLM 2024 Figure 3: Correlation between each pair of text-image consistency metrics – CLIPScore, TIFA, VPEval and DSG – for 4 text-to-image generative models and for real images. Similar to the finding for COCO shown in Figure 1, we find t...

    2024

  21. [42]

    and VQA models (Agrawal et al., 2018; Ray et al., 2019; Shah et al., 2019; Agarwal et al., 2020; Sheng et al.,

    2018

  22. [1960]

    Scaling autoregressive multi- modal models: Pretraining and instruction tuning

    14 Published as a conference paper at COLM 2024 Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi- modal models: Pretraining and instruction tuning. arXi...

    2024 arXiv

  23. [2002]

    Improving text-to- image consistency via automatic prompt optimization

    Oscar Ma ˜nas, Pietro Astolfi, Melissa Hall, Candace Ross, Jack Urbanek, Adina Williams, Aishwarya Agrawal, Adriana Romero-Soriano, and Michal Drozdzal. Improving text-to- image consistency via automatic prompt optimization. arXiv preprint arXiv:2403.17804,

    arXiv

  24. [2013]

    A survey of methods for addressing class imbalance in deep-learning based natural language processing

    11 Published as a conference paper at COLM 2024 Sophie Henning, William Beluch, Alexander Fraser, and Annemarie Friedrich. A survey of methods for addressing class imbalance in deep-learning based natural language processing. In Proceedings of the 17th Conference of the Europe...

    2024

  25. [2015]

    doi: 10.18653/v1/D15-1013

    Association for Computational Linguistics. doi: 10.18653/v1/D15-1013. URL https://aclanthology.org/D15-1013. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. InProceedings...

    doi

  26. [2016]

    Measurement and fairness

    Abigail Z Jacobs and Hanna Wallach. Measurement and fairness. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 375–385,

    2021

  27. [2017]

    Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE

    Yvette Graham. Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 128–137, Lisbon, Portugal, September

    2015

  28. [2018]

    Davidsonian scene graph: Improving relia- bility in fine-grained evaluation for text-image generation

    Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving relia- bility in fine-grained evaluation for text-image generation. arXiv preprint arXiv:2310.18235, 2023a. 10 Pu...

    arXiv

  29. [2019]

    Worldsense: A synthetic benchmark for grounded reasoning in large language models

    Youssef Benchekroun, Megi Dervishi, Mark Ibrahim, Jean-Baptiste Gaya, Xavier Martinet, Gr´egoire Mialon, Thomas Scialom, Emmanuel Dupoux, Dieuwke Hupkes, and Pascal Vincent. Worldsense: A synthetic benchmark for grounded reasoning in large language models. arXiv preprint arXiv...

    arXiv

  30. [2020]

    Robustness of named- entity replacements for in-context learning

    Saeed Goodarzi, Nikhil Kagita, Dennis Minn, Shufan Wang, Roberto Dessi, Shubham Toshniwal, Adina Williams, Jack Lanchantin, and Koustuv Sinha. Robustness of named- entity replacements for in-context learning. In Houda Bouamor, Juan Pino, and Ka- lika Bali (eds.), Findings of t...

    2023

  31. [2021]

    doi: 10.18653/v1/2021.acl-long.81

    Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.81. URL https://aclanthology.org/2021.acl-long.81. Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior research meth...

    2021 doi

  32. [2022]

    arXiv preprint arXiv:2204.06125,

    arXiv

  33. [2023]

    doi: 10.18653/v1/2023.findings-emnlp.728

    Association for Computational Linguis- tics. doi: 10.18653/v1/2023.findings-emnlp.728. URL https://aclanthology.org/2023. findings-emnlp.728. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image und...

    2023 doi

  34. [2024]

    Liu, Tony Lee, Robin Jia, and Percy Liang

    Nelson F. Liu, Tony Lee, Robin Jia, and Percy Liang. Do question answering modeling improvements hold across benchmarks? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13186–13218, Toronto, 12 Published a...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.