Pith. sign in

REVIEW 4 major objections 3 minor 31 references

Automated image-evaluation metrics systematically prefer typical but textually wrong images.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 11:48 UTC pith:GJ4YTDA7

load-bearing objection Useful diagnostic benchmark, but the central claim is confounded by a missing control for attribute insensitivity. the 4 major comments →

arxiv 2601.04946 v3 pith:GJ4YTDA7 submitted 2026-01-08 cs.CV cs.AI

Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metrics

Subhadeep Roy , Gagan Bhatia , Steffen Eger This is my paper
classification cs.CV cs.AI
keywords prototypicality biastext-to-image evaluationmultimodal metricssemantic correctnesscontrastive benchmarksocial stereotypesmetric reliability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Automatic metrics used to judge text-to-image models are not reliably checking whether the image says what the prompt says. On a controlled benchmark, the paper shows that many leading metrics give higher scores to a plausible, prototypical image that violates the prompt than to a correct but less typical image. This 'prototypicality bias' appears across animals, objects, and demographic categories, and is strongest in embedding and reward-based metrics, still present in VQA-based and LLM-based judges, and nearly absent in human ratings. The paper introduces PROTOBIAS, a benchmark for this failure, and PROTOSCORE, a small contrastively trained metric that cuts failure rates and reduces confident misrankings.

Core claim

The central discovery is a systematic failure mode: text-image alignment metrics can rank a semantically incorrect image above a correct one whenever the incorrect image looks more visually or socially canonical. The authors formalize this with triplets (T, I_corr, I_adv), where the adversary swaps in a prototypical main entity and alters a single auxiliary detail such as count, color, or position. Across thousands of generated pairs, embedding-based and preference-based metrics fail roughly 70% of the time, VQA-based and one LLM judge fail over 45%, and human annotators fail essentially 0%. A 7-billion-parameter contrastive evaluator trained on the benchmark reduces failure to about 32% whi

What carries the argument

The load-bearing object is the contrastive triplet: a prompt T, a semantically correct but non-prototypical image I_corr, and a prototypical but semantically incorrect adversary I_adv built by swapping the main entity for a typical exemplar and perturbing one auxiliary knob. Triplets are generated from paired templates, filtered by an independent VLM, and used both to measure metric failure and to train a contrastive evaluator. The comparison M(T,I_corr) > M(T,I_adv) is the single test that separates a semantically faithful metric from a prototype-favoring one.

Load-bearing premise

The benchmark assumes that each contrast isolates prototypicality, but every pair also differs in an auxiliary semantic detail, so failures could be attributed to metrics ignoring that detail rather than to prototypicality itself.

What would settle it

Run the same pairwise preference task with (a) a control where the auxiliary knob changes while the main entity stays prototypical in both images, and (b) a control where only the main entity's typicality changes while all auxiliaries stay identical. If the knob-only control produces failure rates close to the original contrast, prototypicality is not the deciding factor.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Benchmark scores that rely on these metrics are likely overstating how well text-to-image models follow prompts.
  • Model selection and data filtering pipelines using such metrics may actively reward generic, stereotype-reinforcing images over diverse but correct ones.
  • The demography results imply that bias audits of generative models can themselves be biased if they trust the same metrics.
  • A relatively small open-source evaluator can approach the semantic reliability of much larger closed judges, suggesting the fix is accessible.
  • Human judgment on these contrasts is near-perfect, indicating the bias is not an inherent limitation of evaluation but of current metric design.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because each adversarial pair also changes a semantic detail, the reported failures may stem from metrics ignoring auxiliary attributes rather than from prototypicality per se; a knob-only control would separate these mechanisms.
  • The same bias likely affects any embedding-based retrieval or matching system trained on web-scale image-text data, from image search to caption verification, not just T2I scoring.
  • The success of contrastive training suggests a general recipe: collect hard negative pairs that mix typicality with semantic errors and fine-tune the evaluator on them, which could be applied to other multimodal domains.
  • The demographic axis ties prototypicality bias to social stereotypes, so LLM-as-judge systems that show uneven reliability on such pairs may inherit and amplify culturally specific defaults.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper introduces PROTOBIAS, a contrastive benchmark for evaluating prototypicality bias in text-to-image (T2I) metrics. Each triplet consists of a prompt, a semantically correct but non-prototypical image (I_corr), and a prototypical but semantically incorrect adversarial image (I_adv), with the incorrectness introduced via auxiliary knobs such as count, color, or spatial layout. Using the failure criterion in Eq. (5), the authors report high failure rates for CLIPScore, PickScore, VQAScore, GPT-4o, and GPT-5, while human annotators consistently prefer I_corr. They also propose PROTOSCORE, a 7B VLM fine-tuned on PROTOBIAS, which reduces failure rates. The paper claims that automated metrics exhibit a systematic prototypicality bias, preferring visually or socially typical images even when they violate prompt semantics.

Significance. If the central claim is sound, this work identifies a serious and under-studied failure mode in automated T2I evaluation, with direct implications for benchmarking and data filtering. The contrastive benchmark design is a useful resource for the community, and the human evaluation provides a valuable reference. The paper also delivers a promising mitigation baseline. However, the current evidence does not fully distinguish prototypicality bias from attribute-insensitivity, and the absence of a manipulation check for the prototypicality construct weakens the interpretation. The claimed significance is high, but the support is partial.

major comments (4)
  1. [§3.2 (Adversarial construction) and §3.1 Eq. (5)] Each contrast varies two factors simultaneously: the prototypicality of the main entity and the correctness of an auxiliary knob. A metric that largely ignores the knob (a known limitation for CLIP-scale models on count/color/spatial attributes) will fail these contrasts for any typical main entity, independent of any preference for prototypicality. The paper does not include a control condition with the main entity held fixed and only the knob varied, nor one with only prototypicality varied. Without such controls, the reported failure rates (e.g., 71.62% for CLIPScore) are equally consistent with attribute-insensitivity. This is load-bearing because the central claim of 'prototypicality bias' requires that the metric perceives the semantic violation and yet prefers the prototypical image. Please add controls or otherwise rule out this alternative explanation.
  2. [§3.2 (Taxonomy and Demography) and §4 (Human Evaluation)] The benchmark's core construct—prototypicality—is never validated. The paper asserts that e.g., penguin vs robin, bean bag vs chair, or Nigerian vs American differ in prototypicality, but no human typicality ratings are collected to confirm these manipulations. For the Demography domain, the 'prototypicality' assumptions are socially and culturally loaded; without an empirical manipulation check, the interpretation of the results (especially the Demography failure rates) rests entirely on the authors' prior assumptions. A small human typicality rating study for the main entities across all domains would substantiate the construct and strengthen the benchmark's validity.
  3. [§5.1 (Figure 3 text) vs Table 2] The average scores reported in the text for Figure 3 conflict with Table 2. For example, the text states CLIPScore Animals: 0.81 PA vs 0.67 SC and Demography: 0.68 PA vs 0.54 SC, while Table 2 lists CLIPScore Animals: SC 0.63, PA 0.74 and Demography: SC 0.56, PA 0.69. If Figure 3 is computed on the full evaluation set and Table 2 on the 300 human-annotated subset, this should be stated explicitly; if they are supposed to be the same, the numbers need to be reconciled. As written, the inconsistency undermines the reliability of the quantitative claims.
  4. [§4 (Training ProtoScore) and §5.1 (Results)] PROTOSCORE is trained on 10k pairs sampled from the same PROTOBIAS distribution and evaluated on a disjoint split from the same benchmark. This is an in-distribution evaluation; the reported 31.60% failure rate may reflect overfitting to the specific contrast patterns rather than a general improvement in semantic robustness. To support the mitigation claim, the authors should evaluate PROTOSCORE on a held-out domain or on an independent T2I evaluation benchmark, or at least report cross-validation within PROTOBIAS using category-holdout splits. This is not a rejection of the main benchmark, but it limits the generalizability of the PROTOSCORE contribution.
minor comments (3)
  1. [§4 (Dataset Filtration)] The automated filtration uses Qwen2.5-VL with a hand-chosen threshold of ≥8, yielding a 62.05% retention rate. Since Qwen2.5-VL is itself a multimodal VLM, it may share the same prototypicality biases as the metrics under test. The authors should analyze the sensitivity of the results to the threshold and ideally report the effect of filtering on the composition of the retained pairs.
  2. [§4 (Human Evaluation)] The human evaluation uses only 300 image-text pairs and five expert annotators. The paper acknowledges this limitation, but the claim that humans 'prefer I_corr in 100% of cases' is based on a very small sample. A larger and more diverse annotator pool would increase confidence, especially for the Demography domain where cultural backgrounds may affect judgments.
  3. [§3.1, Eq. (3)] The definitions of I_corr and I_adv using argmin/argmax over similarity are not actually solved; the paper correctly notes this with 'We do not solve Eqs. (3) exactly.' However, the connection between the heuristic construction and these extremal definitions should be stated more clearly to avoid overclaiming.

Circularity Check

0 steps flagged

No significant circularity: the central claim is an empirical benchmark finding, and PROTOSCORE is a transparently trained baseline, not a derived prediction.

full rationale

The central claim—that widely used T2I metrics prefer prototypical but semantically incorrect images over correct non-prototypical ones—is an empirical benchmark finding, not a derivation from definitions. PROTOBIAS is constructed through an external pipeline (taxonomy → template → prompt → image generation), filtered by an independent VLM, and validated by human annotators who prefer I_corr in 100% of cases. The failure criterion Eq. (5) is a measurement rule, not a definition that forces the conclusion. The closest candidates to circularity do not hold. First, the adversarial construction in §3.2 changes both main-entity prototypicality and the auxiliary knob, so high failure rates could partly reflect attribute-insensitivity; however, this is a confound or construct-validity risk, not a circularity, because the paper does not identify prototypicality bias with failure on these contrasts by definition, nor is the observed failure a fitted value used to define the construct. Second, PROTOSCORE is trained on 10k pairs sampled from the same filtered dataset and evaluated on a disjoint split; this is transparent supervised evaluation of a mitigation baseline, not a prediction of an independent quantity derived from the benchmark. The paper does not rename its training objective as a novel result. Third, the self-citations (Chen & Eger 2023; Leiter & Eger 2024; Leiter et al. 2025) appear as related work and inspiration, not as load-bearing justification; the grounding for prototypicality is external (Rosch 1975; Ma & Correll 2011). There is no uniqueness theorem, no ansatz smuggled via self-citation, and no renaming of a known result. The Limitations section acknowledges scope constraints (limited categories, Western-centric distributions, small expert annotation set), but these are scope limitations, not circular steps. The derivation chain is therefore self-contained.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

No new physical or ontological entities are introduced. The load-bearing postulates are benchmark-labeling choices: a filter threshold, a prototype-theory grounding, a stereotype mapping, and a trust assumption about generators/filters.

free parameters (1)
  • Qwen2.5-VL semantic-alignment filter threshold = ≥8 on a 1–10 scale
    Chosen by the authors and used to retain 62.05% of generated pairs; it shapes the benchmark composition and therefore all reported metric failure rates.
axioms (4)
  • domain assumption Prototype theory (Rosch 1975) is the correct cognitive model for metric behavior and dataset design.
    Invoked in §3.2 to justify why prototypical exemplars dominate training distributions and influence metrics.
  • domain assumption FLUX.1-schnell with 5 inference steps produces images that faithfully realize the template semantics, and Qwen2.5-VL filtering correctly labels semantic correctness.
    Both generator and filter serve as ground truth in §4; any systematic error transfers to the benchmark, and human validation covers only 300 of ~19k retained images.
  • domain assumption The privileged/disadvantaged mapping in Demography operationalizes prototypicality.
    §3.2 and Appendix A.1 define prototypicality via social stereotypes (Christian/American/heterosexual as default); this is contestable but used to construct labels.
  • domain assumption Human annotations on 300 pairs by five expert annotators generalize to the full benchmark.
    Used to claim humans prefer SC in 100% of cases, but the sample is small and not representative of the broader population.

pith-pipeline@v1.3.0-alltime-deepseek · 13398 in / 10996 out tokens · 111934 ms · 2026-08-03T11:48:17.771557+00:00 · methodology

0 comments
read the original abstract

Automatic metrics are widely used to evaluate text-to-image models, often replacing human judgment in benchmarking, model selection, and large-scale data filtering. Yet they may reward images that look plausible or prototypical rather than images that faithfully satisfy the prompt. We identify prototypicality bias as a systematic blindspot in multimodal evaluation: metrics can prefer a semantically incorrect but visually or socially prototypical image over a correct but less prototypical one. We introduce PROTOBIAS, a controlled diagnostic benchmark across Animals, Objects, and Demography, where semantically correct images are contrasted with plausible prototypical adversaries containing a single controlled semantic violation. Grounded in prototype theory and social-category prototypicality, PROTOBIAS is constructed with multiple prompt generators, image generators, and independent VLM filters, and validated through prompt-quality, human-annotation, and image-quality controls. Using PROTOBIAS, we show that widely used embedding, reward, VQA-based, and VLM-as-judge metrics frequently fail these contrasts, while human judgments remain more faithful to semantic correctness. We further introduce PROTOSCORE, a lightweight contrastively trained evaluator, as an initial mitigation baseline. PROTOBIAS provides a focused benchmark for measuring prototypicality-driven metric failures and developing more semantically faithful T2I evaluators.

Figures

Figures reproduced from arXiv: 2601.04946 by Gagan Bhatia, Steffen Eger, Subhadeep Roy.

Figure 1
Figure 1. Figure 1: Examples of prototypicality bias in multimodal metrics. Each row contrasts a semantically correct but non-prototypical image (SC) with a prototypical but semantically incorrect adversarial image (PA) generated from the same prompt. The rightmost column indicates whether each evaluator ranks SC above PA (✓) or prefers PA (✗). controlled adversarial benchmark spanning Animals, Objects, and Demography, explic… view at source ↗
Figure 2
Figure 2. Figure 2: Failure rates of multimodal evaluation metrics [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Pairwise weighted kappa agreement between [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 21 linked inside Pith

  1. [4]

    Leander Girrbach, Stephan Alaniz, Genevieve Smith, Trevor Darrell, and Zeynep Akata

    Mllm- bench: Evaluating multimodal llms with per-sample criteria.2311.13951v3. Leander Girrbach, Stephan Alaniz, Genevieve Smith, Trevor Darrell, and Zeynep Akata

    Pith/arXiv arXiv

  2. [5]

    Tianxing He, Jingyu Zhang, Tianle Wang, Sachin Kumar, Kyunghyun Cho, James Glass, and Yulia Tsvetkov

    Person- centric annotations of laion-400m: Auditing bias and its transfer to models.2510.03721v1. Tianxing He, Jingyu Zhang, Tianle Wang, Sachin Kumar, Kyunghyun Cho, James Glass, and Yulia Tsvetkov

    arXiv

  3. [7]

    Yerin Hwang, Dongryeol Lee, Kyungmin Min, Taeg- wan Kang, Yong il Kim, and Kyomin Jung

    Bias in gender bias benchmarks: How spurious fea- tures distort evaluation.2509.07596v2. Yerin Hwang, Dongryeol Lee, Kyungmin Min, Taeg- wan Kang, Yong il Kim, and Kyomin Jung

    arXiv

  4. [8]

    Sepehr Janghorbani and Gerard de Melo

    Fooling the lvlm judges: Visual biases in lvlm-based evaluation.2505.15249v1. Sepehr Janghorbani and Gerard de Melo

    arXiv

  5. [9]

    Yukun Jiang, Zheng Li, Xinyue Shen, Yugeng Liu, Michael Backes, and Yang Zhang

    Multi- modal bias: Introducing a framework for stereotypi- cal bias assessment beyond gender and race in vision language models.Preprint, arXiv:2303.12734. Yukun Jiang, Zheng Li, Xinyue Shen, Yugeng Liu, Michael Backes, and Yang Zhang

    Pith/arXiv arXiv

  6. [10]

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy

    ModSCAN: Measuring stereotypical bias in large vision-language models from vision and language modalities.2410.06967v1. Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy

    Pith/arXiv arXiv

  7. [11]

    Tony Lee, Haoqin Tu, Chi Heem Wong, Zijun Wang, Siwei Yang, Yifan Mai, Yuyin Zhou, Cihang Xie, and Percy Liang

    Pick-a- pic: An open dataset of user preferences for text-to- image generation.Preprint, arXiv:2305.01569. Tony Lee, Haoqin Tu, Chi Heem Wong, Zijun Wang, Siwei Yang, Yifan Mai, Yuyin Zhou, Cihang Xie, and Percy Liang

    Pith/arXiv arXiv

  8. [12]

    Christoph Leiter, Yuki M

    Ahelm: A holistic evaluation of audio-language models.2508.21376v2. Christoph Leiter, Yuki M. Asano, Margret Keuper, and Steffen Eger

    Pith/arXiv arXiv

  9. [13]

    Christoph Leiter and Steffen Eger

    Croc: Evaluating and training t2i metrics with pseudo- and human-labeled contrastive robustness checks.arXiv preprint arXiv:2505.11314. Christoph Leiter and Steffen Eger

    Pith/arXiv arXiv

  10. [14]

    InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA

    PrExMe! large scale prompt exploration of open source LLMs for machine translation and summarization evaluation. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA. Association for Computational Lin- guistics. Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang...

    2024

  11. [15]

    Hanjun Luo, Haoyu Huang, Ziye Deng, Xinfeng Li, Hewei Wang, Yingbin Jin, Yang Liu, Wenyuan Xu, and Zuozhu Liu

    Evaluating text-to-visual gen- eration with image-to-text generation.Preprint, arXiv:2404.01291. Hanjun Luo, Haoyu Huang, Ziye Deng, Xinfeng Li, Hewei Wang, Yingbin Jin, Yang Liu, Wenyuan Xu, and Zuozhu Liu

    Pith/arXiv arXiv

  12. [16]

    Debbie S

    Bigbench: A unified bench- mark for evaluating multi-dimensional social biases in text-to-image models.2407.15240v6. Debbie S. Ma and Joshua Correll

    Pith/arXiv arXiv

  13. [17]

    2509.23499v1

    Multi-modal data spec- trum: Multi-modal datasets are multi-dimensional. 2509.23499v1. P. McIntosh. 1988.White Privilege and Male Privilege: A Personal Account of Coming to See Correspon- dences Through Work in Women’s Studies. Working paper (Wellesley College. Center for Research on Women). Wellesley College, Center for Research on Women. Ninareh Mehrabi...

    arXiv 1988

  14. [18]

    A survey on bias and fairness in machine learning.Preprint, arXiv:1908.09635. Diego A. B. Moreira, Alef Iury Ferreira, Jhessica Silva, Gabriel Oliveira dos Santos, Luiz Pereira, João Medrado Gondim, Gustavo Bonil, Helena Maia, Nádia da Silva, Simone Tiemi Hashiguti, Je- fersson A. dos Santos, Helio Pedrini, and Sandra Avila

    Pith/arXiv arXiv 1908

  15. [19]

    Rahul Nair, Bhanu Tokas, and Hannah Kerner

    Fairpivara: Reducing and assessing biases in clip-based multimodal models.Preprint, arXiv:2409.19474. Rahul Nair, Bhanu Tokas, and Hannah Kerner

    Pith/arXiv arXiv

  16. [20]

    Aravind Narayanan, Vahid Reza Khazaie, and Shaina Raza

    A woman with a knife or a knife with a woman? measuring directional bias amplification in image captions.2503.07878v4. Aravind Narayanan, Vahid Reza Khazaie, and Shaina Raza

    Pith/arXiv arXiv

  17. [21]

    Vishal Narnaware, Ashmal Vayani, Rohit Gupta, Sirnam Swetha, and Mubarak Shah

    Bias in the picture: Benchmarking vlms with social-cue news images and llm-as-judge assess- ment.2509.19659v1. Vishal Narnaware, Ashmal Vayani, Rohit Gupta, Sirnam Swetha, and Mubarak Shah

    arXiv

  18. [22]

    2502.08779v2

    Sb-bench: Stereo- type bias benchmark for large multimodal models. 2502.08779v2. Chahat Raj, Bowen Wei, Aylin Caliskan, Antonios Anas- tasopoulos, and Ziwei Zhu

    arXiv

  19. [23]

    2505.22897v1

    Vignette: Socially grounded bias evaluation for vision-language models. 2505.22897v1. Adrienne Rich

    Pith/arXiv arXiv

  20. [24]

    Huichan Seo, Sieun Choi, Minki Hong, Yi Zhou, Junseo Kim, Lukman Ismaila, Naome Etori, Mehul Agarwal, Zhixuan Liu, Jihie Kim, and Jean Oh

    Deconstructing bias: A multifaceted framework for diagnosing cultural and compositional inequities in text-to-image generative models.2505.01430v1. Huichan Seo, Sieun Choi, Minki Hong, Yi Zhou, Junseo Kim, Lukman Ismaila, Naome Etori, Mehul Agarwal, Zhixuan Liu, Jihie Kim, and Jean Oh

    Pith/arXiv arXiv

  21. [25]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y

    Expos- ing blindspots: Cultural bias evaluation in generative image models.2510.20042v1. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo

    Pith/arXiv arXiv

  22. [26]

    Yifan Shen, Yangyang Shu, Hye young Paik, and Yulei Sui

    Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models.Preprint, arXiv:2402.03300. Yifan Shen, Yangyang Shu, Hye young Paik, and Yulei Sui

    Pith/arXiv arXiv

  23. [27]

    Preprint, arXiv:2510.00796

    Metalogic: Robustness evaluation of text- to-image models via logically equivalent prompts. Preprint, arXiv:2510.00796. Eric Slyman, Mehrab Tanjim, Kushal Kafle, and Stefan Lee

    arXiv

  24. [28]

    Yixin Wan, Arjun Subramonian, Anaelia Ovalle, Zongyu Lin, Ashima Suvarna, Christina Chance, Hritik Bansal, Rebecca Pattichis, and Kai-Wei Chang

    Calibrating mllm-as-a-judge via multi- modal bayesian prompt ensembles.2509.08777v1. Yixin Wan, Arjun Subramonian, Anaelia Ovalle, Zongyu Lin, Ashima Suvarna, Christina Chance, Hritik Bansal, Rebecca Pattichis, and Kai-Wei Chang

    Pith/arXiv arXiv

  25. [29]

    2404.01030v3

    Survey of bias in text-to-image generation: Definition, evaluation, and mitigation. 2404.01030v3. Youze Wang, Zijun Chen, Ruoyu Chen, Shishen Gu, Wenbo Hu, Jiayang Liu, Yinpeng Dong, Hang Su, Jun Zhu, Meng Wang, and Richang Hong

    Pith/arXiv arXiv

  26. [30]

    2506.12336v2

    Understanding and benchmarking the trustworthi- ness in multimodal llms for video understanding. 2506.12336v2. Wenqian Ye, Guangtao Zheng, Yunsheng Ma, Xu Cao, Bolin Lai, James M. Rehg, and Aidong Zhang

    arXiv

  27. [31]

    A Appendix A.1 Taxonomy Animals.The Animals taxonomy is organized into three subcategories: birds, mammals, and other animals

    Mm-spubench: Towards better understanding of spu- rious biases in multimodal llms.2406.17126v1. A Appendix A.1 Taxonomy Animals.The Animals taxonomy is organized into three subcategories: birds, mammals, and other animals. Each subcategory defines non- prototypical to prototypical pairs based on per- ceptual familiarity rather than biological correct- nes...

    arXiv

  28. [2022]

    Clipscore: A reference- free evaluation metric for image captioning.Preprint, arXiv:2104.08718. Yusuke Hirota, Ryo Hachiuma, Boyi Li, Ximing Lu, Michael Ross Boone, Boris Ivanovic, Yejin Choi, Marco Pavone, Yu-Chiang Frank Wang, Noa Garcia, Yuta Nakashima, and Chao-Han Huck Yang

    Pith/arXiv arXiv

  29. [2023]

    Preprint, arXiv:2208.07316

    Menli: Robust evaluation metrics from natural language inference. Preprint, arXiv:2208.07316. Juan Manuel Contreras

    Pith/arXiv arXiv

  30. [2024]

    A critical look at meta-evaluating summarisation evalu- ation metrics.2409.19507v1. Wentao Ge, Shunian Chen, Guiming Hardy Chen, Jun- ying Chen, Zhihong Chen, Nuo Chen, Wenya Xie, Shuo Yan, Chenghao Zhu, Ziyue Lin, Song Dingjie, Xidong Wang, Anningzhe Gao, Zhang Zhiyi, Jian- quan Li, Xiang Wan, and Benyou Wang

    Pith/arXiv arXiv

  31. [2025]

    2509.07050v1

    Automated evaluation of gender bias across 13 large multimodal models. 2509.07050v1. Xiang Dai, Sarvnaz Karimi, and Biaoyan Fang

    Pith/arXiv arXiv