pith. sign in

arxiv: 2605.24702 · v1 · pith:KGHMTST3new · submitted 2026-05-23 · 💻 cs.CV

Do Image-Text Metrics Respect Semantic Invariances?

Amit Agarwal , Hitesh Laxmichand Patel , Meizhu Liu , Jyotika Singh , Karan Dua , Hansa Meghwani , Matthew Rowe , Michael Avendi
show 4 more authors
Yassi Abbasi Tao Sheng Sujith Ravi Dan Roth
This is my paper

Pith reviewed 2026-06-30 13:13 UTC · model grok-4.3

classification 💻 cs.CV
keywords image-text metricssemantic invariancereference-free evaluationmetric sensitivityperturbation analysiscaption scoring robustness
0
0 comments X

The pith

Reference-free image-to-text evaluators shift scores 6-9% under semantics-preserving edits like spatial flips and rephrasings.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether five popular reference-free metrics for scoring image-caption alignment stay stable when meaning is unchanged. It applies three families of perturbations—spatial edits such as flips and light rotations, object changes such as scale adjustments, and socio-linguistic phrasing variations with length-matched controls—to slices of detection and caption datasets. The metrics register average score movements of 6-9 percent, enough to reverse rankings between systems separated by only 0.7 percent in up to 37 percent of cases, especially under spatial changes. A small human study finds that annotators view the perturbed pairs as equally correct, indicating the score shifts arise from metric behavior rather than semantic difference. The authors also introduce a post-hoc invariance-calibrated scoring adjustment that roughly halves median absolute sensitivity while preserving correlation with other evaluators.

Core claim

Reference-free image-to-text evaluators exhibit consistent non-semantic sensitivities, with benign spatial edits and simple phrasing changes shifting scores by approximately 6-9% on average and causing ranking flips in up to 37% of cases for closely performing systems, while a human study confirms that annotators judge perturbed pairs as equally correct.

What carries the argument

An invariance probe that applies semantics-preserving perturbations along spatial, object, and socio-linguistic axes to reference-free evaluators, followed by a post-hoc invariance-calibrated scoring adjustment.

If this is right

  • Small performance gaps between systems can be overturned by minor edits that preserve meaning.
  • Spatial perturbations produce the largest effects on both absolute scores and relative rankings.
  • Invariance-calibrated scoring reduces median absolute sensitivity by roughly half.
  • The calibrated scores retain correlation with other learned caption evaluators.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Metric developers could add invariance probes as a routine check when releasing new evaluators.
  • The same perturbation approach might expose similar non-semantic sensitivities in text-to-image generation metrics.
  • Current metrics may be overweighting superficial layout cues relative to core semantic content.

Load-bearing premise

The chosen perturbations preserve the underlying semantic correctness of the image-caption pairs and do not introduce new meaning or errors.

What would settle it

A larger human study in which annotators consistently rate the perturbed image-caption pairs as less correct than the originals would show that the observed score shifts track real semantic change rather than metric artifact.

Figures

Figures reproduced from arXiv: 2605.24702 by Amit Agarwal, Dan Roth, Hansa Meghwani, Hitesh Laxmichand Patel, Jyotika Singh, Karan Dua, Matthew Rowe, Meizhu Liu, Michael Avendi, Sujith Ravi, Tao Sheng, Yassi Abbasi.

Figure 1
Figure 1. Figure 1: Pipeline Overview. Starting from curated single-object image–caption pairs, we construct matched spatial, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: RQ1a (Vertical flips). Median %∆ (95% CIs) across evaluators and datasets; positive values indicate higher post-flip scores (orientation sensitivity). while keeping Spearman correlation with learned caption evaluators (UMIC, FLEUR) within 0.01 of the uncalibrated metric. We report both sensitivity reductions and changes in RRF. 4 Experiments We structure our experiments to find answers to targeted question… view at source ↗
Figure 3
Figure 3. Figure 3: RQ1b (Repositioning). Median %∆ (BR−TL) across evaluators and datasets (95% CIs). Repositioning induces sizable shifts (≈7–9%); BR>TL. COCO OpenImages Objects365 0 2 4 6 Median Relative Change (% ) CLIPScore PAC-S UMIC FLEUR Judge [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 7
Figure 7. Figure 7: RQ3a (Cultural). Median %∆ vs. neutral on COCO across evaluators (95% CIs). Consistent ordering with larger shifts in embedding-similarity metrics. dian change (≈ -7%), while American/European are mildly positive (≈ +1%); UMIC/FLEUR and the JUDGE track the same order at lower magni￾tude. Because the image content is unchanged, these shifts reflect sensitivity to non-visual cultural framing rather than scen… view at source ↗
Figure 8
Figure 8. Figure 8: RQ3b (Economic). COCO: median %∆ vs. neutral across evaluators (95% CIs). Cheap is mildly positive, while expensive is consistently negative. 5.4 RQ4: Cross-Dataset behavior Patterns from RQ1–RQ3 persist across evalua￾tors and transfer to three caption-evaluation suites (Flickr8k-CF, Pascal-50S, COMPOSITE). On these external corpora, vertical flips remain positive for all evaluator–corpus combinations: med… view at source ↗
Figure 10
Figure 10. Figure 10: RQ4 (External-Economic). Median %∆ vs. neutral on external corpora (avg. cheap/expensive). Cheap is mildly positive while expensive is negative. -6%. (iii) Multi-object pilot. A 200-image MS￾COCO probe with 2-4 prominent objects (Ap￾pendix A.10) shows that spatial effect directions match the single-object slice, with slightly atten￾uated magnitudes, higher variance, and no qual￾itative reversals. We there… view at source ↗
Figure 11
Figure 11. Figure 11: RQ5 (Flip risk). Median RRF (%) by perturbation family (95% CIs). Ranking instability is highest under spatial probes, especially repositioning. Spatial Societal 0 1 2 3 4 5 6 7 Median absolute sensitivity Before After 0.0 0.2 0.4 0.6 0.8 1.0 ¸ 0.77 0.78 0.79 0.80 0.81 0.82 Spearman correlation UMIC FLEUR [PITH_FULL_IMAGE:figures/full_fig_p009_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: RQ6 (Calibration). Sensitivity by axis before calibration (left) and utility retention via dev￾split Spearman vs. λ (right); λ ⋆ chosen under ϵ=0.01 w.r.t. UMIC/FLEUR. invert on roughly one in three evaluation pairs un￾der benign, semantics-preserving changes, most acutely for object repositioning. 5.6 RQ6: Invariance-calibrated scoring We apply invariance-calibrated scoring to down￾weight non-semantic se… view at source ↗
Figure 13
Figure 13. Figure 13: Category composition of curated slices. Share (%) of images per harmonized category in COCO/OpenImages/Objects365 [PITH_FULL_IMAGE:figures/full_fig_p016_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Gender modifiers over COCO. Median %∆ vs. neutral with 95% BCa CIs for all evaluators. Practical significance and reporting. Be￾cause phrasing choices induce measurable score shifts, we recommend (i) fixing evaluator prompts/templates, (ii) reporting modifier-wise deltas alongside macro scores, and (iii) including the fairness-card summary for socio-linguistic [PITH_FULL_IMAGE:figures/full_fig_p023_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Emotion modifiers over COCO. Median %∆ vs. neutral with 95% BCa CIs for all evaluators. Evaluator Flickr8k-CF Pascal-50S COMPOSITE CLIPScore 7.2% [5.3,9.1] 7.4% [5.5,9.3] 7.0% [5.2,8.9] PAC-S 6.3% [4.5,8.0] 6.5% [4.7,8.3] 6.2% [4.5,7.9] UMIC 4.1% [2.7,5.6] 4.9% [3.4,6.4] 3.8% [2.4,5.2] FLEUR 3.2% [1.9,4.5] 3.9% [2.5,5.3] 3.8% [2.4,5.2] Judge 3.2% [1.9,4.4] 2.5% [1.3,3.7] 3.1% [1.8,4.3] [PITH_FULL_IMAGE:f… view at source ↗
read the original abstract

Reference-free image-to-text evaluators are now standard for scoring image-caption alignment, yet it is unclear whether they respect semantic invariances. We present an invariance probe on five popular evaluators (CLIPScore, PAC-S, UMIC, FLEUR, and a deterministic LLM judge) under semantics-preserving perturbations along three axes -- spatial (flips, context-preserving repositioning, light rotations), object (scale, category), and socio-linguistic framing (cultural/economic adjectives with neutral and length-matched controls). Across curated slices of three detection datasets and three caption evaluation suites, we find consistent non-semantic sensitivities, where benign spatial edits and simple phrasing changes shift scores by $\approx$6--9\% on average, and for systems separated by just 0.7\%, these shifts can cause ranking flips in up to $\sim$37\% of cases, particularly under spatial changes. A small human study also supports this finding and confirms that annotators generally judge perturbed pairs as equally correct, so these shifts reflect metric behavior rather than semantic change. We further propose invariance-calibrated scoring, a post-hoc adjustment that roughly halves median absolute sensitivity while retaining correlation with learned caption evaluators.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims that five reference-free image-to-text evaluators (CLIPScore, PAC-S, UMIC, FLEUR, and a deterministic LLM judge) fail to respect semantic invariances. Across curated slices of three detection datasets and three caption suites, semantics-preserving perturbations along spatial (flips, repositioning, light rotations), object (scale/category), and socio-linguistic axes produce average score shifts of ≈6–9% and ranking flips in up to ∼37% of cases for systems separated by only 0.7%. A small human study is cited to confirm that the perturbed pairs remain equally correct, and the authors propose an invariance-calibrated post-hoc scoring adjustment that halves median absolute sensitivity while preserving correlation with learned evaluators.

Significance. If the central empirical findings hold after fuller validation, the work would be significant for multimodal evaluation: it demonstrates a systematic limitation in metrics now standard for image-caption alignment and model selection. The multi-dataset design, direct measurement of ranking instability, and the practical invariance-calibrated scoring proposal are concrete strengths. The study is free of circular derivations or fitted parameters and grounds claims in observed score changes rather than self-referential predictions.

major comments (2)
  1. [Human Study] Human Study section: the claim that observed score shifts reflect metric non-invariance rather than semantic change rests entirely on the human study confirming that perturbed pairs remain equally correct. The abstract and methods describe the study only as 'small' and provide no numbers for annotators, items per perturbation type, selection protocol, or agreement statistics; these details are load-bearing for the central interpretation.
  2. [Results] Results section (ranking-flip analysis): the reported ∼37% flip rate for systems 0.7% apart requires explicit specification of how the 0.7% separation threshold is computed (pairwise vs. global), whether the rate is averaged over all perturbation types or reported per axis, and whether confidence intervals or significance tests accompany the figure. This directly supports the practical claim about ranking instability.
minor comments (2)
  1. [Abstract] Abstract: the three detection datasets and three caption suites are not named; listing them would aid reproducibility and context.
  2. Perturbation descriptions: the exact implementation details (e.g., rotation angles, scale factors, adjective lists for socio-linguistic framing) should be moved from supplementary material into the main text or a dedicated table for clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the two major comments point-by-point below and will incorporate clarifications and expansions in the revised manuscript.

read point-by-point responses

  1. Referee: [Human Study] Human Study section: the claim that observed score shifts reflect metric non-invariance rather than semantic change rests entirely on the human study confirming that perturbed pairs remain equally correct. The abstract and methods describe the study only as 'small' and provide no numbers for annotators, items per perturbation type, selection protocol, or agreement statistics; these details are load-bearing for the central interpretation.

    Authors: We agree that the current description of the human study is insufficiently detailed for the interpretive weight it carries. In the revision we will expand the Human Study section (and update the abstract/methods summary) to report the exact number of annotators, the number of items evaluated per perturbation axis, the item selection and presentation protocol, and inter-annotator agreement statistics. These additions will make the evidence that perturbed pairs remain semantically equivalent fully transparent. revision: yes

  2. Referee: [Results] Results section (ranking-flip analysis): the reported ∼37% flip rate for systems 0.7% apart requires explicit specification of how the 0.7% separation threshold is computed (pairwise vs. global), whether the rate is averaged over all perturbation types or reported per axis, and whether confidence intervals or significance tests accompany the figure. This directly supports the practical claim about ranking instability.

    Authors: We will revise the ranking-flip subsection to state that the 0.7% separation is computed from pairwise differences between the original (unperturbed) scores of the two systems, to indicate whether the 37% figure is an aggregate or is broken down by perturbation axis, and to add bootstrap confidence intervals (or appropriate significance tests) around the flip-rate estimates. These changes will make the practical implication of ranking instability fully reproducible and statistically grounded. revision: yes

Circularity Check

0 steps flagged

No circularity: direct empirical measurement with external human validation

full rationale

The paper performs an empirical probe by applying fixed perturbations (spatial, object, socio-linguistic) to existing image-caption datasets and recording score changes on five off-the-shelf evaluators. The central claim that observed shifts reflect metric non-invariance rests on a separate human study confirming semantics preservation, which is an independent data collection step rather than a self-referential definition or fitted parameter renamed as prediction. No equations, derivations, self-citation chains, or ansatzes are invoked to justify the results; the invariance-calibrated scoring is described only as a post-hoc adjustment without any reduction to the input measurements themselves. The analysis is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim depends on the premise that the perturbations preserve semantics, which is treated as a domain assumption validated by the human study rather than derived from first principles.

axioms (1)
  • domain assumption Perturbations along the three axes preserve semantic meaning and correctness of image-caption pairs
    This premise allows score shifts to be attributed to metric behavior rather than semantic change; it is invoked to interpret all results and is supported by the human study mentioned in the abstract.

pith-pipeline@v0.9.1-grok · 5776 in / 1312 out tokens · 36633 ms · 2026-06-30T13:13:39.515407+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 5 canonical work pages

  1. [1]

    InInternational conference on ma- chine learning, pages 19730–19742

    Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. InInternational conference on ma- chine learning, pages 19730–19742. PMLR. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. InInterna...

    2022

  2. [2]

    Kazuki Matsuda, Yuiga Wada, and Komei Sugiura

    Cobra effect in reference-free image caption- ing metrics.arXiv preprint arXiv:2402.11572. Kazuki Matsuda, Yuiga Wada, and Komei Sugiura

    work page arXiv

  3. [3]

    InProceedings of the Asian Conference on Computer Vision, pages 3570–3586

    Deneb: A hallucination-robust automatic eval- uation metric for image captioning. InProceedings of the Asian Conference on Computer Vision, pages 3570–3586. Hansa Meghwani, Amit Agarwal, Priyaranjan Pat- tnayak, Hitesh Laxmichand Patel, and Srikant Panda

  4. [4]

    InProceedings of the 63rd Annual Meeting of the Association for Com- putational Linguistics (Volume 6: Industry Track), pages 1013–1026, Vienna, Austria

    Hard negative mining for domain-specific re- trieval in enterprise systems. InProceedings of the 63rd Annual Meeting of the Association for Com- putational Linguistics (Volume 6: Industry Track), pages 1013–1026, Vienna, Austria. Association for Computational Linguistics. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method f...

    work page arXiv 2002

  5. [5]

    Sara Sarto, Nicholas Moratelli, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara

    Springer. Sara Sarto, Nicholas Moratelli, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2025. Positive- augmented contrastive learning for vision-and- language evaluation and training.Preprint, arXiv:2410.07336. Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Jing Li, Xiangyu Zhang, and Jian Sun

    work page arXiv 2025

  6. [6]

    InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 8425–8434

    Objects365: A large-scale, high-quality dataset for object detection. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 8425–8434. Harman Singh, Pengchuan Zhang, Qifan Wang, Mengjiao Wang, Wenhan Xiong, Jingfei Du, and Yu Chen. 2023. Coarse-to-fine contrastive learning in image-text-graph space for improved vision-language c...

    2023

  7. [7]

    InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 883– 902, Suzhou (China)

    Can LLMs narrate tabular data? an evaluation framework for natural language representations of text-to-SQL system outputs. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 883– 902, Suzhou (China). Association for Computational Linguistics. Brandon Abreu Smith, Miguel Farinha, Siob- han Macken...

    2025

  8. [8]

    InProceed- ings of the 37th International Conference on Neu- ral Information Processing Systems, pages 15903– 15935

    Imagereward: learning and evaluating human preferences for text-to-image generation. InProceed- ings of the 37th International Conference on Neu- ral Information Processing Systems, pages 15903– 15935. Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. 2018. Attngan: Fine-grained text to image generation with atten...

    2018

  9. [9]

    InProceedings of the 32nd ACM International Conference on Multimedia, MM ’24, page 866–875

    Hicescore: A hierarchical metric for image captioning evaluation. InProceedings of the 32nd ACM International Conference on Multimedia, MM ’24, page 866–875. ACM. Jing Zhang, Jing Huang, Sheng Jin, and Shuicheng Lu

  10. [10]

    ArXivabs/2311.01361(2023)

    Vision-language models for vision tasks: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence. To appear. Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. 2023. Gpt-4v (ision) as a generalist evaluator for vision-language tasks. arXiv preprint arXiv:2311.0...

    work page arXiv 2023

  11. [11]

    There is a [object]

    Understanding and evaluating racial biases in image captioning. InProceedings of the IEEE/CVF international conference on computer vision, pages 14830–14840. Amir Zur, Elisa Kreiss, Karel D’Oosterlinck, Christo- pher Potts, and Atticus Geiger. 2024. Updating clip to prefer descriptions over captions.Preprint, arXiv:2406.09458. A Appendix A.1 Extended Rela...

    work page arXiv 2024

  12. [12]

    Instantiate probes.Define a small set of semantics-preserving transforms in T ⋆ and gen- erate paired variants (x(t), c(t)) on a held-out development set

  13. [13]

    Measure sensitivity.Compute ∆S(x, c;T ⋆) = mediant∈T ⋆ S(x(t), c(t))−S(x, c) per item, matching the estimator in Sec.subsection 3.5

  14. [14]

    UMIC/FLEUR) used in the main calibration; either keep uniform axis weights or use the sensitivity-proportional scheme of Appendix A.7

    Fit weight.Add T ⋆ to T and re-run the λ- sweep under the same correlation-preservation constraint ( ϵ=0.01 vs. UMIC/FLEUR) used in the main calibration; either keep uniform axis weights or use the sensitivity-proportional scheme of Appendix A.7

  15. [15]

    This recipe makes the mitigation test-suite-style: as new invariance axes are documented, they can be added incrementally

    Deploy.At inference, calibration is a constant-time post-processing subtraction on top of the base metric ( ˆS(x, c) =S(x, c)− λP T wT ∆S(x, c;T) ); the estimated ∆ terms are precomputed once on the dev sweep. This recipe makes the mitigation test-suite-style: as new invariance axes are documented, they can be added incrementally. The trade-off is explici...