Pith. sign in

REVIEW 4 major objections 5 minor 63 references

MASS: Overcoming Language Bias in Image-Text Matching

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Subtracting a black-image baseline from token likelihoods cancels language bias in image-text matching.

desk verdict MASS debiasing works empirically and is honestly framed, but the derivation of the text-only marginal is wrong and the null-image surrogate is unvalidated. read the letter →

arxiv 2501.11469 v1 pith:AAITS4U4 submitted 2025-01-20 cs.CV cs.LG

Jiwan Chung , Seungwon Lim , Sangkyu Lee , Youngjae Yu This is my paper · ORCID
classification cs.CVcs.LG
keywords languagebiasimage-textmatchingpointwisemutualinformationvision-languagemodelsinference-timedebiasingWinogroundcompositionalitymultimodalassociationscore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Pretrained vision-language models that score captions by token likelihood have a known flaw: they lean on language priors—'a tomato is red'—and sometimes ignore the image. This paper introduces Multimodal ASsociation Score (MASS), an inference-time re-scoring that subtracts the text-only marginal likelihood, approximated by the model's output for a black image, from the image-conditioned token likelihood. MASS is an estimate of pointwise mutual information between the image and each text token, and the paper shows it reduces color, counting, and gender bias while preserving, even improving, compositionality on Winoground and SVO-Probes. No additional training is required, and MASS applies to any autoregressive vision-language model.

What carries the argument

MASS (Multimodal ASsociation Score) is the central mechanism, defined by Eq. 9 as the per-token log-ratio $\log \frac{p_{\bar\theta}(x_t | x_{<t}, c)}{p_{\bar\theta}(x_t | x_{<t}, c_\emptyset)}$, with $c_\emptyset$ a black-filled image. The black-image baseline is the load-bearing device: it gives a one-pass estimate of the text-only marginal, letting the score subtract the language prior without Monte Carlo sampling across many images. Averaging these per-token log-ratios over the sequence produces a debiased similarity score that can be plugged into any off-the-shelf autoregressive vision-language model.

What would settle it

Compute MASS on a set of 100 captions where language prior conflicts with visual evidence, and compare its ranking with a Monte Carlo estimate of pointwise mutual information using many natural images as the marginal; if the black-image score ranks true captions below rivals while the Monte Carlo version ranks them above, the null-image approximation is the failure point.

Watch

Extended reading notes

Core claim

On the paper's own terms, the log-likelihood a captioning model assigns to a caption decomposes into a linguistic component (the text prior) and an association component that measures genuine image-text correspondence. MASS separates the two by computing, per token, the log ratio of the image-conditioned token probability to the token probability given a black image, which the paper argues approximates the text-only marginal $p_{\bar\theta}(x_t | x_{<t})$. Averaged over the caption, this ratio is an estimate of pointwise mutual information; using it as the similarity metric instead of raw token likelihood removes the language prior. The paper shows that MASS outperforms token likelihood and CLIP-based scores on color debiasing, adversarial counting, and gender-balanced retrieval, and that on Winoground and SVO-Probes it improves groupscore and image-score, indicating that the debiasing does not come at the cost of linguistic compositionality.

Load-bearing premise

The load-bearing premise is that a black image makes the model produce the same token probabilities as it would with no visual information at all, so the subtraction removes the language prior rather than adding an arbitrary per-token constant.

Editorial extensions

If this is right

  • MASS turns any autoregressive vision-language model into a debiased image-text similarity scorer with only two forward passes per sample, no training, and no hyperparameters.
  • On adversarial counting foils (true captions with large numbers versus foils with small numbers), MASS raises OFA-large's pairwise ranking accuracy from 66.8 to 76.7, matching the best multitask baseline.
  • On Winoground, MASS lifts OFA-large's groupscore from 15.8 (token likelihood) to 20.3, with the gain concentrated on the No-Tag subset that isolates compositionality.
  • In gender-balanced retrieval on MS-COCO, MASS improves both text-to-image and image-to-text recall over token likelihood while keeping bias scores low, unlike debiasing baselines that trade recall for fairness.
  • The improvement transfers across backbone families: MASS raises groupscore for OFA, BLIP-2, and LLaVA, suggesting the language-prior subtraction generalizes beyond a single architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We would test the null-image shortcut directly: on a set of captions with a strong language prior (e.g., gray tomato), compare MASS ranking against a Monte Carlo marginal estimate averaged over many natural images; if the rankings diverge on high-bias items, the black-image approximation, not the PMI idea, is the fragile component.
  • The likelihood-ratio recipe may transfer to other tasks that suffer from language priors, such as hallucination detection in image captioning or debiasing text-to-image generation scores.
  • Because the method needs no retraining, it can serve as a cheap audit tool: applying MASS to a new VLM reveals where its token likelihoods are dominated by textual stereotypes rather than visual evidence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MASS, an inference-time score for image-text matching that subtracts a text-only marginal term from the token-level conditional likelihood of an autoregressive vision-language model, aiming to reduce language bias. The score is defined as the average per-token log-ratio of the image-conditioned likelihood to the likelihood under a black 'null image' surrogate, which is claimed to approximate the text-only marginal. Experiments cover color bias, counting, gender bias in retrieval, and the compositionality benchmarks Winoground and SVO-Probes, using OFA, BLIP-2, and LLaVA backbones. The central empirical claim is that MASS improves over raw token likelihood and CLIP-style baselines on these tasks without additional training.

Significance. If the empirical claim holds, the paper would offer a simple, training-free, hyperparameter-free recipe for reducing language bias in image-text matching, with potential practical value in retrieval and fairness-oriented applications. The paper is commendable for evaluating across multiple backbones, reporting a fine-grained Winoground category breakdown, and including qualitative examples. However, the theoretical framing of MASS as pointwise mutual information rests on an invalid Monte Carlo identity and an unvalidated null-image surrogate, and the closest prior method using null images and PMI is acknowledged but not compared experimentally. These issues leave the mechanism of the reported improvements unclear, even though the empirical results are suggestive and mostly consistent with the paper's claims. The manuscript is worth further consideration after the load-bearing validation and comparison gaps are addressed.

major comments (4)
  1. [§3.2, Eqs. (7)-(8)] The displayed Monte Carlo identity is mathematically incorrect. The marginal log-likelihood is log pθ(xt|x<t) = log ∫_C pθ(xt|x<t,c) p(c) dc, not ∫_C log pθ(xt|x<t,c) dc. Averaging log-conditionals as in Eq. (8) estimates E_c[log pθ(xt|x<t,c)], which by Jensen's inequality is not equal to log E_c[pθ(xt|x<t,c)], and the prior p(c) over images is missing. Since this derivation is presented as the motivation for the black-image surrogate in Eq. (9), the paper should either correct it or explicitly state that Eq. (9) is a heuristic and not a valid Monte Carlo estimate of the marginal.
  2. [§3.2, Eq. (9)] The central approximation that the black-filled image c∅ satisfies log pθ(xt|x<t,c∅) ≈ log pθ(xt|x<t) is asserted without evidence. This is load-bearing: if c∅ is out-of-distribution for a model trained on natural images, MASS is not pointwise mutual information but token likelihood shifted by per-token, per-prefix offsets log pθ(xt|x<t,c∅), and the debiasing effect has no theoretical grounding. The authors should validate the approximation empirically, for example by comparing pθ(·|x<t,c∅) against a Monte Carlo estimate using random natural images (with the corrected formula), and by testing sensitivity to the choice of null input (black, gray, noise, or blurred images).
  3. [§6, Related Work] Lin et al. (2024), which the paper explicitly acknowledges as sharing the null-image and PMI methodology, is never compared experimentally. Since the claim that MASS outperforms strong baselines is central to the paper, a direct comparison on at least Winoground and SVO-Probes (or a clearly stated reason why a direct comparison is infeasible) is needed. Without it, the incremental contribution over the closest prior method is not established.
  4. [Tables 1-4] No uncertainty quantification is reported for any of the main results. Several improvements are small in absolute terms, for example the OFA-large balanced counting gain of +1.5 in Table 1, the BLIP-2 Winoground group-score gain of +2.5 in Table 3, and the decreases in some BLIP-2/LLaVA rows. Bootstrapped confidence intervals or paired significance tests over the benchmark samples are necessary to assess whether these differences are reliable, even though the scoring functions themselves are deterministic.
minor comments (5)
  1. [§2.2, Eq. (3) and §3.1, Eq. (6)] The summation index is written as t<l, which would exclude the final token; the sums should run over t=1,...,l. Please correct the notation.
  2. [Appendix D, Table 5] The UNITER baseline appears in Table 5 but is not cited or introduced in the main text; please add the appropriate reference and clarify where it appears in the experiments.
  3. [§4.3] The phrase 'at an expanse of high degradation' should be 'at the expense of high degradation'; the same typo occurs in the following paragraph.
  4. [Author list and references] There are formatting issues in the author list ('Y oungjae Yu') and in the VAL SE dataset name ('V ALSE'); please clean these up.
  5. [§3.2] The phrase 'we discovered that using image input as a null image c∅ ... is a good alternative' is a strong claim; please replace it with a verifiable statement and provide the validation requested in the major comments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MASS is a fixed, benchmark-free function of off-the-shelf model likelihoods, and its gains are not fitted into existence.

full rationale

The derivation chain for MASS (Eqs. 5-9) is not circular. Eq. 6 defines the score as a per-token log-ratio of image-conditioned to text-only likelihood, and Eq. 9 substitutes a black-image conditional for the unavailable marginal. No parameter is tuned on any evaluation benchmark: the same frozen score is applied to the color, counting, gender, Winoground, and SVO-Probes tests, so the reported improvements cannot be an artifact of fitting the method to the test labels. The load-bearing step is the identification of p_theta(xt|x<t,c_empty) with p_theta(xt|x<t). This is an empirical approximation, not a definitional identity, and the paper's own derivation of the marginal is mathematically invalid: Eq. 7 writes log p(x) as an integral of log-conditionals, whereas the correct marginal is the log of an integral of conditionals weighted by p(c). That is a correctness and validation gap, not circularity: the score would be a poor PMI estimate if the surrogate fails, but the score itself is not defined in terms of the benchmarks it is claimed to predict. The only self-citation (Yu et al. 2023, in the introduction) is non-load-bearing context about CLIP-score uses, and the paper explicitly acknowledges the closely related null-image PMI method of Lin et al. (2024), so no load-bearing premise is smuggled in by self-citation. The Broader Impact section candidly notes that MASS 'relies on the output of visual-language models' and cannot audit hidden model biases, a stated limitation rather than a circular step.

Assumptions & free parameters 0 free parameters · 3 assumptions · 1 invented entities

The method introduces no fitted numeric parameters, so free_parameters is empty. The central assumption is that a black image behaves like 'no image'; this is an ad hoc surrogate used to estimate the text-only marginal. The paper also assumes captioning model likelihoods are well-calibrated and that averaging token PMI yields a valid sequence score. The null image has no independent evidence outside the paper, and the same construct appears in Lin et al. 2024.

assumptions (3)
  • ad hoc to paper The black image c∅ conditional pθ(xt|x<t,c∅) approximates the text-only marginal pθ(xt|x<t).
    Eq 9 replaces the marginal in the PMI denominator with a single black-image forward pass; the paper calls this a 'discovered' alternative and provides no validation against Monte Carlo averaging.
  • domain assumption Autoregressive captioning likelihoods are well-calibrated estimates of the true conditional p(x|c).
    MASS treats pθ(xt|x<t,c) as a faithful proxy for the true image-text conditional probability throughout Sec 3.
  • domain assumption Averaging token-level PMI over the sequence gives a valid image-text similarity score.
    Eq 6 defines MASS as a mean over tokens without discussing length normalization, function words, or token weighting.
invented entities (1)
  • Null image c∅ (black-filled image)
    purpose: Acts as a surrogate for 'no image' to estimate the text-only marginal in the PMI denominator.
    Introduced in Sec 3.2 Eq 8-9 without empirical validation of the approximation; the same construct appears in Lin et al. 2024, which is cited but not compared experimentally.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MASS: Overcoming Language Bias in Image-Text Matching." pith.science (2026). https://pith.science/paper/AAITS4U4

@misc{pith2026250111469,
  author       = {Pith},
  title        = {Pith review of: MASS: Overcoming Language Bias in Image-Text Matching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AAITS4U4}},
  note         = {Machine review of arXiv:2501.11469}
}
read the original abstract

Pretrained visual-language models have made significant advancements in multimodal tasks, including image-text retrieval. However, a major challenge in image-text matching lies in language bias, where models predominantly rely on language priors and neglect to adequately consider the visual content. We thus present Multimodal ASsociation Score (MASS), a framework that reduces the reliance on language priors for better visual accuracy in image-text matching problems. It can be seamlessly incorporated into existing visual-language models without necessitating additional training. Our experiments have shown that MASS effectively lessens language bias without losing an understanding of linguistic compositionality. Overall, MASS offers a promising solution for enhancing image-text matching performance in visual-language models.

Figures

Figures reproduced from arXiv: 2501.11469 by the authors.

Figure 1
Figure 1. Captions retrieved with each method given the im [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) Given an image of a girl playing tennis, the visual-language model falsely retrieves captions describing the [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Data samples and experimental results from our [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The Pareto frontier of recall-bias trade-off in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Top 5 COCO Captions image-to-text retrieval results, sorted by decreasing retrieval score. The token likelihood score [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: A qualitative sample from our Winoground (Thrush et al. 2022) experiment. The ground-truth image-text pairs are [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: failure case analysis [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 30 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Agrawal, A.; Batra, D.; and Parikh, D. 2016. Analyzing the Behavior of Visual Question Answering Models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 1955--1960

    work page 2016

  4. [4]

    Agrawal, A.; Batra, D.; Parikh, D.; and Kembhavi, A. 2018. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, 4971--4980

    work page 2018

  5. [5]

    Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 23716--23736

    2022

  6. [6]

    S.; and Muzaffar, A

    Anwar, S.; Tahir, M.; Li, C.; Mian, A.; Khan, F. S.; and Muzaffar, A. W. 2020. Image colorization: A survey and dataset. arXiv preprint arXiv:2008.10774

    arXiv 2020

  7. [7]

    Beaumont, R. 2022. Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them. GitHub

    work page 2022

  8. [8]

    U.; and Kahembwe, E

    Birhane, A.; Prabhu, V. U.; and Kahembwe, E. 2021. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963

    arXiv 2021

Show all 63 references
  1. [9]

    Chen, X.; Fang, H.; Lin, T.-Y.; Vedantam, R.; Gupta, S.; Doll \'a r, P.; and Zitnick, C. L. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325

    2015 arXiv

  2. [10]

    E.; et al

    Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ quality, March 2023. URL https://lmsys. org/blog/2023-03-30-vicuna, 3(5)

    2023

  3. [11]

    Davidson, T.; Bhattacharya, D.; and Weber, I. 2019. Racial bias in hate speech and abusive language detection datasets. arXiv preprint arXiv:1905.12516

    2019 arXiv

  4. [12]

    de Vassimon Manela, D.; Errington, D.; Fisher, T.; van Breugel, B.; and Minervini, P. 2021. Stereotype and skew: Quantifying gender bias in pre-trained and fine-tuned language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computat...

    2021

  5. [13]

    Diwan, A.; Berry, L.; Choi, E.; Harwath, D.; and Mahowald, K. 2022. Why is winoground hard? investigating failures in visuolinguistic compositionality. arXiv preprint arXiv:2211.00768

    2022 arXiv

  6. [14]

    Elazar, Y.; Zhang, H.; Goldberg, Y.; and Roth, D. 2021. Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 10486--10500

    2021

  7. [15]

    Garcia, N.; Hirota, Y.; Wu, Y.; and Nakashima, Y. 2023. Uncurated image-text datasets: Shedding light on demographic bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6957--6966

    2023

  8. [16]

    Goyal, Y.; Khot, T.; Summers-Stay, D.; Batra, D.; and Parikh, D. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, 6904--6913

    2017

  9. [17]

    A.; Burns, K.; Saenko, K.; Darrell, T.; and Rohrbach, A

    Hendricks, L. A.; Burns, K.; Saenko, K.; Darrell, T.; and Rohrbach, A. 2018. Women also snowboard: Overcoming bias in captioning models. In Proceedings of the European conference on computer vision (ECCV), 771--787

    2018

  10. [18]

    A.; and Nematzadeh, A

    Hendricks, L. A.; and Nematzadeh, A. 2021. Probing Image-Language Transformers for Verb Understanding. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 3635--3644

    2021

  11. [19]

    Hessel, J.; Holtzman, A.; Forbes, M.; Le Bras, R.; and Choi, Y. 2021. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 7514--7528

    2021

  12. [20]

    Honnibal, M.; and Montani, I. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 7(1): 411--420

    2017

  13. [21]

    Karpathy, A.; and Fei-Fei, L. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, 3128--3137

    2015

  14. [22]

    Lee, K.-H.; Chen, X.; Hua, G.; Hu, H.; and He, X. 2018. Stacked cross attention for image-text matching. In Proceedings of the European conference on computer vision (ECCV), 201--216

    2018

  15. [23]

    Li, J.; Galley, M.; Brockett, C.; Gao, J.; and Dolan, B. 2015. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055

    2015 arXiv

  16. [24]

    Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, 19730--19742. PMLR

    2023

  17. [25]

    Li, J.; Li, D.; Xiong, C.; and Hoi, S. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, 12888--12900. PMLR

    2022

  18. [26]

    Li, J.; Selvaraju, R.; Gotmare, A.; Joty, S.; Xiong, C.; and Hoi, S. C. H. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34: 9694--9705

    2021

  19. [27]

    H.; Yatskar, M.; Yin, D.; Hsieh, C.-J.; and Chang, K.-W

    Li, L. H.; Yatskar, M.; Yin, D.; Hsieh, C.-J.; and Chang, K.-W. 2019. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557

    2019 arXiv

  20. [28]

    Lin, Z.; Chen, X.; Pathak, D.; Zhang, P.; and Ramanan, D. 2024. Revisiting the Role of Language Priors in Vision-Language Models. In Forty-first International Conference on Machine Learning

    2024

  21. [29]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2024. Visual instruction tuning. Advances in neural information processing systems, 36

    2024

  22. [30]

    Lu, J.; Batra, D.; Parikh, D.; and Lee, S. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32

    2019

  23. [31]

    Lu, J.; Clark, C.; Zellers, R.; Mottaghi, R.; and Kembhavi, A. 2022. Unified-io: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations

    2022

  24. [32]

    Lu, J.; Goswami, V.; Rohrbach, M.; Parikh, D.; and Lee, S. 2020. 12-in-1: Multi-Task Vision and Language Representation Learning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2020

  25. [33]

    Nikolaus, M.; Salin, E.; Ayache, S.; Fourtassi, A.; and Favre, B. 2022. Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing

    2022

  26. [34]

    Niu, Y.; Tang, K.; Zhang, H.; Lu, Z.; Hua, X.; and rong Wen, J. 2020. Counterfactual VQA: A Cause-Effect Look at Language Bias. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 12695--12705

    2020

  27. [35]

    Oord, A. v. d.; Li, Y.; and Vinyals, O. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748

    2018 arXiv

  28. [36]

    Parcalabescu, L.; Cafagna, M.; Muradjan, L.; Frank, A.; Calixto, I.; and Gatt, A. 2022. VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisti...

    2022

  29. [37]

    Parcalabescu, L.; Gatt, A.; Frank, A.; and Calixto, I. 2021. Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting tasks. IWCS 2021, 32

    2021

  30. [38]

    E.; Darrell, T.; Rohrbach, A.; and Rohrbach, M

    Petryk, S.; Whitehead, S.; Gonzalez, J. E.; Darrell, T.; Rohrbach, A.; and Rohrbach, M. 2024. Simple token-level confidence improves caption correctness. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 5742--5752

    2024

  31. [39]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning

    2021

  32. [40]

    Ramakrishnan, S.; Agrawal, A.; and Lee, S. 2018. Overcoming language priors in visual question answering with adversarial regularization. Advances in Neural Information Processing Systems, 31

    2018

  33. [41]

    Ross, C.; Katz, B.; and Barbu, A. 2021. Measuring Social Biases in Grounded Vision and Language Embeddings. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 998--1008

    2021

  34. [42]

    Salin, E.; Farah, B.; Ayache, S.; and Favre, B. 2022. Are vision-language transformers learning multimodal representations? a probing perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 11248--11257

    2022

  35. [43]

    Shekhar, R.; Pezzelle, S.; Klimovich, Y.; Herbelot, A.; Nabi, M.; Sangineto, E.; and Bernardi, R. 2017. FOIL it! Find One mismatch between Image and Language caption. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap...

    2017

  36. [44]

    Si, Q.; Liu, Y.; Meng, F.; Lin, Z.; Fu, P.; Cao, Y.; Wang, W.; and Zhou, J. 2022. Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning. In Findings of the Association for Computational Linguistics: EMNLP 2022, 6650--6662

    2022

  37. [45]

    Srinivasan, T.; and Bisk, Y. 2021. Worst of both worlds: Biases compound in pre-trained vision-and-language models. arXiv preprint arXiv:2104.08666

    2021 arXiv

  38. [46]

    Tan, H.; and Bansal, M. 2019. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490

    2019 arXiv

  39. [47]

    Thrush, T.; Jiang, R.; Bartolo, M.; Singh, A.; Williams, A.; Kiela, D.; and Ross, C. 2022. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5238--5248

    2022

  40. [48]

    Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi \`e re, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  41. [49]

    Tschannen, M.; Kumar, M.; Steiner, A.; Zhai, X.; Houlsby, N.; and Beyer, L. 2024. Image captioners are scalable vision learners too. Advances in Neural Information Processing Systems, 36

    2024

  42. [50]

    Van der Poel, L.; Cotterell, R.; and Meister, C. 2022. Mutual information alleviates hallucinations in abstractive summarization. arXiv preprint arXiv:2210.13210

    2022 arXiv

  43. [51]

    Wang, J.; Liu, Y.; and Wang, X. 2021. Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Search. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 1995--2008

    2021

  44. [52]

    Wang, J.; Yang, Z.; Hu, X.; Li, L.; Lin, K.; Gan, Z.; Liu, Z.; Liu, C.; and Wang, L. 2022 a . Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100

    2022 arXiv

  45. [53]

    Wang, P.; Yang, A.; Men, R.; Lin, J.; Bai, S.; Li, Z.; Ma, J.; Zhou, C.; Zhou, J.; and Yang, H. 2022 b . Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning, 23318--23340. PMLR

    2022

  46. [54]

    Yamada, Y.; Tang, Y.; and Yildirim, I. 2022. When are lemons purple? the concept association bias of clip. arXiv preprint arXiv:2212.12043

    2022 arXiv

  47. [55]

    Young, P.; Lai, A.; Hodosh, M.; and Hockenmaier, J. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2: 67--78

    2014

  48. [56]

    S.; Lu, X.; Zellers, R.; Ammanabrolu, P.; Le Bras, R.; Kim, G.; et al

    Yu, Y.; Chung, J.; Yun, H.; Hessel, J.; Park, J. S.; Lu, X.; Zellers, R.; Ammanabrolu, P.; Le Bras, R.; Kim, G.; et al. 2023. Fusing pre-trained language models with multimodal prompts through reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision...

    2023

  49. [57]

    Yuksekgonul, M.; Bianchi, F.; Kalluri, P.; Jurafsky, D.; and Zou, J. 2022. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations

    2022

  50. [58]

    Zhang, P.; Goyal, Y.; Summers-Stay, D.; Batra, D.; and Parikh, D. 2016. Yin and yang: Balancing and answering binary visual questions. In Proceedings of the IEEE conference on computer vision and pattern recognition, 5014--5022

    2016

  51. [59]

    Zhang, P.; Li, X.; Hu, X.; Yang, J.; Zhang, L.; Wang, L.; Choi, Y.; and Gao, J. 2021. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 5579--5588

    2021

  52. [60]

    Zhao, D.; Wang, A.; and Russakovsky, O. 2021. Understanding and evaluating racial biases in image captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 14830--14840

    2021

  53. [61]

    Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

    2023 arXiv

  54. [62]

    , " * write output.state after.block = add.period write

    ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...

  55. [63]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.