{"id":"c00cf942-7b5d-474f-ad72-1b3ec178cc1c","arxiv_id":"2501.11469","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MASS re-scores image-text pairs with pointwise mutual information, estimated by comparing caption likelihood on the real image versus a black image, and reduces language bias on color, counting, gender, and compositionality benchmarks.","lead":"This paper proposes MASS, a training-free score that re-weights image-text matches by subtracting a text-only 'language bias' estimate from a captioning model's likelihood. It matters because it offers a cheap way to make off-the-shelf image-text models less biased on color, number, and gender while keeping retrieval accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The PMI interpretation of MASS is unsupported: the Monte Carlo derivation (Eqs. 7-8) is mathematically invalid, and the black-image surrogate in Eq. 9 is never validated against the text-only marginal it is claimed to approximate.","rationale":"Eq. 9 is the method, so the validity of c_empty as a stand-in for the text-only marginal determines whether MASS is PMI or an arbitrary rescoring. The paper's own derivation of the marginal is incorrect (Eqs. 7-8), so there is no theoretical support; the experimental sections validate downstream task performance but never test the surrogate. This is load-bearing because the title and framing attribute the gains to removing language priors, and because the black image could introduce its own out-of-distribution artifacts. That said, the experiments show consistent gains across four benchmarks and several backbones, which is real evidence that the score has useful signal even if it is not literally PMI; this is why the concern supports a conditional acceptance with a validation requirement rather than rejection. The reader's weakest_assumption identifies the same issue, so I agree with the conditional verdict.","tokens_in":17781,"tokens_out":7416,"duration_ms":85220,"concrete_test":"Use a held-out set of Winoground and VALSE captions with OFA-large. For each token position, estimate the model's text-only marginal with the correct Monte Carlo identity log p_theta(x_t|x_<t) ≈ log((1/N) sum_i p_theta(x_t|x_<t,c_i)), where c_i are N=1000 random images from the COCO validation split (the same distribution the model was fine-tuned on). Compare these estimates token-by-token with log p_theta(x_t|x_<t,c_empty) from Eq. 9, and also recompute MASS rankings and Winoground groupscore using the MC marginal in place of c_empty. If the per-token deviations are large (mean |Δ| > 0.5 nats) or the groupscore changes materially, the black-image surrogate is not a valid estimate of the marginal and the PMI interpretation fails. As a secondary control, repeat with gray, white, and random-noise null images to check how much MASS depends on the specific choice of c_empty.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MASS removes language bias rests on treating Eq. 9 as pointwise mutual information, which requires log p_theta(x_t|x_<t,c_empty) to approximate the text-only marginal log p_theta(x_t|x_<t). This is the single load-bearing assumption, and it is not established. First, the derivation offered for the marginal is wrong: Eq. 7 writes log p_theta(x_t|x_<t) = integral_C log p_theta(x_t|x_<t,c) dc, and Eq. 8 approximates it by an average of log-conditionals. The correct marginal is log integral p_theta(x_t|x_<t,c) p(c) dc; averaging log-likelihoods estimates E_c[log p_theta(x|c)], not log E_c[p_theta(x|c)], and the p(c) weighting is missing. Thus the paper provides no valid Monte Carlo baseline against which c_empty could be checked. Second, a fully black image is likely out-of-distribution for OFA, which was trained on natural images; the conditional distribution under c_empty may reflect artifacts of an atypical input rather than the model's language prior. If c_empty is a poor surrogate, MASS is just token likelihood shifted by per-token, per-prefix offsets log p_theta(x_t|x_<t,c_empty), and the improvements on color, counting, gender, and Winoground could come from an arbitrary reweighting rather than from suppression of language priors. The empirical results are consistent and suggestive, but they do not test the mechanism, and no comparison to Lin et al. (2024), who use null images for the same purpose, is provided.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MASS, an inference-time score for image-text matching that subtracts a text-only marginal term from the token-level conditional likelihood of an autoregressive vision-language model, aiming to reduce language bias. The score is defined as the average per-token log-ratio of the image-conditioned likelihood to the likelihood under a black 'null image' surrogate, which is claimed to approximate the text-only marginal. Experiments cover color bias, counting, gender bias in retrieval, and the compositionality benchmarks Winoground and SVO-Probes, using OFA, BLIP-2, and LLaVA backbones. The central empirical claim is that MASS improves over raw token likelihood and CLIP-style baselines on these tasks without additional training.","tokens_in":18107,"tokens_out":3584,"duration_ms":43986,"significance":"If the empirical claim holds, the paper would offer a simple, training-free, hyperparameter-free recipe for reducing language bias in image-text matching, with potential practical value in retrieval and fairness-oriented applications. The paper is commendable for evaluating across multiple backbones, reporting a fine-grained Winoground category breakdown, and including qualitative examples. However, the theoretical framing of MASS as pointwise mutual information rests on an invalid Monte Carlo identity and an unvalidated null-image surrogate, and the closest prior method using null images and PMI is acknowledged but not compared experimentally. These issues leave the mechanism of the reported improvements unclear, even though the empirical results are suggestive and mostly consistent with the paper's claims. The manuscript is worth further consideration after the load-bearing validation and comparison gaps are addressed.","major_comments":[{"comment":"The displayed Monte Carlo identity is mathematically incorrect. The marginal log-likelihood is log pθ(xt|x<t) = log ∫_C pθ(xt|x<t,c) p(c) dc, not ∫_C log pθ(xt|x<t,c) dc. Averaging log-conditionals as in Eq. (8) estimates E_c[log pθ(xt|x<t,c)], which by Jensen's inequality is not equal to log E_c[pθ(xt|x<t,c)], and the prior p(c) over images is missing. Since this derivation is presented as the motivation for the black-image surrogate in Eq. (9), the paper should either correct it or explicitly state that Eq. (9) is a heuristic and not a valid Monte Carlo estimate of the marginal.","section":"§3.2, Eqs. (7)-(8)"},{"comment":"The central approximation that the black-filled image c∅ satisfies log pθ(xt|x<t,c∅) ≈ log pθ(xt|x<t) is asserted without evidence. This is load-bearing: if c∅ is out-of-distribution for a model trained on natural images, MASS is not pointwise mutual information but token likelihood shifted by per-token, per-prefix offsets log pθ(xt|x<t,c∅), and the debiasing effect has no theoretical grounding. The authors should validate the approximation empirically, for example by comparing pθ(·|x<t,c∅) against a Monte Carlo estimate using random natural images (with the corrected formula), and by testing sensitivity to the choice of null input (black, gray, noise, or blurred images).","section":"§3.2, Eq. (9)"},{"comment":"Lin et al. (2024), which the paper explicitly acknowledges as sharing the null-image and PMI methodology, is never compared experimentally. Since the claim that MASS outperforms strong baselines is central to the paper, a direct comparison on at least Winoground and SVO-Probes (or a clearly stated reason why a direct comparison is infeasible) is needed. Without it, the incremental contribution over the closest prior method is not established.","section":"§6, Related Work"},{"comment":"No uncertainty quantification is reported for any of the main results. Several improvements are small in absolute terms, for example the OFA-large balanced counting gain of +1.5 in Table 1, the BLIP-2 Winoground group-score gain of +2.5 in Table 3, and the decreases in some BLIP-2/LLaVA rows. Bootstrapped confidence intervals or paired significance tests over the benchmark samples are necessary to assess whether these differences are reliable, even though the scoring functions themselves are deterministic.","section":"Tables 1-4"}],"minor_comments":[{"comment":"The summation index is written as t<l, which would exclude the final token; the sums should run over t=1,...,l. Please correct the notation.","section":"§2.2, Eq. (3) and §3.1, Eq. (6)"},{"comment":"The UNITER baseline appears in Table 5 but is not cited or introduced in the main text; please add the appropriate reference and clarify where it appears in the experiments.","section":"Appendix D, Table 5"},{"comment":"The phrase 'at an expanse of high degradation' should be 'at the expense of high degradation'; the same typo occurs in the following paragraph.","section":"§4.3"},{"comment":"There are formatting issues in the author list ('Y oungjae Yu') and in the VAL SE dataset name ('V ALSE'); please clean these up.","section":"Author list and references"},{"comment":"The phrase 'we discovered that using image input as a null image c∅ ... is a good alternative' is a strong claim; please replace it with a verifiable statement and provide the validation requested in the major comments.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's overlap with Lin et al. (2024) is acknowledged in the related-work section, but the absence of any experimental comparison may raise novelty concerns during review. I would ask the editor to require the authors to add such a comparison or to provide a clear rationale for its absence. Otherwise, I see no evidence of questionable research practices; the main issues are mathematical rigor and validation of the proposed surrogate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: the headline result holds up. MASS is a training-free, inference-time score that subtracts a text-only surrogate from the image-conditioned token likelihood, and on the evidence here it consistently reduces language bias. Winoground groupscore on OFA-large goes from 15.8 to 20.3, and it beats raw token likelihood on the color, counting, gender, and SVO-Probes benchmarks. The appendix breakdown shows the Winoground gain is concentrated in the No-Tag compositional subset, which is exactly the right place.\n\nWhat is actually new: the null-image PMI trick is not new. The authors openly say Lin et al. 2024 did the same thing independently. So the contribution is the bias-aware evaluation: the color/counting/gender benchmark suite and the recall-bias Pareto analysis. Those are legitimate and useful. The paper is also honest about its limitations, and it ships enough implementation detail to reproduce the experiments.\n\nThe soft spots are real but not fatal. The Monte Carlo derivation in Eqs. 7-8 is mathematically wrong: you cannot approximate the log of an integral by an integral of logs. That means the stated justification for the black-image surrogate collapses. And the surrogate itself is never validated against any estimate of the true text-only marginal. A black image is likely out-of-distribution for OFA, so Eq. 9 might be nothing more than token likelihood shifted by per-token offsets. The debiasing wins are consistent across backbones, so empirically the method does something useful, but the mechanism is not established. Also, there is no comparison to Lin et al. 2024, who use the same method; and there are no error bars. One small thing: on some backbones (BLIP-2, OFA-tiny) MASS lowers Textscore on Winoground even though groupscore improves.\n\nI would send this to a serious referee. The core empirical claim deserves checking, and the flaws are fixable: correct the derivation or reframe it as intuition, validate the null image against a Monte Carlo estimate on a subset, add the Lin et al. comparison, and report variance. If those land, this becomes a genuinely useful reference for training-free debiasing.\n\nFor your purposes: worth reading, worth citing if your work touches multimodal fairness or inference-time debiasing, but don't take the PMI interpretation at face value until the surrogate is validated.","headline":"MASS debiasing works empirically and is honestly framed, but the derivation of the text-only marginal is wrong and the null-image surrogate is unvalidated.","tokens_in":18656,"tokens_out":3507,"would_cite":true,"duration_ms":39217,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Subtracting a black-image baseline from token likelihoods cancels language bias in image-text matching.","keywords":["language bias","image-text matching","pointwise mutual information","vision-language models","inference-time debiasing","Winoground","compositionality","multimodal association score"],"falsifier":"Compute MASS on a set of 100 captions where language prior conflicts with visual evidence, and compare its ranking with a Monte Carlo estimate of pointwise mutual information using many natural images as the marginal; if the black-image score ranks true captions below rivals while the Monte Carlo version ranks them above, the null-image approximation is the failure point.","tokens_in":17543,"feed_emoji":"🖼️","tokens_out":8539,"duration_ms":77344,"temperature":0.7,"pith_summary":"Pretrained vision-language models that score captions by token likelihood have a known flaw: they lean on language priors—'a tomato is red'—and sometimes ignore the image. This paper introduces Multimodal ASsociation Score (MASS), an inference-time re-scoring that subtracts the text-only marginal likelihood, approximated by the model's output for a black image, from the image-conditioned token likelihood. MASS is an estimate of pointwise mutual information between the image and each text token, and the paper shows it reduces color, counting, and gender bias while preserving, even improving, compositionality on Winoground and SVO-Probes. No additional training is required, and MASS applies to any autoregressive vision-language model.","feed_headline":"Black-image trick cancels language bias in image-text scores","feed_subtitle":"One black-image forward pass estimates and removes the text prior; no retraining needed.","key_machinery":"MASS (Multimodal ASsociation Score) is the central mechanism, defined by Eq. 9 as the per-token log-ratio $\\log \\frac{p_{\\bar\\theta}(x_t | x_{<t}, c)}{p_{\\bar\\theta}(x_t | x_{<t}, c_\\emptyset)}$, with $c_\\emptyset$ a black-filled image. The black-image baseline is the load-bearing device: it gives a one-pass estimate of the text-only marginal, letting the score subtract the language prior without Monte Carlo sampling across many images. Averaging these per-token log-ratios over the sequence produces a debiased similarity score that can be plugged into any off-the-shelf autoregressive vision-language model.","core_discovery":"On the paper's own terms, the log-likelihood a captioning model assigns to a caption decomposes into a linguistic component (the text prior) and an association component that measures genuine image-text correspondence. MASS separates the two by computing, per token, the log ratio of the image-conditioned token probability to the token probability given a black image, which the paper argues approximates the text-only marginal $p_{\\bar\\theta}(x_t | x_{<t})$. Averaged over the caption, this ratio is an estimate of pointwise mutual information; using it as the similarity metric instead of raw token likelihood removes the language prior. The paper shows that MASS outperforms token likelihood and CLIP-based scores on color debiasing, adversarial counting, and gender-balanced retrieval, and that on Winoground and SVO-Probes it improves groupscore and image-score, indicating that the debiasing does not come at the cost of linguistic compositionality.","pith_inferences":["We would test the null-image shortcut directly: on a set of captions with a strong language prior (e.g., gray tomato), compare MASS ranking against a Monte Carlo marginal estimate averaged over many natural images; if the rankings diverge on high-bias items, the black-image approximation, not the PMI idea, is the fragile component.","The likelihood-ratio recipe may transfer to other tasks that suffer from language priors, such as hallucination detection in image captioning or debiasing text-to-image generation scores.","Because the method needs no retraining, it can serve as a cheap audit tool: applying MASS to a new VLM reveals where its token likelihoods are dominated by textual stereotypes rather than visual evidence."],"forward_implications":["MASS turns any autoregressive vision-language model into a debiased image-text similarity scorer with only two forward passes per sample, no training, and no hyperparameters.","On adversarial counting foils (true captions with large numbers versus foils with small numbers), MASS raises OFA-large's pairwise ranking accuracy from 66.8 to 76.7, matching the best multitask baseline.","On Winoground, MASS lifts OFA-large's groupscore from 15.8 (token likelihood) to 20.3, with the gain concentrated on the No-Tag subset that isolates compositionality.","In gender-balanced retrieval on MS-COCO, MASS improves both text-to-image and image-to-text recall over token likelihood while keeping bias scores low, unlike debiasing baselines that trade recall for fairness.","The improvement transfers across backbone families: MASS raises groupscore for OFA, BLIP-2, and LLaVA, suggesting the language-prior subtraction generalizes beyond a single architecture."],"supporting_citations":[{"why":"Supplies the token-level likelihood (TL) similarity function that MASS builds on and compares against.","marker":"Petryk et al. 2024"},{"why":"Winoground benchmark that provides the main compositionality evaluation for MASS.","marker":"Thrush et al. 2022"},{"why":"SVO-Probes benchmark used to test subject, verb, and object understanding.","marker":"Hendricks and Nematzadeh 2021"},{"why":"VALSE counting benchmark with balanced, small, and adversarial foil sets.","marker":"Parcalabescu et al. 2022"},{"why":"Natural Colors Dataset used for the color debiasing experiment.","marker":"Anwar et al. 2020"},{"why":"OFA backbone model, the main vision-language model MASS is applied to.","marker":"Wang et al. 2022b"},{"why":"Provides the gender-bias evaluation setup and neutralized captions for retrieval experiments.","marker":"Wang, Liu, and Wang 2021"},{"why":"CLIP baseline representing sequence-level contrastive similarity functions.","marker":"Radford et al. 2021"}],"fun_headline_variants":["MASS strips language priors from image-text matching","Black-image prior subtraction rebalances image-text scores","One black image removes text bias from matching scores","MASS: test-time fix for language-biased image-text models","Image-text matching debiased by black-image reference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a black image makes the model produce the same token probabilities as it would with no visual information at all, so the subtraction removes the language prior rather than adding an arbitrary per-token constant.","fun_headline_variants_meta":{"raw":{"variants":["MASS strips language priors from image-text matching","Black-image prior subtraction rebalances image-text scores","One black image removes text bias from matching scores","MASS: test-time fix for language-biased image-text models","Image-text matching debiased by black-image reference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1258,"prompt_tokens":837,"completion_tokens":421,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":344}},"tokens_in":453,"tokens_out":421,"duration_ms":4522,"temperature":1.0,"reasoning_tokens":344,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:13:42.444397+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute MASS on a set of 100 captions where language prior conflicts with visual evidence, and compare its ranking with a Monte Carlo estimate of pointwise mutual information using many natural images as the marginal; if the black-image score ranks true captions below rivals while the Monte Carlo version ranks them above, the null-image approximation is the failure point.","supporting_citations":[{"cited_title":"A.; and Nematzadeh, A","cited_arxiv_id":null,"evidence_quote":"SVO-Probes benchmark used to test subject, verb, and object understanding."},{"cited_title":"E.; Darrell, T.; Rohrbach, A.; and Rohrbach, M","cited_arxiv_id":null,"evidence_quote":"Supplies the token-level likelihood (TL) similarity function that MASS builds on and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Winoground benchmark that provides the main compositionality evaluation for MASS."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VALSE counting benchmark with balanced, small, and adversarial foil sets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the gender-bias evaluation setup and neutralized captions for retrieval experiments."}],"review_version":1}