REVIEW 2 major objections 5 minor 21 references
Visual Credit Audit for Multimodal Spatial Reasoning
T0 review · 2 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read This paper claims that accuracy on spatial yes/no benchmarks overstates image-grounded correctness, and that a label-free decision-level audit can decompose benchmark success into correctness, additional image support, and relation-consiste
desk verdict Careful, mostly convincing label-free audit of image support in spatial yes/no benchmarks; headline C-U numbers hold under the paper's operationalization, with two fixable soft spots. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the prediction-aligned dependence event D_i = 1{r_{i,o} > max(r_{i,t}, r_{i,b})}, where r is the model's margin for its own declared answer under the original image, text-only, and blank contexts, with the event fixed before correctness labels are seen. This yields the four-cell accuracy-credit decomposition and the headline quantities D-CC and C-U. The companion relation axis uses the fixed-pixel true-false separation H_c and the interaction G_int = H_o − max(H_t, H_b), plus the Joint conjunction, to distinguish marginal image support from relation-consistent visual response. The matched same-split image permutation is the calibration mechanism that tests whether the o
What would settle it
Run the same audit with a control that preserves scene structure but removes the queried objects, or with a set of unrelated natural images matched for low-level statistics; if D-CC rises to near accuracy — that is, if the 13–26% correct-but-uncredited mass largely disappears — then the C-U estimate is an artifact of the particular text-only/blank controls rather than a stable property of the models.
Extended reading notes
Core claim
The paper's central claim is that a single label-free event — whether the original image gives the model's declared answer more support than both a text-only prompt and a fixed gray blank — separates benchmark success into four exhaustive cells, and that a substantial share of correct answers fall in the "correct but uncredited" cell. In the reported runs, this share ranges from 12.73% to 26.25%, meaning ordinary accuracy overstates support-qualified correctness by that amount. The paper further shows that this gap is not an artifact of the specific controls: replacing the benchmark image with a deterministic same-split unrelated image lowers dependence-credited correctness by 21.25 to 47.80
Load-bearing premise
The audit's load-bearing premise is that a text-only prompt and a fixed mid-gray blank image are neutral stand-ins for "no benchmark image," so that any extra support the original image provides over both is genuinely due to the image's content; if the gray blank acts as an active distractor or as a hidden "objects absent" cue, the correct-but-uncredited share is overstated.
Editorial extensions
If this is right
- Benchmark reports would need to include D-CC (dependence-credited correctness) alongside accuracy, since two models with similar accuracy can differ by more than 20 points in support-qualified success.
- Because 12.73–26.25% of correct decisions are not better supported by the image than by text-only or blank controls, accuracy is an upper bound on image-grounded correctness, not a measure of it.
- Credit is a relative-support statement, not answer necessity: a decision can be uncredited even when the no-image controls do not flip the answer, so "no image support" should not be read as "image irrelevant."
- Correct-but-uncredited decisions are often still visually responsive: under controlled relation reversal, 81.57–100.00% of such decisions move in the correct direction and 32.11% change their answer, so a single audit number cannot capture both support and relation response.
- The audit is training- and label-free and needs only three cached contexts per item, so it can be applied to any frozen model and any fixed binary answer interface without retraining.
Reading between the lines
- The same four-cell decomposition would likely transfer to other closed-form multimodal benchmarks (VQA, visual entailment, object presence), where a text-only or blank control can also be defined; the spatial setting may just be the easiest place to see it.
- Because the dependence event is label-free, it could serve as a data-filtering signal to remove benchmark items that reward answer priors, complementing label-based filtering methods.
- The deterministic same-split permutation could be replaced by a distribution over unrelated images to produce a tighter and more general null for "generic image" support; the paper tests one permutation per sample.
- The high directional response among uncredited decisions suggests that "uncredited" means redundant visual support rather than absent visual support; a useful next step would be to separate those two sub-cases inside the C-U cell.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Visual Credit Audit (VCA), a label-free, decision-level procedure for closed yes/no spatial benchmarks. For each query it caches a frozen MLLM's yes/no continuation margin under the original image, a text-only prompt, and a fixed mid-gray blank image, and defines a dependence event D = 1{r_o > max(r_t, r_b)}. After labels are applied, the paper obtains D-CC = E[κD] and C-U = E[κ(1−D)], and shows that on correct items the event reduces to a gold-aligned positive-gain statistic. Across four MLLMs and VSR/GSR-COCO, it reports 12.73–26.25% correct-but-uncredited decisions and a matched same-split image-permutation D-CC excess of 21.25–47.80 points. A relation axis (SRC, PairCredit, Joint) and a 3×3 evidence-source factorial are used to argue that marginal image-support advantage and fixed-pixel relation response are separate estimands. The paper concludes that benchmark accuracy can overstate image-grounded success.
Significance. The paper is unusually transparent and the accounting is sound: D is assigned before labels, the four-cell decomposition in Eq. (3) is complete, and the correct-item restriction in Eq. (4) exactly recovers gold-aligned gain. The paired-bootstrap intervals, deterministic same-split permutation calibration, threshold-mass analysis, multi-verbalizer checks, and direction-balanced sensitivities are all appropriate and mostly well reported. The paper ships code, deterministic data builders, cached margins, and deidentified annotations, which is a real strength. The two-estimand separation — relative image support versus relation response — is a useful corrective, and the factorial convincingly shows why null-control marginal support cannot identify relation response. If the central C-U estimate survives a neutral no-objects control, the finding that 12.73–26.25% of correct decisions lack control-relative image support is important for benchmark interpretation.
major comments (2)
- [§3.1, Eq. (1); Table 10; Tables 14 and 28] The headline C-U estimate treats the fixed mid-gray blank as a neutral 'no benchmark image' baseline. For a spatial yes/no probe, a gray canvas is a visible scene with no objects, which can directly license 'no'. The paper's strata show the signature: in Table 14, LLaVA VSR C-U is 36.94 for gold-no vs 2.10 for gold-yes; InternVL VSR is 34.78 for gold-yes vs 1.20 for gold-no. C-U is concentrated in the direction favored by no-image/blank priors. Table 28's strict T/B requirement collapses Dep (e.g., InternVL GSR 42.84→10.45), consistent with controls usually agreeing with the original answer rather than being neutral. The mean-color blank (Table 37) changes low-level color but still has no objects, so it does not remove the 'objects absent' cue. This is load-bearing for the 12.73–26.25% range. Please add a no-objects background control (or text-only-only reporting) to isolate the gray-bla
- [Abstract/Conclusion vs Table 15] The headline range is not direction-invariant. Prediction-balanced C-U is 11.73–43.36% while gold-balanced is 12.73–26.25%, and VSR orderings change. The supplement discloses this, but the main text (abstract, conclusion) states the range without the qualification. Since the C-U estimate is also sensitive to the blank-control concern in Comment 1, the main text should explicitly say that the 12.73–26.25% figure is benchmark-distribution-specific and should report the direction-balanced estimates in the main robustness section.
minor comments (5)
- [Abstract/§3.1] Throughout, 'no-image controls' is imprecise for the blank: a gray canvas is still an image. Suggest 'image-absent' or 'content-free image' to avoid implying that the blank contains no visual input.
- [Table 1] The last column header 'C-U∆ perm DCC' is hard to parse; split into 'C-U' and 'Δ perm D-CC [CI]' and explain in the caption that Δ is the paired original-minus-permutation D-CC difference.
- [§4.3/Figure 1] The 3×3 evidence-source factorial is central to the response-vs-support separation but is only described in text; a diagram of the nine cells (or at least the agreement and conflict cells) would make the design easier to verify.
- [Table 2/§4.2] PairCredit = 1{G_int > 0} can be positive even when the original true–false separation is negative if the controls are more negative. The caption's phrase 'image-specific semantic separation' should explicitly state that PairCredit does not require H_o > 0; PairCorrect plays that role.
- [Limitations] The Limitations section is honest about causal attribution but does not list the gray-blank neutrality assumption as a limitation; add it explicitly, together with the no-objects control that would test it.
Circularity Check
No significant circularity: the audit quantities are prespecified, the two definitional reductions are explicitly disclosed and non-load-bearing, and the headline empirical claims rest on independent control contrasts.
full rationale
The derivation chain is self-contained. D_i = 1{G_img_i > 0} with G_img_i = r_i,o - max(r_i,t, r_i,b) is a prespecified, label-free event; no parameter is fitted to labels, and the zero threshold is fixed before labels are consulted (Eqs. 1-2). The paper explicitly discloses that on correct items this event is algebraically identical to a gold-aligned gain: Eq. (4) states kappa_i D_i = kappa_i 1{s_i m_i,o > max(s_i m_i,t, s_i m_i,b)}, and the text says the event 'intentionally reduces exactly to a gold-aligned positive gain' but 'extends the same order event label-free to all decisions.' That is an honest characterization, not a hidden prediction. Likewise, the Joint => G_int > 0 implication is explicitly labeled 'a structural property' and 'definitional,' and the paper reports PairCredit and mean G_int as the empirical quantities outside Joint, so no derived result is being passed off as evidence. The headline C-U estimates are direct empirical frequencies from cached margins and benchmark labels, not outputs of a fitted model. The matched same-split permutation calibration changes only the image while holding the audited decision fixed; this is an independent empirical contrast, and the 21.25-47.80 point drops are not forced by the definition of D. The factorial and 108-edit natural-image checks provide external, non-self-referential validation. The main substantive threat identified by the skeptical review is whether the fixed mid-gray blank operationalizes 'no benchmark image' neutrally; but the paper states that 'dependence' means this control-relative support advantage, so any concern is about construct validity of the control, not circularity of the derivation. No load-bearing self-citations or imported uniqueness theorems appear.
Assumptions & free parameters
free parameters (2)
- Dependence threshold =
0
- Edit renderer constraints (center gap ≥0.05, new-box IoU ≤0.20, padding 4, inpaint radius 3) =
see supplement
assumptions (6)
- domain assumption Next-token continuation-loss margins on a fixed forced-choice answer interface are a valid measure of the MLLM's support for a decision.
- domain assumption Text-only (no image) and blank mid-gray image are valid no-evidence controls representing absence of benchmark-image support.
- domain assumption Sampling the same relation pair with true and false statements cancels answer-surface offsets; H_c differences are comparable across contexts.
- domain assumption Procedural rendered scenes in the 3×3 factorial are informative about behavior on natural benchmark images.
- domain assumption The deterministic same-split image permutation is unrelated enough to act as a mismatch calibration.
- domain assumption Human majority consensus on 108 edits is a reliable ground truth for relation flips and object visibility.
Cite this review
Pith. "Pith review of Visual Credit Audit for Multimodal Spatial Reasoning." pith.science (2026). https://pith.science/paper/UKQVWBP5
@misc{pith2026260727069,
author = {Pith},
title = {Pith review of: Visual Credit Audit for Multimodal Spatial Reasoning},
year = {2026},
howpublished = {\url{https://pith.science/paper/UKQVWBP5}},
note = {Machine review of arXiv:2607.27069}
}
read the original abstract
Closed yes/no spatial benchmarks can reward a correct answer even when the image adds little support beyond no-image contexts. Under a fixed forced-choice interface, Visual Credit Audit (VCA) separates two estimands: whether the benchmark image gives the model's declared decision more support than text-only and blank controls, and whether the model responds to relation-specific visual evidence. The first audit is training- and label-free and does not require an answer flip. Applying labels yields dependence-credited correctness (D-CC); on correct items, it equals same-control gold-aligned positive gain, while prediction alignment extends the audit to errors. Across four open MLLMs and two spatial benchmarks, 12.73-26.25% of decisions are correct yet uncredited. Matched same-split image permutation reduces D-CC by 21.25-47.80 points, with every paired 95% interval above zero. Fixed-pixel relation contrasts and a 3x3 evidence-source factorial show why null controls cannot identify relation response. Among controlled correct-but-uncredited agreement decisions, response to relation reversal spans 81.57-100.00%, while 32.11% pooled change answer. Independently audited outcomes on 108 geometry-compatible edits provide a bounded natural-image correspondence check. VCA thereby decomposes benchmark success into correctness, additional image support, and relation-consistent response.
Figures
Reference graph
Works this paper leans on
-
[1]
Answer the user’s question with exactly yes or no,
Continuous Raw-Gap Distributions BinaryDeprecordsthegapsign;itsmagnitudedistinguishes near-zeroboundarycasesfromstrongper-itemevidence.Ta- ble17showsbroadoverlapacrossmodelsanddatasetsrather thanadegeneratemassatzero.Inparticular,InternVL/GSR- COCO has a negative median and mean but a positive upper quartile, explaining how its Dep rate can remain 42.84%....
2026
-
[4]
VisNec: Measuring and Leveraging Visual NecessityforMultimodalInstructionTuning.arXivpreprint arXiv:2603.01195. Fu, H. Y.; Yang, C.; Chang, T.-Y.; Wang, Z.; Wu, Z.; Zhou, J.; and Fu, D
-
[6]
InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2901–2910
CLEVR: A Diag- nostic Dataset for Compositional Language and Elementary Visual Reasoning. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2901–2910. Kamath,A.;Hessel,J.;andChang,K.-W.2023. What’s“up” withVision-LanguageModels?InvestigatingTheirStruggle with Spatial Reasoning. InProceedings of the 2023 Confer- enceonEmpiric...
2023
-
[7]
From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQA.arXiv preprint arXiv:2606.30220. Lan, Z.; Sun, L.; Walter, M. R.; and Zhou, J
-
[8]
Leng, S.; Zhang, H.; Chen, G.; Li, X.; Lu, S.; Miao, C.; and Bing, L
Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?arXiv preprint arXiv:2605.22903. Leng, S.; Zhang, H.; Chen, G.; Li, X.; Lu, S.; Miao, C.; and Bing, L
-
[11]
Ma, C.; Lu, K.; Cheng, T.-Y.; Trigoni, N.; and Markham, A
Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning.arXiv preprint arXiv:2306.14565. Ma, C.; Lu, K.; Cheng, T.-Y.; Trigoni, N.; and Markham, A
-
[12]
SpatialPIN: Enhancing Spatial Reasoning Capa- bilities of Vision-Language Models through Prompting and Interacting 3D Priors.arXiv preprint arXiv:2403.13438. Mistral AI
- [13]
Show all 21 references
-
[14]
Reich,D.;Putze,F.;andSchultz,T.2023.MeasuringFaithful and Plausible Visual Grounding in VQA
GSR-BENCH: A Bench- mark for Grounded Spatial Reasoning Evaluation via Multi- modal LLMs.arXiv preprint arXiv:2406.13246. Reich,D.;Putze,F.;andSchultz,T.2023.MeasuringFaithful and Plausible Visual Grounding in VQA. InFindings of the Association for Computational Linguistics: E...
2023 arXiv
-
[16]
Sun, Z.; Shen, S.; Cao, S.; Liu, H.; Li, C.; Shen, Y.; Gan, C.; Gui, L.-Y.; Wang, Y.-X.; Yang, Y.; et al
Do Vision-Language Models See or Guess? Measuring and Re- ducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark.arXiv preprint arXiv:2606.10400. Sun, Z.; Shen, S.; Cao, S.; Liu, H.; Li, C.; Shen, Y.; Gan, C.; Gui, L.-Y.; Wang, Y.-X.; Yang, Y.; et al
-
[17]
arXiv preprint arXiv:2309.14525
Aligning LargeMultimodalModelswithFactuallyAugmentedRLHF. arXiv preprint arXiv:2309.14525. Thrush, T.; Jiang, R.; Bartolo, M.; Singh, A.; Williams, A.; Kiela, D.; and Ross, C
-
[19]
In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 10006–10030
MMoE: Enhancing Multimodal ModelswithMixturesofMultimodalInteractionExperts. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 10006–10030. Zhang, C.; Ding, W.; Liu, J.; Wu, M.; Wu, Q.; and Mooney, R
2024
-
[20]
Zhang, L.; Zhai, X.; Zhao, Z.; Zong, Y.; Wen, X.; and Zhao, B
Do Images Speak Louder than Words? Investigat- ing the Effect of Textual Misinformation in VLMs.arXiv preprint arXiv:2601.19202. Zhang, L.; Zhai, X.; Zhao, Z.; Zong, Y.; Wen, X.; and Zhao, B
-
[305]
Li, Y.; Tian, W.; Jiao, Y.; and Chen, J. 2024b. Eyes Can Deceive: Benchmarking Counterfactual Reasoning Capabil- ities of Multimodal Large Language Models.arXiv preprint arXiv:2404.12966. Liang, X.; Guo, X.; Jin, Z.; Pan, W.; Shang, P.; Cai, D.; Lin, B.;andYe,J.2025.EnhancingS...
2025 arXiv
-
[2017]
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6904–6913
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6904–6913. Guo,C.;Pleiss,G.;Sun,Y.;andWeinberger,K.Q.2017. On Calibration of Modern Neural Netw...
2017
-
[2018]
InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 4035–4045
Object Hallucination in Image Caption- ing. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 4035–4045. Singla, P.; Garg, S.; Singh, V.; and Chopra, P
2018
-
[2022]
InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5238–5248
Winoground: Probing Vision andLanguageModelsforVisio-LinguisticCompositionality. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5238–5248. Wang,W.;Gao,Z.;Gu,L.;Pu,H.;Cui,L.;Wei,X.;Liu,Z.; Jing,L.;Ye,S.;Shao,J.;etal.2025.InternVL3.5:Advanci...
2025 arXiv
-
[2023]
Bai, S.; et al
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, TextReading,andBeyond.arXivpreprintarXiv:2308.12966. Bai, S.; et al
-
[2024]
Li, B.; Zhang, Y.; Guo, D.; Zhang, R.; Li, F.; Zhang, H.; Zhang, K.; Zhang, P.; Li, Y.; Liu, Z.; and Li, C
Mitigating Object Hallucinations in Large Vision-LanguageModelsthroughVisualContrastiveDecod- ing.ProceedingsoftheIEEE/CVFConferenceonComputer Vision and Pattern Recognition, 13872–13882. Li, B.; Zhang, Y.; Guo, D.; Zhang, R.; Li, F.; Zhang, H.; Zhang, K.; Zhang, P.; Li, Y.; L...
2023 arXiv
-
[2025]
Qwen3-VL Technical Report.arXiv preprint arXiv:2511.21631. Bhat, S. D.; and Yamasaki, T
-
[2026]
arXiv preprint arXiv:2607.09068
OmniMapBench: Benchmark- ing Visual-Centric Reasoning on Diverse Map Documents. arXiv preprint arXiv:2607.09068. Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; et al. 2024b. In- ternVL: Scaling up Vision Foundation Models and Ali...
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.