{"id":"a73a172d-910d-48e5-83be-c3d8af2362cb","arxiv_id":"2608.11024","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A null-image diagnostic shows VLM attribute hallucinations track the visual logit margin, not the language prior, and a routed Calib/Abstain/Adapt framework reduces them.","lead":"This paper tests why vision-language models sometimes describe an object's attributes incorrectly, like calling denim \"leather.\" It finds the model's visual signal, not its language bias, predicts these errors, and builds a diagnostic system that routes each error to the right fix.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline correlation is definitional: hallucination is defined as δvis ≥ 0, so δvis must predict it; the causal 'visual signal drives hallucination' claim rests on an unvalidated null-image decomposition and word-level statistics that are partly mechanical.","rationale":"The reader's weakest assumption (null image and Eq. 4) is correct as far as it goes, but the more load-bearing problem is upstream: δvis is the decision variable itself, so its correlation with hallucination is definitional. This makes Table II incapable of supporting the 'visual signal drives hallucination' claim, regardless of whether the null image is a clean prior probe. The reader did flag the sign-matching/Spearman inconsistency, which is the same issue viewed from the other side, so I mark partial agreement. The paper's descriptive contributions—color/state vs material failure modes, layer-wise SNR localization, and the routing framework—are real and can be salvaged by reframing the central claim and adding the missing non-circular analyses (text-only prior for all models, δvis-only accuracy, word-level residual regressions, and held-out word evaluation for Calib/Adapt). Under those conditions the conditional verdict stands; the concern does not by itself warrant rejection because the taxonomy and the intervention comparisons are separable from the causal slogan.","tokens_in":12463,"tokens_out":11366,"duration_ms":105844,"concrete_test":"Report the classification accuracy of sign(δvis) on the 10,791 negative samples and the accuracy of sign(δvis − δprior). If the former is 100%, Table II is definitional; then regress word-level FPR on mean δprior and mean residual (δvis − δprior) separately for each attribute type, using held-out words if any calibration parameter γ is tuned. If the residual's word-level coefficient is not robustly positive or the text-only δprior (no image tokens) differs materially from gray-null δprior for any model, the abstract's causal framing should be restated as a descriptive decision-rule decomposition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III defines hallucination for a negative sample as δvis ≥ 0 (Eq. 1). With greedy decoding, 'yes' is selected exactly when δvis > 0, so the binary hallucination label is the sign of δvis. Table II's ρ(δvis, halluc) = 0.755 is therefore a rank-biserial correlation between a variable and its own thresholded sign, not evidence about what caused the sign. The text even says sign(δvis) matches the label on all 10,791 samples, implying δvis-only accuracy is 100%, yet only prior-only accuracy (58.5%) is reported. The real test is the residual δvis − δprior, but Eq. 4 is a formal Bayes identity, not a validated measurement; δprior is the language prior only if a gray 128×128 image carries no visual prior, and text-only agreement is reported for Qwen and InternVL, not LLaVA. Moreover, δprior is constant per word, so the sample-level Spearman is misspecified for prior-dominance: a word constant cannot explain within-word image variation, and the paper's own word-level results are mixed (color ρ = +0.420, material ρ = +0.034). The SNR–FPR correlation (ρ = −0.916) is also approximately mechanical under normality since FPR ≈ Φ(signed SNR). Thus the causal claim currently has no non-tautological support.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VISOR, a framework that diagnoses attribute hallucination in vision-language models by comparing real-image and blank-image forward passes. It defines a visual logit difference δvis, a null-image prior logit difference δprior, and a residual interpreted as the visual likelihood contribution. Across 10,791 negative-ground-truth attribute probes on the VAW benchmark with Qwen2.5-VL-3B, InternVL3.5-4B, and LLaVA-1.5-7B, the paper reports that δvis strongly predicts false positives while δprior is near chance. It then proposes a taxonomy of two failure modes, low-margin color/state errors and low-SNR material errors, and evaluates three routed interventions: calibration, abstention, and per-word visual LoRA adaptation.","tokens_in":12779,"tokens_out":10280,"duration_ms":96053,"significance":"If the mechanistic claim were established, the paper would challenge the prevailing language-prior-dominance account of attribute hallucination and reframe mitigation around visual-signal quality. The paper deserves credit for a clean controlled probing setup, layer-wise SNR tracing, and a targeted per-word LoRA adaptation scheme with MME stability checks; the late-layer SNR improvement after adaptation is a concrete, falsifiable prediction. However, the headline correlation is partially definitional, and the null-image decomposition that carries the causal claim is not independently validated. The significance is therefore conditional on substantial revision of the central evidence.","major_comments":[{"comment":"The manuscript defines hallucination for a negative sample as δvis ≥ 0 and then reports ρ(δvis, halluc.) in Table II. Under greedy decoding, the hallucination label is exactly sign(δvis), so the Spearman correlation is a rank-biserial correlation between a variable and its own thresholded sign; the text's statement that sign(δvis) matches the label on all 10,791 samples makes this explicit. This correlation provides no causal evidence that the visual signal 'strongly predicts false positives' and is at most a self-consistency check of the decoding rule. Please report a non-tautological predictive analysis, such as the sign of δvis − δprior against the dataset label, and also report the δvis-only accuracy, which the current text implies is 100%.","section":"Section III, Eq. (1); Table II"},{"comment":"Equation (4) is a formal Bayes identity, not a measurement of a visual likelihood. The identification of δprior with the language prior requires that the blank image I0 has zero visual likelihood ratio, and the identification of δvis − δprior with the visual contribution requires that the residual is not contaminated by image-independent biases. The stability checks across gray, black, white, and random-noise images do not validate either identification, and the text reports text-only agreement only for Qwen and InternVL, not LLaVA. Without such validation, the conclusion that the language prior does not drive attribute hallucination does not follow; the authors should compare null-image priors with true text-only priors on all three models and demonstrate that δvis − δprior tracks a known visual signal in a controlled setting.","section":"Section III, Eq. (4)"},{"comment":"The prior-positive conflict analysis is a restatement of the decision rule. The 278 cases with δprior > 0 and δvis < 0 are, by construction, cases where the model says 'no', so the 0.0% hallucination rate in Table III is entailed by the definition of hallucination rather than evidence that the image overrides the prior. The argument would need to show that δprior, measured independently of the model's final logits, predicts behavior differently; as it stands, Table III cannot support the claim that real image content 'actively modifies model behavior' in a causal sense.","section":"Section IV, Table III"},{"comment":"The sample-level Spearman correlation for δprior is not a well-specified test of language-prior dominance because the null-image prior contains no image-specific variation and is defined from the same forward pass as δvis; the word-level results in Section IV are more appropriate but mixed (color ρ = +0.420, material ρ = +0.034). In addition, the SNR–FPR correlation of −0.916 in Table V is approximately mechanical under a Gaussian model: if FPR is the mass of the δvis distribution above zero, it is essentially a function of SNR, so this correlation does not by itself establish SNR as a causal driver of hallucination severity.","section":"Section IV, Correlational analysis; Table V"}],"minor_comments":[{"comment":"The name 'LLaV A' appears repeatedly in the abstract and in Table II; it should read 'LLaVA' consistently.","section":"Throughout"},{"comment":"The main text announces the correlational analysis without citing Table II at that point; add a citation and consider presenting the image-sensitivity analysis table in the main text rather than referring to it implicitly.","section":"Section IV, Correlational analysis"},{"comment":"Equation (2) writes δprior(M, w), but the query prompt contains an object token; clarify whether the prior is defined for the object–attribute pair and update the notation accordingly.","section":"Section III, Eq. (2)"},{"comment":"The phrase 'carries no semantic content' should be replaced by an explicit mathematical assumption about the visual likelihood ratio of the null image, since 'semantic content' is not a formal condition.","section":"Section III, Eq. (4)"},{"comment":"The per-word Spearman correlations in Section IV are reported with p-values but without effect sizes or confidence intervals; include these for the main comparisons.","section":"Section IV, Word-level correlations"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper's core mechanistic claim currently rests on a definitional correlation and an unvalidated null-image decomposition. If the authors cannot supply an independent validation of the decomposition and a non-tautological predictive test, the paper should be rejected; the targeted LoRA adaptation results alone would not support the title and framing. The supplementary material referenced for the proof and validation was not available in the arXiv posting and should be checked carefully in any revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Candid take: the null-image probe is a genuinely useful diagnostic, and the material/color asymmetry looks real. But the paper's headline — visual signal drives attribute hallucination, language prior near chance — is not actually supported by the reported numbers, because the hallucination label is defined as the sign of δvis. That makes the correlation with δvis something close to a self-correlation, not evidence about causation.\n\nWhat's new: the first attribute-level null-image decomposition, the B1/B2 annotation-noise versus representation-failure split, layer-wise SNR collapse in the late decoder layers, and the routed Calib/Abstain/Adapt framework. The negative controls in Table IX are the best part of the paper: language-layer DPO does nothing, shared LoRA degenerates to always-No, and the L28 probe recovers signal before the final projection. The MME sanity check and the honest limitations paragraph also speak well for the authors.\n\nSoft spots: (1) Definitional circularity. Hallucination is δvis ≥ 0, so δvis must predict it. The paper even says sign(δvis) matches the label on all 10,791 samples — that implies δvis-only accuracy is 100%, yet Table II reports Spearman rho between 0.755 and 0.835. Those statements are inconsistent if rho is a rank-biserial; if it is something else, the metric needs to be stated. (2) The causal decomposition rests on Eq. 4, a formal Bayes identity, not a measurement. The residual δvis − δprior is only the visual likelihood contribution if the gray null image carries no visual prior. The paper reports text-only agreement for Qwen and InternVL, not LLaVA. (3) δprior is constant per word, so a sample-level Spearman for prior-dominance is misspecified; the word-level results are mixed (color +0.420, material +0.034). (4) The SNR–FPR correlation (−0.916) is approximately mechanical under normality, since FPR ≈ Φ(signed SNR). (5) Several intervention gains use fitted gamma, tau, and per-word LoRA trained on the evaluation words; held-out generalization is not established.\n\nWhat holds up: the scoped claim that the null-image prior is a weak sample-level predictor, and that color/state differs from material, is defensible. That is worth having.\n\nRecommendation: send to peer review with a request for major revision. The authors need to redo the analysis on the non-definitional residual, report per-word statistics properly, validate the null-image assumption, and release code/data. A serious referee can extract real value from this; it should not be desk-rejected, but it should not be accepted as-is either.","headline":"Useful diagnostic, but the headline causal claim is partly a tautology; the scoped material/color asymmetry is the real contribution.","tokens_in":701,"tokens_out":766,"would_cite":false,"duration_ms":23935,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Attribute hallucination in vision-language models is driven by weak or misaligned visual signals, not by a dominant language prior.","keywords":["attribute hallucination","vision-language models","visual signal quality","language prior","null-image diagnosis","mechanistic analysis","calibration","visual adaptation"],"falsifier":"Construct an attribute word whose blank-image prior is strongly positive yet whose real-image visual margin is negative, and show that false positives occur at a high rate despite the negative visual margin; alternatively, show that varying the gray level of the null image changes $\\delta_{\\mathrm{prior}}$ in a way that is correlated with hallucination, indicating the null image itself carries a visual prior.","tokens_in":35,"feed_emoji":"👁️","tokens_out":7576,"duration_ms":119563,"temperature":0.7,"pith_summary":"This paper claims that attribute hallucination in vision-language models—saying a visibly present object has the wrong color, material, or state—is not primarily caused by a dominant language prior, the model's tendency to lean on word co-occurrence statistics. Instead, the raw visual signal itself predicts false \"yes\" answers, either because it is too weak to cross the decision threshold or because it is misaligned in the final output layer. The authors measure, for each attribute query, the logit difference between \"yes\" and \"no\" on the real image and on a blank gray image that should carry no visual content. Across 10,791 negative examples from three model families, the visual logit difference correlates with hallucination while the language-prior difference is near chance. If the finding holds, hallucination mitigation should start by diagnosing the visual signal rather than uniformly suppressing language priors.","feed_headline":"Visual signals, not language priors, drive VLM attribute errors","feed_subtitle":"A blank-image probe shows false \"yes\" answers track weak visual signals; fixes are then routed to the right failure mode.","key_machinery":"The load-bearing object is the VSNR diagnostic, which decomposes each yes/no attribute prediction into a visual logit difference, $\\delta_{\\mathrm{vis}} = \\mathrm{logit}(\\mathrm{yes}|I,q_w) - \\mathrm{logit}(\\mathrm{no}|I,q_w)$, and a prior logit difference, $\\delta_{\\mathrm{prior}} = \\mathrm{logit}(\\mathrm{yes}|I_\\emptyset,q_w) - \\mathrm{logit}(\\mathrm{no}|I_\\emptyset,q_w)$, computed with a uniform gray null image. The decision rule $\\mathrm{sign}(\\delta_{\\mathrm{vis}})$ matches the hallucination label on all 10,791 samples, and a Bayesian decomposition writes the model's log-odds as a visual likelihood ratio plus a language prior, so $\\delta_{\\mathrm{vis}} - \\delta_{\\mathrm{prior}}$ approximates the visual contribution. The same coordinate yields a per-layer signal-to-noise ratio that localizes material failures to the final decoder layers and routes each query to calibration, abstention, or per-word visual adaptation.","core_discovery":"The central discovery is that attribute false positives are better explained by visual-signal quality than by language-prior dominance. For negative-ground-truth queries, the Spearman rank correlation between the visual logit margin and the hallucination outcome is high (0.755–0.835 across models), while the correlation between the null-image prior logit margin and hallucination is near zero (0.066–0.177), and a prior-only classifier barely beats the majority baseline. Two failure modes emerge: color and state attributes have directionally correct but low-margin visual signals, correctable by moving the decision threshold; material attributes have low signal-to-noise ratio or wrong-direction visual signals in late decoder layers, which threshold moves cannot fix. The paper concludes that language-prior suppression is misaligned with the actual mechanism for attribute hallucination.","pith_inferences":["A testable extension is to apply the same null-image decomposition to other hallucination types—object existence, spatial relations, counting—to see whether they also follow visual-signal quality rather than language prior.","If the bottleneck is late-layer visual-to-logit projection, interventions at intermediate layers, such as early-exit or layer-wise adapters, might recover material accuracy at lower cost than per-word LoRAs.","The two-forward-pass VSNR coordinate is cheap enough to serve as a runtime abstention signal in high-stakes deployments, since it requires no training and no extra data.","The paper leaves open whether material failures stem from pretraining coverage or from projection, so a natural follow-up is to train on material-rich captions and check whether final-layer SNR rises."],"forward_implications":["Prior-suppression decoding methods cannot fix attribute false positives where the visual margin is already wrong; the paper shows VCD and ICD produce zero color-FPR improvement while VISOR-Calib cuts color FPR by 11–17 percentage points.","Material attribute hallucination requires either abstaining on low-SNR queries or adapting the visual projection per word; VISOR-Abstain lowers Qwen material FPR from 15.7% to 10.2% at a 14.2% abstention rate, and VISOR-Adapt cuts the six-word mean from 38.1% to 23.9%.","The diagnosis generalizes beyond the VAW format: the $\\delta_{\\mathrm{vis}} \\ge 0$ criterion yields zero false-positive violations on the GQA color subset across all three models.","Late-layer SNR collapse in layers 28–36 implies that material attribute errors are a projection problem, not a missing-visual-evidence problem, so targeted visual adaptation is more appropriate than output-level correction."],"supporting_citations":[{"why":"Supplies the VAW attribute-annotation dataset that yields the 10,791 negative-ground-truth yes/no probes.","marker":"(Pham et al., 2021)"},{"why":"Defines VCD, the main prior-suppression baseline whose failure mode the paper analyzes.","marker":"(Leng et al., 2024)"},{"why":"Defines ICD, the second prior-suppression baseline compared against VISOR.","marker":"(Wang et al., 2024)"},{"why":"Provides the POPE balanced yes/no probing protocol that the benchmark follows.","marker":"(Li et al., 2023b)"},{"why":"Defines Qwen2.5-VL-3B, one of the three model families used for the experiments.","marker":"(Bai et al., 2025)"},{"why":"Defines InternVL3.5-4B, the second model family used for cross-model validation.","marker":"(Chen et al., 2024)"},{"why":"Defines LLaVA-1.5-7B, the third model family used for cross-model validation.","marker":"(Liu et al., 2024b)"},{"why":"Supplies the MME hallucination subset used to check that per-word adapters do not degrade general capability.","marker":"(Fu et al., 2023)"}],"fun_headline_variants":["Weak visual signals, not language prior, cause VLM attribute errors","VLM attribute errors stem from weak visual signals, not language bias","Attribute hallucination in VLMs traced to visual signal quality, not prior dominance","Visual signal quality, not language prior, explains VLM attribute mistakes","Diagnosis: weak visual signals, not prior suppression, fix VLM attribute errors"],"cache_read_input_tokens":15360,"weakest_assumption_plain":"The conclusion depends on the blank gray image having no visual content relevant to the attribute, so the logit difference it produces truly measures the language prior, and on the residual $\\delta_{\\mathrm{vis}} - \\delta_{\\mathrm{prior}}$ being a valid measure of the visual likelihood contribution rather than just a definition of the leftover.","fun_headline_variants_meta":{"raw":{"variants":["Weak visual signals, not language prior, cause VLM attribute errors","VLM attribute errors stem from weak visual signals, not language bias","Attribute hallucination in VLMs traced to visual signal quality, not prior dominance","Visual signal quality, not language prior, explains VLM attribute mistakes","Diagnosis: weak visual signals, not prior suppression, fix VLM attribute errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000797,"raw_usage":{"total_tokens":3507,"prompt_tokens":945,"completion_tokens":2562,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":2466}},"tokens_in":561,"tokens_out":2562,"duration_ms":14566,"temperature":1.0,"reasoning_tokens":2466,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:47:52.017513+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct an attribute word whose blank-image prior is strongly positive yet whose real-image visual margin is negative, and show that false positives occur at a high rate despite the negative visual margin; alternatively, show that varying the gray level of the null image changes $\\delta_{\\mathrm{prior}}$ in a way that is correlated with hallucination, indicating the null image itself carries a visual prior.","supporting_citations":[],"review_version":1}