Pith. sign in

REVIEW 5 major objections 5 minor 64 references

Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VaLSe traces which image regions drive each output word, then steers the model's internal state toward those regions to reduce hallucinated objects.

desk verdict Credible training-free steering with a solid CHAIR gain on LLaVA-1.5, but the interpretability maps are under-validated and the mitigation result does not actually require them to be faithful; still worth a serious referee. read the letter →

arxiv 2505.17812 v1 pith:27XOPCZW submitted 2025-05-23 cs.CV cs.AI

Boxu Chen , Ziwei Zheng , Le Yang , Zeyu Geng , Zhengyu Zhao , Chenhao Lin , Chao Shen This is my paper
classification cs.CVcs.AI
keywords objecthallucinationvision-languagemodelsinterpretabilityattentionmapslatentsteeringCHAIRbenchmarksingularvaluedecompositionvisualgrounding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VaLSe tries to show that object hallucination in vision-language models can be mitigated after the fact, without retraining, by first making the model's visual reasoning visible. For each word that depends strongly on the image, the method builds a visual contribution map: a heatmap over image regions that raised that word's probability, obtained by propagating relevance through the model's attention layers and removing known activation artifacts. The maps are then used to mask irrelevant parts of the image, the model's internal features with and without those parts are contrasted, and the dominant difference direction is added to the hidden states during generation. The paper reports that this lowers the sentence-level CHAIR hallucination rate of LLaVA-1.5 from 48.7 to 36.2 and beats all compared methods on the 64-token CHAIR setting, while general benchmarks stay roughly level. If the maps are faithful, the same procedure is simultaneously a token-level explanation tool and a training-free fix, and it provides evidence that some words flagged as hallucinated by current metrics are actually grounded in the image.

What carries the argument

The central object is the visual token contribution map: a per-token heatmap over image tokens built by propagating an identity relevance matrix through all LLM layers with gradient-weighted head averaging, $\bar{A}^l = \left(\sum_h \frac{\partial \mathcal{L}_{y_s}}{\partial A^l_h} \odot A^l_h\right)^+$, and reading off the image-token entries of the last row. Artifact activations are removed by contrasting the target token's map with that of a non-semantic special token, which suppresses the fixed spatial positions that fire regardless of input. The map does two jobs: it defines which image patches are semantically relevant (positive samples are the original image with all sub-mean-relevance patches masked), and it is the basis for the steering signal, since the per-layer difference between positive- and negative-sample MLP features is decomposed by SVD and the top right-singular vector becomes the steering direction added at inference.

What would settle it

Run the same steering pipeline with random masking at the same masking ratios on a large set of images; if random masks reduce CHAIR scores as much as VaLSe's relevance-guided masks, or if deleting the top-ranked regions of a contribution map changes the target word's probability no more than deleting random regions, then the maps are not carrying the causal signal the paper claims.

Watch

Extended reading notes

Core claim

The paper's central claim is that the gradient-weighted attention propagation of Eq. (5), after subtracting artifact activations, yields visual contribution maps that faithfully trace how each image region contributes to a given output token. Those maps identify the visual evidence behind each word, and aggregating them over the selected words defines a semantic mask: positive samples keep only the high-contribution patches, while the original image and response form the negative sample. Contrasting the model's MLP features for positive and negative samples at every layer, then taking the top right-singular vector of the difference via SVD, produces a per-layer steering direction that is added to hidden states at inference. In the paper's experiments this realigns attention toward semantically relevant content and reduces hallucinated outputs on CHAIR, AMBER, POPE, and MMHal for LLaVA-1.5 and Qwen2-VL with little or no loss on MME, GQA, and LLaVA-Bench. The visualization analysis further argues that some words CHAIR labels as hallucinated are actually correct, visually grounded descriptions, which the authors take as evidence that existing OH metrics need revision.

Load-bearing premise

The load-bearing assumption is that the contribution map for a word really marks the image regions the model used to generate that word; if the map points to the wrong regions, both the explanations and the steering built from them lose their basis.

Editorial extensions

If this is right

  • With VaLSe, LLaVA-1.5's sentence-level CHAIR hallucination rate drops from 48.7 to 36.2, and its 64-token CHAIR score is lower than all compared mitigation methods in the paper's experiments.
  • Hallucination scores improve on AMBER, POPE, and MMHal for both LLaVA-1.5 and Qwen2-VL, with general-task performance on MME, GQA, and LLaVA-Bench roughly preserved.
  • The contribution maps make token-level visual grounding visible, allowing users to see which image region drove a given word and to catch cases where CHAIR flags a visually correct word as a hallucination.
  • Because the whole pipeline is training-free and built from a small set of paired samples, it can be applied to open-source LVLMs by shifting hidden states at inference time.
  • The method's usefulness is tied to spatial alignment: the paper states it works when visual features enter the LLM through a linear projection that preserves layout, and degrades for architectures such as Q-former or multi-scale encoders that compress or rearrange visual tokens.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would expect the same interpretation-then-steer recipe to generalize beyond hallucination, for example to typographic attacks or any failure mode in which the model attends to the wrong image region; the paper already shows a typographic-deception case where the maps reveal misdirected attention.
  • If the maps are taken seriously, hallucination evaluation could be redefined at the pixel level: a word is a true hallucination only when the regions that most raised its probability do not contain the claimed object, rather than when the word is missing from a caption-based ground-truth list.
  • A decisive and inexpensive test of the map-fidelity premise would be a large-sample comparison of relevance-guided masking against random masking at matched mask ratios.
  • For Q-former and multi-scale-encoder models, the paper's limitation discussion implies that a token-alignment or feature-resampling step would need to be inserted before the same loop could reach LLaVA-level gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes VaLSe, a training-free framework for object-hallucination mitigation and interpretability in LVLMs. The method selects visual-based tokens via a log-likelihood ratio (Eq. 3), computes visual contribution maps through gradient-weighted attention rollout (Eq. 5), removes activation artifacts by contrasting with special tokens, and uses these maps to mask low-contribution image regions and construct positive/negative feature pairs. Singular value decomposition of the feature differences provides a steering vector, which is added to LLM hidden states during inference. The authors report reduced hallucination on CHAIR, AMBER, POPE, MMHal, and MMVP, with preserved general ability on MME, GQA, and LLaVA-Bench, and they provide qualitative case studies arguing that existing CHAIR labels can be false positives. The main claims are that VaLSe produces faithful visual contribution maps and that steering by these maps reduces object hallucination.

Significance. If the central claim is upheld, VaLSe would be a useful contribution: it is training-free, provides a unified interpretation-and-mitigation pipeline, and reports a substantial CHAIR improvement on LLaVA-1.5 (sentence-level hallucination from 48.7 to 36.2 in Table 2). The random-masking ablation in Table 7 is a genuine attempt to isolate the contribution of relevance-guided masking, and the inclusion of code is a practical strength. The preservation of general performance on GQA, LLaVA-Bench, and MME is also encouraging. However, the paper's two central claims—map fidelity and the causal role of those maps in mitigation—are coupled, and the current evidence for map fidelity is thin. The significance of the work therefore depends on the authors' ability to decouple the interpretation and mitigation claims and to validate the contribution maps more rigorously than the current 8-sample evaluation.

major comments (5)
  1. [§3.2, Eq. (5), and Appendix F] The visual contribution maps are the foundation of both the interpretability and the steering mechanism, yet their fidelity is validated on only 8 samples, and the paper itself states that the deletion/insertion curves 'do not exhibit a consistent trend.' This is load-bearing because the same maps are used to construct the positive samples for the steering direction in §3.2; if the maps do not accurately localize the image evidence actually used by the model, then the steering direction may be driven by arbitrary masking artifacts rather than by vision-aware grounding. Please provide a more extensive quantitative validation of Eq. (5), for example a pointing-game comparison against segmentation masks or a larger deletion/insertion study with confidence intervals, and disclose the artifact-elimination details that are currently deferred to the supplementary materials.
  2. [§3.2 and Appendix B] Appendix B concedes that for MiniGPT-4 and Qwen2-VL, the Q-former, pixel-shuffle, or multi-scale encoders destroy the spatial correspondence required by Eq. (5). Yet Tables 1 and 2 still report VaLSe gains on these models. This creates a tension: if the contribution maps are unreliable for these architectures, the reported improvements cannot be attributed to the claimed vision-aware localization mechanism. To make the central claim defensible, either restrict the vision-aware interpretation claim to spatially aligned architectures such as LLaVA-1.5, or provide evidence on these models that the gains are mediated by the maps—for example, by comparing steering directions derived from contribution maps with direction derived from random or constant masks on the same models.
  3. [§3.2 and Appendix D.2] The description of positive-sample construction is internally inconsistent. Section 3.2 says that for each sample the method masks p% of image patches with low contribution values, and the ablations in Tables 4 and 5 vary p. Appendix D.2, however, says 'Instead of masking a fixed percentage p, we adopt an adaptive strategy by masking all tokens whose relevance scores are below the mean relevance value.' This ambiguity makes the method unreproducible and blurs what the p-ablation actually demonstrates. Please specify the exact masking rule used in the main experiments and in each ablation table.
  4. [Appendix D.2] The intervention strength β is tuned per model and per benchmark (e.g., β=0.5 for LLaVA-1.5 on CHAIR and AMBER but β=0.4 on other experiments; β=0.2 for Qwen2-VL on MMVP and MME but β=0.5 on other experiments), and α is set per model. Because the headline claim is that VaLSe 'outperforms all compared methods' on CHAIR, this per-benchmark tuning poses a risk of overfitting to the specific evaluation. Please report results under a single fixed hyperparameter configuration, or provide a sensitivity analysis that covers the full range of β and α on all benchmarks, so that the reader can assess the robustness of the reported gains.
  5. [§3.3, Eq. (8)] The theoretical justification relies on the assumptions f(A(noise))=0 and that the remainder R is negligible, but neither is justified. For real LVLMs, replacing the image with noise does not generally zero out attention matrices, and the higher-order terms in the Taylor expansion are not controlled. As written, Eq. (8) is a heuristic analogy rather than a proof. If the purpose is only to provide intuition, please state this explicitly; otherwise, the argument needs to be made rigorous or removed from the central narrative.
minor comments (5)
  1. [§3.2, 'Artifacts Elimination'] The artifact-elimination procedure is described only qualitatively ('identify positions P exhibiting artifacts') and is deferred to the supplementary materials. Please provide the full procedure in the main text or appendix, since it is essential for reproducing the contribution maps.
  2. [§3.2, Eq. (3)] The 'noise image' eI is not defined. Please specify its distribution, size, and how it is passed through the LVLM in the single forward pass described in the text.
  3. [Figure 3 and Table 3] Figure 3 reports MME results in a plot without numeric values. Since the text claims 'improved performance in color and positional understanding' and 'notable gains in OCR and code-related tasks,' please provide the exact numerical scores for all MME subtasks.
  4. [Appendix D.1] The model list includes 'Mipha-3B' in addition to the models evaluated in the main text, but no experimental results for Mipha-3B are reported anywhere. Please clarify its role or remove it.
  5. [§4.2] The four hallucination categories ('truly hallucinated,' 'factual,' 'unclear,' and 'false' hallucinated words) are each supported by a single qualitative example. Please state whether these categories were observed at quantified frequencies on a larger sample, or explicitly present them as illustrative rather than measured taxonomies.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Eq. 5 and the steering vector are built from external attention-rollout methods and SVD of paired samples, with benchmark outcomes measured against external annotations.

full rationale

VaLSe's contribution maps are computed by a gradient-weighted attention rollout that explicitly follows Chefer et al. [45,46]; the paper does not define a map's correctness in terms of CHAIR or the steering result. The steering direction is the top right singular vector of E_l = X_l^+ - X_l^-, where positive samples are masked images; this is a function of the maps, but the CHAIR, POPE, AMBER, MMHal, and MMVP targets are external ground-truth/annotation-based metrics, so no reported number reduces to the maps by construction. Hyperparameters alpha, p, and beta are tuned on the evaluation benchmarks (Appendices D.2 and E), which may inflate results, but this is a test-set-tuning concern, not an equation-level circularity; moreover, Table 7 shows relevance-guided masking outperforms random masking, so the central mitigation result has independent content. Appendix B concedes that for Q-former/pixel-shuffle and multi-scale encoders the spatial correspondence required by Eq. 5 is degraded, and Appendix F reports insertion/deletion curves on only 8 samples with 'no consistent trend'; these are validity and evidence-strength limitations, not circular reductions. The only self-citation (Nullu [11]) appears in related-work lists of steering methods and is not load-bearing. No claimed derivation is equivalent to its input by definition, and no fitted parameter is renamed as a prediction of the evaluation metric it was fitted to.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The method relies on three fitted hyperparameters, several domain assumptions inherited from prior interpretability work, and one ad hoc step in its explanatory analysis. None of these invalidate the empirical claims by themselves, but they determine how much of the reported improvement is attributable to the proposed mechanism versus tuning.

free parameters (3)
  • alpha (LLR threshold) = 3 for LLaVA-1.5 and Qwen2-VL, 1.8 for MiniGPT-4
    Controls which tokens are treated as visual-based for contribution maps and steering construction; tuned empirically per model based on LLR distribution (Appendix D.2).
  • masking ratio p = 0.9
    Controls how many low-relevance image patches are masked when building positive steering samples; tuned through CHAIR ablations (Table 5), though Appendix D.2 also describes an adaptive mean-based rule, leaving ambiguity.
  • intervention strength beta = 0.4 to 0.5 depending on model and benchmark
    Scales the latent shift applied at inference; set separately for CHAIR and AMBER versus other benchmarks for LLaVA-1.5 and separately for MMVP and MME versus others for Qwen2-VL (Appendix D.2).
assumptions (6)
  • domain assumption Gradient-weighted attention propagation (Eq. 5) faithfully estimates visual-token relevance for an LVLM output token.
    Invoked in Section 3.2 for visual contribution maps; inherited from Chefer et al.; the paper validates it on only 8 samples (Appendix F), so map fidelity is assumed, not proven.
  • domain assumption Artifact activations occur at fixed spatial positions and can be removed by contrasting with a non-semantic special token's contribution map.
    Section 3.2, Artifacts Elimination; based on refs [24-26], with details deferred to supplementary materials.
  • domain assumption Replacing the input image with a noise image gives a valid no-visual-context baseline for the log-likelihood ratio in Eq. 3.
    Section 3.2, Visual-based Token Selection; if the noise image is not truly content-free, token selection is distorted.
  • domain assumption A single top singular direction of the positive-minus-negative MLP feature difference, added at every layer, shifts generation toward desirable visual focus.
    Section 3.2, Vision-aware Latent Steering; follows VTI [21] and is empirical, with per-benchmark beta tuning.
  • ad hoc to paper In the Taylor analysis, f(A(noise))=0 and the remainder R is negligible.
    Eq. 8, Section 3.3; required to identify the contribution-map term with VCD-style debiasing, but not justified.
  • standard math SVD decomposition of the feature difference and first-order Taylor expansion are standard tools.
    Used in Eq. 7 and Eq. 8; these are standard and not points of contention.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations." pith.science (2026). https://pith.science/paper/27XOPCZW

@misc{pith2026250517812,
  author       = {Pith},
  title        = {Pith review of: Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/27XOPCZW}},
  note         = {Machine review of arXiv:2505.17812}
}
read the original abstract

Large Vision-Language Models (LVLMs) have achieved remarkable success but continue to struggle with object hallucination (OH), generating outputs inconsistent with visual inputs. While previous work has proposed methods to reduce OH, the visual decision-making mechanisms that lead to hallucinations remain poorly understood. In this paper, we propose VaLSe, a Vision-aware Latent Steering framework that adopts an interpretation-then-mitigation strategy to address OH in LVLMs. By tackling dual challenges of modeling complex vision-language interactions and eliminating spurious activation artifacts, VaLSe can generate visual contribution maps that trace how specific visual inputs influence individual output tokens. These maps reveal the model's vision-aware focus regions, which are then used to perform latent space steering, realigning internal representations toward semantically relevant content and reducing hallucinated outputs. Extensive experiments demonstrate that VaLSe is a powerful interpretability tool and an effective method for enhancing model robustness against OH across multiple benchmarks. Furthermore, our analysis uncovers limitations in existing OH evaluation metrics, underscoring the need for more nuanced, interpretable, and visually grounded OH benchmarks in future work. Code is available at: https://github.com/Ziwei-Zheng/VaLSe.

Figures

Figures reproduced from arXiv: 2505.17812 by the authors.

Figure 1
Figure 1. The proposed VaLSe can effectively (a) eliminate artifacts and provide high quality [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. VaLSe mainly contains three modules: (a) A visualization module that generates visual [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Results on MME. Model GQA LLaVA-Bench Binary Open Acc. Acc. Detail. LLaVA-1.5 77.9 47.1 61.2 5.4 5.2 VaLSe 78.3 46.9 61.3 6.2 5.8 Qwen2-VL 83.1 45.1 62.5 7.0 6.5 VaLSe 82.6 45.3 62.4 7.3 6.5 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The visualization and analysis results via VaLSe of four different types of hallucination [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Further analysis with visualization results using LLaVA-1.5. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Visualization results of different four LVLMs using VaLSe. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Insertion and deletion curves on 8 samples using three different visualization methods on [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Real-world applications of the proposed system. [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Gradio Demo of VaLSe for LLaVA-1.5-7b Visualization. [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Additional visualization results for four different LVLMs using VaLSe. [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Additional visualization results for four different LVLMs using VaLSe. [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Cases of VaLSe on LLaVA-Bench. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 34 canonical work pages

  1. [2]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  2. [3]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    arXiv 2023

  3. [4]

    Visual instruction tuning.Adv

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Adv. Neural Inform. Process. Syst., 2024

    work page 2024

  4. [5]

    Improved baselines with visual instruction tuning.arXiv preprint arXiv:2310.03744, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning.arXiv preprint arXiv:2310.03744, 2023

    arXiv 2023

  5. [6]

    Instructblip: Towards general-purpose vision-language models with instruction tuning.arXiv preprint arXiv:2305.06500, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning.arXiv preprint arXiv:2305.06500, 2023

    arXiv 2023

  6. [7]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    arXiv 2023

  7. [8]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  8. [9]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    arXiv 2024

Show all 64 references
  1. [10]

    Hallucination of multimodal large language models: A survey.arXiv preprint arXiv:2404.18930, 2024

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey.arXiv preprint arXiv:2404.18930, 2024

    2024 arXiv

  2. [11]

    Nullu: Mitigating object hallucinations in large vision-language models via halluspace projection

    Le Yang, Ziwei Zheng, Boxu Chen, Zhengyu Zhao, Chenhao Lin, and Chao Shen. Nullu: Mitigating object hallucinations in large vision-language models via halluspace projection. InIEEE Conf. Comput. Vis. Pattern Recog., 2025

    2025

  3. [12]

    Truthprint: Mitigating lvlm object hallucination via latent truthful-guided pre-intervention.arXiv preprint arXiv:2503.10602, 2025

    Jinhao Duan, Fei Kong, Hao Cheng, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. Truthprint: Mitigating lvlm object hallucination via latent truthful-guided pre-intervention.arXiv preprint arXiv:2503.10602, 2025

    2025

  4. [13]

    Analyzing and mitigating object hallucination in large vision-language models

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. InInt. Conf. Learn. Represent., 2024

    2024

  5. [14]

    Mitigating hallucination in large multi-modal models via robust instruction tuning.arXiv preprint arXiv:2306.14565, 2023

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning.arXiv preprint arXiv:2306.14565, 2023

    2023 arXiv

  6. [15]

    Hallucination augmented contrastive learning for multimodal large language model

    Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. InIEEE Conf. Comput. Vis. Pattern Recog., 2024. 10

    2024

  7. [16]

    Exposing and mitigating spurious correlations for cross-modal retrieval

    Jae Myung Kim, A Koepke, Cordelia Schmid, and Zeynep Akata. Exposing and mitigating spurious correlations for cross-modal retrieval. InIEEE Conf. Comput. Vis. Pattern Recog., 2023

    2023

  8. [17]

    Mitigating object hallucinations in large vision-language models through visual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In IEEE Conf. Comput. Vis. Pattern Recog., 2024

    2024

  9. [18]

    Debiasing large visual language models

    Yi-Fan Zhang, Weichen Yu, Qingsong Wen, Xue Wang, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Debiasing large visual language models. InEur. Conf. Comput. Vis., 2024

    2024

  10. [19]

    Halc: Object hallucination reduction via adaptive focal-contrast decoding

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Halc: Object hallucination reduction via adaptive focal-contrast decoding. InInt. Conf. Machine Learn., 2024

    2024

  11. [20]

    Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large vision-language models.arXiv preprint arXiv:2411.15268, 2024

    Junzhe Chen, Tianshu Zhang, Shiyu Huang, Yuwei Niu, Linfeng Zhang, Lijie Wen, and Xuming Hu. Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large vision-language models.arXiv preprint arXiv:2411.15268, 2024

    2024 arXiv

  12. [21]

    Reducing hallucinations in large vision-language models via latent space steering

    Sheng Liu, Haotian Ye, and James Zou. Reducing hallucinations in large vision-language models via latent space steering. InInt. Conf. Learn. Represent., 2025

    2025

  13. [22]

    Where do large vision-language models look at when answering questions?arXiv preprint arXiv:2503.13891, 2025

    Xiaoying Xing, Chia-Wen Kuo, Li Fuxin, Yulei Niu, Fan Chen, Ming Li, Ying Wu, Longyin Wen, and Sijie Zhu. Where do large vision-language models look at when answering questions?arXiv preprint arXiv:2503.13891, 2025

    2025 arXiv

  14. [23]

    Lvlm-intrepret: An interpretability tool for large vision-language models, 2024

    Gabriela Ben Melech Stan, Raanan Yehezkel Rohekar, Yaniv Gurwicz, Matthew Lyle Olson, Anahita Bhiwandiwalla, Estelle Aflalo, Chenfei Wu, Nan Duan, Shao-Yen Tseng, and Vasudev Lal. Lvlm-intrepret: An interpretability tool for large vision-language models, 2024

    2024

  15. [24]

    See what you are told: Visual attention sink in large multimodal models.arXiv preprint arXiv:2503.03321, 2025

    Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models.arXiv preprint arXiv:2503.03321, 2025

    2025 arXiv

  16. [25]

    Vision transformers need registers

    Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16588, 2023

    2023 arXiv

  17. [26]

    Massive activations in large language models

    Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402.17762, 2024

    2024 arXiv

  18. [27]

    Object hallucination in image captioning.arXiv preprint arXiv:1809.02156, 2018

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning.arXiv preprint arXiv:1809.02156, 2018

    2018 arXiv

  19. [28]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. InIEEE Conf. Comput. Vis. Pattern Recog., 2024

    2024

  20. [29]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04840, 2024

    2024 arXiv

  21. [30]

    Llava-phi: Efficient multi-modal assistant with small language model

    Yichen Zhu, Minjie Zhu, Ning Liu, Zhiyuan Xu, and Yaxin Peng. Llava-phi: Efficient multi-modal assistant with small language model. InProceedings of the 1st International Workshop on Efficient Multimedia Computing under Limited, 2024

    2024

  22. [31]

    Deepseek-vl: towards real-world vision-language understanding.arXiv preprint arXiv:2403.05525, 2024

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision-language understanding.arXiv preprint arXiv:2403.05525, 2024

    2024 arXiv

  23. [32]

    Detecting and preventing hallucinations in large vision language models

    Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. InAAAI, 2024

    2024

  24. [33]

    Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023

    2023 arXiv

  25. [34]

    Dress: Instructing large vision-language models to align and interact with humans via natural language feedback

    Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Dress: Instructing large vision-language models to align and interact with humans via natural language feedback. InIEEE Conf. Comput. Vis. Pattern Recog., 2024

    2024

  26. [35]

    Woodpecker: Hallucination correction for multimodal large language models.arXiv preprint arXiv:2310.16045, 2023

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models.arXiv preprint arXiv:2310.16045, 2023. 11

    arXiv

  27. [36]

    Mitigating object hallucination in large vision-language models via image-grounded guidance

    Linxi Zhao, Yihe Deng, Weitong Zhang, and Quanquan Gu. Mitigating object hallucination in large vision-language models via image-grounded guidance. InNeurips Safe Generative AI Workshop, 2024

    2024

  28. [37]

    Paying more attention to image: A training-free method for alleviating hallucination in lvlms

    Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. InEur. Conf. Comput. Vis., 2024

    2024

  29. [38]

    Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding.arXiv preprint arXiv:2402.18476, 2024

    Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding.arXiv preprint arXiv:2402.18476, 2024

    2024 arXiv

  30. [39]

    Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. InIEEE Conf. Comput. Vis. Pattern Recog., 2024

    2024

  31. [40]

    Multi-modal hallucination control by visual information grounding

    Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual information grounding. InIEEE Conf. Comput. Vis. Pattern Recog., 2024

    2024

  32. [41]

    Alphaedit: Null-space constrained knowledge editing for language models.arXiv preprint arXiv:2410.02355, 2024

    Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He, and Tat- Seng Chua. Alphaedit: Null-space constrained knowledge editing for language models.arXiv preprint arXiv:2410.02355, 2024

    2024 arXiv

  33. [42]

    Agla: Mitigating object hallucinations in large vision-language models with assembly of global and local attention.arXiv preprint arXiv:2406.12718, 2024

    Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations in large vision-language models with assembly of global and local attention.arXiv preprint arXiv:2406.12718, 2024

    2024 arXiv

  34. [43]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. InInt. Conf. Comput. Vis., 2017

    2017

  35. [44]

    Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks

    Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In2018 IEEE winter conference on applications of computer vision (WACV), 2018

    2018

  36. [45]

    Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers

    Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. InInt. Conf. Comput. Vis., 2021

    2021

  37. [46]

    Transformer interpretability beyond attention visualization

    Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. InIEEE Conf. Comput. Vis. Pattern Recog., 2021

    2021

  38. [47]

    Vl- interpret: An interactive visualization tool for interpreting vision-language transformers

    Estelle Aflalo, Meng Du, Shao-Yen Tseng, Yongfei Liu, Chenfei Wu, Nan Duan, and Vasudev Lal. Vl- interpret: An interactive visualization tool for interpreting vision-language transformers. InIEEE Conf. Comput. Vis. Pattern Recog., 2022

    2022

  39. [48]

    Fastrm: An efficient and automatic explainability framework for multimodal generative models.arXiv preprint arXiv:2412.01487, 2024

    Gabriela Ben-Melech Stan, Estelle Aflalo, Man Luo, Shachar Rosenman, Tiep Le, Sayak Paul, Shao-Yen Tseng, and Vasudev Lal. Fastrm: An efficient and automatic explainability framework for multimodal generative models.arXiv preprint arXiv:2412.01487, 2024

    2024 arXiv

  40. [49]

    Explaining multi-modal large language models by analyzing their vision perception.arXiv preprint arXiv:2405.14612, 2024

    Loris Giulivi and Giacomo Boracchi. Explaining multi-modal large language models by analyzing their vision perception.arXiv preprint arXiv:2405.14612, 2024

    2024 arXiv

  41. [50]

    From redundancy to relevance: Information flow in lvlms across reasoning tasks.arXiv preprint arXiv:2406.06579, 2024

    Xiaofeng Zhang, Yihao Quan, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie, Wenxiao Wang, Chaochen Gu, Hao Tang, and Jieping Ye. From redundancy to relevance: Information flow in lvlms across reasoning tasks.arXiv preprint arXiv:2406.06579, 2024

    2024 arXiv

  42. [51]

    Finding and editing multi-modal neurons in pre-trained transformers.arXiv preprint arXiv:2311.07470, 2023

    Haowen Pan, Yixin Cao, Xiaozhi Wang, Xun Yang, and Meng Wang. Finding and editing multi-modal neurons in pre-trained transformers.arXiv preprint arXiv:2311.07470, 2023

    2023 arXiv

  43. [52]

    An llm-free multi-dimensional benchmark for mllms hallucination evaluation.arXiv preprint arXiv:2311.07397, 2023

    Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An llm-free multi-dimensional benchmark for mllms hallucination evaluation.arXiv preprint arXiv:2311.07397, 2023

    2023 arXiv

  44. [53]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  45. [54]

    Aligning large multimodal models with factually augmented rlhf

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. InAnnual Meeting of the Association for Computational Linguistics, 2024. 12

    2024

  46. [55]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.Adv

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.Adv. Neural Inform. Process. Syst., 2022

    2022

  47. [56]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  48. [57]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InIEEE Conf. Comput. Vis. Pattern Recog., 2019

    2019

  49. [58]

    Beam search strategies for neural machine translation.arXiv preprint arXiv:1702.01806, 2017

    Markus Freitag and Yaser Al-Onaizan. Beam search strategies for neural machine translation.arXiv preprint arXiv:1702.01806, 2017

    2017 arXiv

  50. [59]

    Dola: Decoding by contrasting layers improves factuality in large language models.arXiv preprint arXiv:2309.03883, 2023

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models.arXiv preprint arXiv:2309.03883, 2023

    2023 arXiv

  51. [60]

    Blended diffusion for text-driven editing of natural images

    Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. InIEEE Conf. Comput. Vis. Pattern Recog., 2022

    2022

  52. [61]

    Unveiling typographic deceptions: Insights of the typographic vulnerability in large vision-language models

    Hao Cheng, Erjia Xiao, Jindong Gu, Le Yang, Jinhao Duan, Jize Zhang, Jiahang Cao, Kaidi Xu, and Renjing Xu. Unveiling typographic deceptions: Insights of the typographic vulnerability in large vision-language models. InEur. Conf. Comput. Vis., 2024

    2024

  53. [62]

    Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network

    Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. InIEEE Conf. Comput. Vis. Pattern Recog., 2016

    2016

  54. [63]

    Towards interpreting visual information processing in vision-language models

    Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual information processing in vision-language models. InInt. Conf. Learn. Represent., 2025

    2025

  55. [64]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  56. [65]

    Please describe this image in detail

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms.arXiv preprint arXiv:2401.06209, 2024. 13 A Broader Impacts This work presents VaLSe, a training-free framework for mitigating ob...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.