Pith. sign in

REVIEW 3 major objections 3 minor 31 references

Vision-language models can be made to ground their answers in the image by reading the model's own 'gaze shifts'—positive jumps in visual attention as it processes the question—and using that map to amplify attention to the right regions du

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 08:09 UTC pith:BOVAFVCJ

load-bearing objection Novel gaze-shift saliency idea, but the key ablation is missing — worth a serious referee, not a clean accept. the 3 major comments →

arxiv 2510.22067 v3 pith:BOVAFVCJ submitted 2025-10-24 cs.CV

Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation

Zheng Qi , Chao Shang , Evangelia Spiliopoulou , Nikolaos Pappas This is my paper
classification cs.CV
keywords vision-language modelshallucination mitigationattention steeringvisual saliencygaze shiftcross-modal fusionattention sinkinference-time intervention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tackles hallucination in vision-language models—outputs that cannot be supported by the image or the question. It proposes GIFT, an inference-time correction that first watches where the model's visual attention jumps, or 'gaze shifts,' as it reads information-rich words in the user's question, and turns those jumps into a saliency map of the image regions that matter for the task. During decoding, GIFT amplifies attention to those salient visual regions and simultaneously scales up attention to the question tokens, preserving the balance between the two modalities. Across LLaVA-1.5 7B/13B and Qwen2-VL 7B, the method reduces hallucination on CHAIR, POPE, and MMHal-Bench—up to 20.7% better than greedy decoding on CHAIR—while leaving general vision-language benchmarks roughly unchanged and adding only about 13% latency. The paper's bet is that a single precomputed map, derived from query comprehension, can guide the entire generation.

Core claim

The central claim is that positive changes in visual-token attention across consecutive information-rich query tokens—not raw attention values—reveal the image regions a task needs. GIFT forms a saliency map Ŝ by averaging these positive shifts over the top 50% of visual-attending heads at the layer where shifts peak (Eq. 3). During decoding, in chosen fusion layers, it multiplies visual attention to token j by exp(αŜ_j) and query attention by β times the visual amplification ratio, keeping cross-modal balance. After renormalization, this steers the model toward grounded regions and away from attention-sink tokens. A 1,000-image bounding-box test shows the shift map concentrates saliency i

What carries the argument

The carrying object is the gaze-shift saliency map: the average, over top-50% visual-attending heads and over consecutive pairs of information-rich query tokens, of the positive part of the attention difference to each visual token. This map is computed once during prefill and replaces per-step visual attention as the guide for attention steering. The second piece is balanced enhancement: visual attention is rescaled by exp(αŜ_j) and query attention by a proportional factor, so cross-modal fusion is not skewed.

Load-bearing premise

The load-bearing premise is that positive jumps in a vision-language model's visual attention while it reads content words in a question point to the image regions that actually answer that question—if those jumps track syntactic surprise or local attention noise instead, the saliency map amplifies the wrong places and the method's gains disappear.

What would settle it

Take a set of images with two questions that target different objects (e.g., 'What color is the left car?' vs. 'What color is the right car?') and measure the overlap of the resulting gaze-shift maps. If the maps do not move to the queried side, or if replacing the positive-shift signal with random positive increments yields the same hallucination scores, the core mechanism is not doing the claimed work.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If gaze shifts locate task-relevant regions, then steering decoding toward those regions reduces object hallucination on captioning (CHAIR) and VQA (MMHal-Bench, POPE), with the largest reported gains on captioning.
  • Because the map is computed once during prefilling, the method avoids the doubled forward passes of contrastive decoding; measured latency is 1.13× greedy decoding on LLaVA-1.5 7B.
  • Preserving cross-modal balance matters: ablations that boost only visual attention, or only recalibrate within the visual stream, underperform the joint enhancement by up to 21.9% on the tested hallucination metrics.
  • Setting the enhancement coefficient α too high trades reasoning for grounding: on MME, α values beyond 5.0 in LLaVA-1.5 7B fall below greedy decoding, so the method's gains depend on tuning α per model.
  • The method transfers across architectures (LLaVA-1.5 7B/13B, Qwen2-VL 7B) with model-specific layer choices and coefficients, suggesting it is not tied to one attention pattern.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension not tested here: the gaze-shift map could serve as a cheap, query-conditioned saliency oracle for other tasks—visual grounding, referring-expression comprehension, or input-image cropping—since it is computed in a single forward pass through early layers.
  • The layer that maximizes positive shift volume is chosen from just 50 sampled questions; a robustness check across domains would show whether this 'sweet-spot layer' is stable or must be re-tuned per benchmark.
  • GIFT targets a different failure mode than contrastive decoding and sink-recalibration methods, so combining them may yield further gains; the paper does not explore such combinations.
  • The choice β=1.0 preserves the model's original visual-to-query balance, but an adaptive β that increases when query attention is low, or that scales with the layer, is a direct follow-up the authors leave open.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes GIFT, an inference-time attention-steering method for reducing hallucination in vision-language models. GIFT first computes a visual saliency map by tracking positive changes in attention from information-rich query tokens to visual tokens (Eq. 3), then, during decoding, scales visual-token attention by exp(alpha * saliency) and query-token attention by a proportional factor at selected cross-modal fusion layers. Experiments are reported on LLaVA-1.5 7B/13B and Qwen2-VL 7B across CHAIR, POPE, MMHal-Bench, SEED-Bench, and MME, with a measured latency of 1.13x relative to greedy decoding. The central claim is that tracking 'gaze shifts' yields a saliency map that is robust to visual attention sinks and that jointly enhancing visual and query attention improves cross-modal fusion.

Significance. If validated, GIFT would provide a lightweight, training-free alternative to contrastive decoding and visual-input-modification hallucination mitigation methods. The idea of deriving task-relevant saliency from attention shifts rather than static averages is well-motivated by the attention-sink phenomenon and is testable. Strengths of the paper include released code, evaluation on three model families, ablations of the two scaling components, and explicit efficiency measurements. However, the central gaze-shift mechanism is not directly ablated on the downstream hallucination benchmarks, and several hyperparameters are tuned on subsets of the evaluation benchmarks. With additional controls, the paper could be a solid contribution; as written, the headline improvements are not fully out-of-sample and the main mechanistic claim lacks a decisive experiment.

major comments (3)
  1. [Appendix C, Tables 2 and 3] Hyperparameters alpha and the cross-modal fusion layer ranges are tuned on 10% 'pseudo-validation' sets sampled from POPE and MME. The reported POPE and MME numbers in Tables 2 and 3 therefore use hyperparameters selected on subsets of those same benchmarks, making these gains partly in-sample. The 3.0% POPE improvement and the MME comparisons are not fully out-of-sample. Please provide an evaluation with hyperparameters fixed without peeking at the test benchmarks, or tune on a separate dataset such as TextVQA and report results on the full benchmarks under that fixed configuration.
  2. [Section 3, Table 1, Table 4] The paper's main novelty and motivation rest on the claim that the gaze-shift saliency map (Eq. 3) is superior to the static average map (Eq. 2), particularly for suppressing attention sinks. The only direct static-vs-shift evidence is Table 1, a bounding-box proxy on 1,000 MSCOCO images. The full GIFT pipeline is never run with the static map substituted for the shift map on CHAIR, POPE, or MMHal-Bench. Table 4 varies the scaling strategy while keeping the shift map fixed, so it cannot separate the saliency-map choice from the cross-modal enhancement. Please add a downstream ablation replacing Eq. 3 with Eq. 2 on the hallucination benchmarks; this is necessary to establish that the gaze-shift mechanism, not just the attention/query scaling, is responsible for the gains.
  3. [Section 6 vs. Section 5.2] Table 4 explicitly excludes CHAIR because 'image captioning queries lack sufficient specificity for query attention enhancement to meaningfully improve cross-modal representations.' Yet the headline 20.7% improvement over greedy decoding is on CHAIR (Table 2). If the user query is too vague for meaningful query attention enhancement, it is unclear why GIFT produces the largest relative gains on CHAIR. This is an internal inconsistency in the evaluation logic. Please either report the Inc. V. and Cal. V. ablations on CHAIR or explain the mechanism by which GIFT improves CHAIR despite uninformative query tokens.
minor comments (3)
  1. [Equation (3) and Section 4] The clipping at three standard deviations is described in Section 4 but is not reflected in Eq. (3), which only shows min-max normalization. Specify the order of clipping and normalization for reproducibility.
  2. [Table 1] The normalized saliency scores (5.40 vs. 11.92) are reported without statistical significance, confidence intervals, or variance across the 1,000 examples. Add error bars or a paired test to support the comparison.
  3. [Section 4] There is a typo: 'intepreting' should be 'interpreting.' Also, the term 'visual attention positive shift' in Figure 2 is not formally defined; define it precisely when introducing Figure 2 to avoid confusion with Eq. (3).

Circularity Check

0 steps flagged

No circularity found: the gaze-shift saliency map is an empirically tested input signal, not a rescaling of the figures it is used to predict.

full rationale

GIFT is an empirical inference-time intervention rather than a mathematical derivation, so the definitional circularity categories do not apply. The saliency map in Eq. (3) is computed from positive changes in prefill visual attention over POS-tagged query tokens; Eqs. (6) and (7) then use that precomputed map to rescale visual and query attention during decoding. The map is not fitted to the downstream hallucination benchmarks: the saliency layer is chosen from 50 TextVQA examples (Section 3, Appendix C), and the map itself is validated independently on 1,000 MSCOCO bounding-box examples (Table 1), where the shift map outperforms the static map. The headline hallucination gains on CHAIR and MMHal-Bench are not tuned on those benchmarks; only cross-modal fusion layers and alpha are tuned on 10% pseudo-validation subsets of POPE and MME (Appendix C), which is a validation-selection caveat for the POPE/MME numbers but not a circular reduction of the output to the input. The paper does not rely on load-bearing self-citation or an imported uniqueness theorem; its citations to Kang et al., Yin et al., and He et al. are external prior work. The central premise that gaze shifts identify task-relevant regions is an explicit empirical hypothesis, acknowledged as query-dependent in the paper's own limitation section, and tested against an external bounding-box signal; the lack of a direct static-vs-shift ablation on CHAIR/POPE/MMHal is a reporting gap, not a circularity.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 0 invented entities

GIFT rests on an empirical hypothesis about VLM attention dynamics (gaze shifts = saliency) that is supported only indirectly by benchmark and bounding-box evidence. Its quantitative gains depend on several hand-set or benchmark-tuned hyperparameters (alpha, layer ranges, head thresholds, POS tag set). No new physical entities are introduced. The method is not derived from first principles, so correctness risk is concentrated in whether the saliency signal transfers across models and query types.

free parameters (7)
  • alpha (visual attention enhancement coefficient) = 5.0 for LLaVA-1.5 7B/13B; 4.0 for Qwen2-VL 7B
    Scales exp(alpha * saliency) in Eq. 6; selected by sweeping 1.0–7.0 on POPE and MME pseudo-validation sets (Appendix C, Fig. 7).
  • beta (query attention enhancement coefficient) = 1.0 for all models
    Set by hand to 'preserve the original cross-modal fusion balance' (Eq. 7; Section 4); not tuned, but still a free choice.
  • cross-modal fusion enhancement layer range L = LLaVA-1.5 7B: 12-22; 13B: 14-20; Qwen2-VL 7B: 5-18
    Selected by grid search on 10% pseudo-validation sets from POPE and MME, constrained by attention proportion curves (Appendix C).
  • visual saliency map computation layer = 11 (LLaVA-1.5 7B), 10 (LLaVA-1.5 13B), 14 (Qwen2-VL 7B)
    Chosen as the layer with largest positive visual-attention shift, measured on 50 TextVQA training examples (Figure 2; Section 3).
  • top-50% attention-head selection threshold = 0.5
    Heuristic adopted from prior work for selecting visual heads (Eqs. 2-3); a fixed hyperparameter that affects saliency and fusion layers.
  • POS tag selection for information-rich tokens = NOUN, PROPN, VERB, ADJ, ADV, NUM
    Chosen by hand via spaCy POS tagging; determines which query tokens drive the gaze-shift map (Section 3).
  • saliency clipping threshold = 3 standard deviations before min-max normalization
    Ad hoc choice to reduce outlier effects in Eq. 6 (Section 4).
axioms (6)
  • domain assumption Attention matrices from VLM forward passes are accessible and can be additively/multiplicatively modified without corrupting generation.
    All of GIFT operates by rewriting attention scores before softmax/normalization (Eqs. 5-7); this assumes such steering is safe and effective.
  • domain assumption Positive changes in visual attention over information-rich query tokens ('gaze shifts') are a faithful and task-relevant saliency signal; irrelevant sink tokens show minimal shifts.
    Core hypothesis of Section 3 and Eq. 3; validated only indirectly via benchmarks and Table 1, not proven.
  • domain assumption Enhancing attention to query tokens alongside visual tokens preserves cross-modal fusion balance; boosting visual alone risks degrading query comprehension.
    Motivates Eq. 7 and the beta scaling; based on Figure 4's correlational attention analysis.
  • domain assumption The top-50% attention heads with highest cumulative visual attention are the functionally relevant heads for steering visual and query attention.
    Follows He et al. (2024), Yin et al. (2025), Kang et al. (2025); used in Eqs. 2-4 and selection of H_l_TV, H_l_OV, H_l_OT.
  • domain assumption 50 randomly sampled TextVQA training examples are representative for selecting the saliency-map layer across models and benchmarks.
    Layer selection in Section 3 uses only 50 examples; if unrepresentative, the saliency map quality degrades.
  • standard math Softmax attention with a bias term B and mask M is a valid model of VLM attention (Eq. 5).
    Standard attention-steering formulation from Zhang et al. (2023); GIFT's multiplicative scaling is equivalent after sum-normalization.

pith-pipeline@v1.3.0-alltime-deepseek · 15787 in / 13133 out tokens · 108150 ms · 2026-08-04T08:09:54.638803+00:00 · methodology

0 comments
read the original abstract

Vision language models (VLMs) often generate hallucination, i.e., content that cannot be substantiated by either textual or visual inputs. Prior work primarily attributes this to over-reliance on linguistic prior knowledge rather than visual inputs. Some methods attempt to mitigate hallucination by amplifying visual token attention proportionally to their attention scores. However, these methods overlook the visual attention sink problem, where attention is frequently misallocated to task-irrelevant visual regions, and neglect cross-modal fusion balance by enhancing only visual attention without adjusting attention to the user query. This can result in amplifying incorrect areas while failing to properly interpret the user query. To address these challenges, we propose a simple yet effective method called Gaze Shift-Guided Cross-modal Fusion Enhancement (GIFT). GIFT pre-computes a holistic visual saliency map by tracking positive changes in visual attention, or "gaze shifts", during user query comprehension, and leverages this map to amplify attention to both salient visual information and the user query at each decoding step. This reduces the impact of visual attention sink, as irrelevant tokens exhibit minimal shifts, while ensuring balanced cross-modal fusion for well-integrated representation. Extensive experiments show that GIFT effectively mitigates hallucination in VLMs across both generative and classification tasks, achieving up to 20.7% improvement over greedy decoding, while maintaining general vision-language performance with low computational overhead.

Figures

Figures reproduced from arXiv: 2510.22067 by Chao Shang, Evangelia Spiliopoulou, Nikolaos Pappas, Zheng Qi.

Figure 1
Figure 1. Figure 1: Examples of visual saliency maps from LLaVA 1.5 7B. The vanilla method (Static) averages visual token attention over all query tokens, while the proposed method (Shift) averages positive changes in visual token attention over information-rich query tokens. Shift more effectively highlights task-relevant visual regions and mitigates the impact of visual attention sink, where irrel￾evant regions receive erro… view at source ↗
Figure 2
Figure 2. Figure 2: Volume of visual attention positive shifts in VLMs when processing information-rich query tokens across layers. The volume reflects how strongly the model reallocates focus within visual regions, with the largest shifts occurring in early to middle layers, indicating that VLMs settle on relevant visual regions once identified. where y<t denotes the sequence of previously generated tokens. Within the model,… view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of GIFT. GIFT tracks positive changes in visual attention, i.e., “gaze shifts” across information-rich query tokens to compute a visual saliency map, which guides enhancement of salient visual attention while preserving cross-modal fusion balance. attention, emphasizing task-relevant regions: ˆSl = Min-max   1 |Hˆl T rV | · |XT r| X h∈Hˆl T rV X i∈XT r max(Al h,i,j − Al h,i−1,j , 0)   , j … view at source ↗
Figure 4
Figure 4. Figure 4: Attention proportions of query and visual tokens from output tokens across layers. The attention proportions of query and visual tokens follow similar patterns across layers, suggesting that effective cross-modal fusion relies on contributions from both modalities. tracks attention flow from query tokens to visual tokens, here we analyze flow from output tokens to both query and visual tokens. Using the 50… view at source ↗
Figure 6
Figure 6. Figure 6: Relative inference latency of different [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Performance on the POPE and MME datasets with varying enhancement coefficients [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: An example from the MMHal-Bench dataset. Hallucinations are highlighted in red. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: An example from the MMHal-Bench dataset. Hallucinations are highlighted in red. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: An example from the MMHal-Bench dataset. Hallucinations are highlighted in red. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: An example from the MMHal-Bench dataset. Hallucinations are highlighted in red. [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: An example from the MMHal-Bench dataset. Hallucinations are highlighted in red. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 25 linked inside Pith

  1. [1]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923,

    Pith/arXiv arXiv

  2. [3]

    Huatuogpt-vision, towards injecting medi- cal visual knowledge into multimodal llms at scale.arXiv preprint arXiv:2406.19280, 2024b

    Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xi- dong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, et al. Huatuogpt-vision, towards injecting medi- cal visual knowledge into multimodal llms at scale.arXiv preprint arXiv:2406.19280, 2024b. Timoth´ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transf...

    Pith/arXiv arXiv

  3. [6]

    Cracking the code of hallucination in lvlms with vision-aware head divergence.arXiv preprint arXiv:2412.13949,

    Jinghan He, Kuan Zhu, Haiyun Guo, Junfeng Fang, Zhenglin Hua, Yuheng Jia, Ming Tang, Tat- Seng Chua, and Jinqiao Wang. Cracking the code of hallucination in lvlms with vision-aware head divergence.arXiv preprint arXiv:2412.13949,

    Pith/arXiv arXiv

  4. [9]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    Pith/arXiv arXiv

  5. [10]

    See what you are told: Visual attention sink in large multimodal models.arXiv preprint arXiv:2503.03321,

    Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models.arXiv preprint arXiv:2503.03321,

    Pith/arXiv arXiv

  6. [11]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.In- ternational journal of computer vision, 128(7):1956–1981,

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Sha- hab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.In- ternational journal of computer vision, 128(7):1956–1981,

    1956

  7. [12]

    Seed-bench: Bench- marking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023a

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023a. Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-...

    Pith/arXiv arXiv

  8. [13]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pp. 19730–19742. PMLR, 2023c. Xiang Li, Cristina Mata, Jongwoo Park, Kumara Kahatapitiya, Yoo Sung Jang, Jinghuan Shang, Kanchana Ranasinghe, Ryan Bur...

    Pith/arXiv arXiv

  9. [14]

    Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355, 2023d

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355, 2023d. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InEurope...

    Pith/arXiv arXiv

  10. [16]

    Mitigating hallucinations in large vision-language models via summary-guided decoding.arXiv preprint arXiv:2410.13321,

    Kyungmin Min, Minbeom Kim, Kang-il Lee, Dongryeol Lee, and Kyomin Jung. Mitigating hallucinations in large vision-language models via summary-guided decoding.arXiv preprint arXiv:2410.13321,

    Pith/arXiv arXiv

  11. [17]

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko

    11 Preprint. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning.arXiv preprint arXiv:1809.02156,

    Pith/arXiv arXiv

  12. [18]

    Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525,

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525,

    Pith/arXiv arXiv

  13. [19]

    Mint: Mitigating hallucinations in large vision- language models via token reduction.arXiv preprint arXiv:2502.00717,

    Chao Wang, Jianming Yang, and Yang Zhou. Mint: Mitigating hallucinations in large vision- language models via token reduction.arXiv preprint arXiv:2502.00717,

    Pith/arXiv arXiv

  14. [20]

    Mllm can see? dynamic correction decoding for hallucination mitigation.arXiv preprint arXiv:2410.11779, 2024a

    Chenxi Wang, Xiang Chen, Ningyu Zhang, Bozhong Tian, Haoming Xu, Shumin Deng, and Huajun Chen. Mllm can see? dynamic correction decoding for hallucination mitigation.arXiv preprint arXiv:2410.11779, 2024a. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision...

    Pith/arXiv arXiv

  15. [21]

    Efficient streaming language models with attention sinks, 2024.URL https://arxiv

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024.URL https://arxiv. org/abs/2309.17453, 1,

    Pith/arXiv arXiv 2024

  16. [22]

    Tarac: Mitigating hallucination in lvlms via temporal attention real-time accumulative connection.arXiv preprint arXiv:2504.04099,

    Chunzhao Xie, Tongxuan Liu, Lei Jiang, Yuting Zeng, Yunheng Shen, Weizhe Huang, Jing Li, Xiaohua Xu, et al. Tarac: Mitigating hallucination in lvlms via temporal attention real-time accumulative connection.arXiv preprint arXiv:2504.04099,

    Pith/arXiv arXiv

  17. [23]

    Mllms know where to look: Training-free perception of small visual details with multimodal llms.arXiv preprint arXiv:2502.17422,

    Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. Mllms know where to look: Training-free perception of small visual details with multimodal llms.arXiv preprint arXiv:2502.17422,

    Pith/arXiv arXiv

  18. [24]

    Tell your model where to attend: Post-hoc attention steering for llms.arXiv preprint arXiv:2311.02262,

    Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, and Tuo Zhao. Tell your model where to attend: Post-hoc attention steering for llms.arXiv preprint arXiv:2311.02262,

    Pith/arXiv arXiv

  19. [25]

    Debiasing multimodal large language models.arXiv preprint arXiv:2403.05262,

    Yi-Fan Zhang, Weichen Yu, Qingsong Wen, Xue Wang, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Debiasing multimodal large language models.arXiv preprint arXiv:2403.05262,

    Pith/arXiv arXiv

  20. [26]

    Looking beyond text: Reducing language bias in large vision-language models via multi- modal dual-attention and soft-image guidance.arXiv preprint arXiv:2411.14279,

    Haozhe Zhao, Shuzheng Si, Liang Chen, Yichi Zhang, Maosong Sun, Mingjia Zhang, and Baobao Chang. Looking beyond text: Reducing language bias in large vision-language models via multi- modal dual-attention and soft-image guidance.arXiv preprint arXiv:2411.14279,

    Pith/arXiv arXiv

  21. [27]

    Minigpt-4: En- hancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592,

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592,

    Pith/arXiv arXiv

  22. [28]

    Is there a frisbee in the image?

    12 Preprint. Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating hallucina- tions in large vision-language models via image-biased decoding. InProceedings of the Computer Vision and Pattern Recognition Conference, pp. 1624–1633, 2025a. Younan Zhu, Linwei Tao, Minjing Dong, and Chang Xu. Mitigating object hallucinations in...

    arXiv 2014

  23. [29]

    is a benchmark designed to evaluate halluci- nation in vision language models (VLMs). It contains 96 challenging questions based on images from the OpenImages dataset (Kuznetsova et al., 2020), each paired with a corresponding ground- truth answers and annotated image content. Model responses are scored using GPT-4 through a pre-defined prompt that assess...

    2020

  24. [30]

    Please just answer yes or no

    is a benchmark designed to assess both perception and cognition capabilities of vision language models across 14 subtasks. Each subtask evaluates a specific aspect of visual understanding or reasoning capability. For all experiments, We report performance using the accuracy metric as defined in the original paper. SEED-Bench.SEED-Bench (Li et al., 2023a) ...

    2025

  25. [31]

    Layers for Cross-Modal Fusion Enhancement.Since the benchmarks we consider lack dedi- cated validation sets for hyperparameter tuning, we follow Kang et al

    Based on Figure 2, this corresponds to layer 11 for LLaV A-1.5 7B, layer 10 for LLaV A-1.5 13B, and layer 14 for Qwen2-VL 7B. Layers for Cross-Modal Fusion Enhancement.Since the benchmarks we consider lack dedi- cated validation sets for hyperparameter tuning, we follow Kang et al. (2025) by randomly sample 10% of the POPE and MME datasets as ”pseudo-vali...

    2025

  26. [2014]

    Mitigat- ing hallucination in large multi-modal models via robust instruction tuning.arXiv preprint arXiv:2306.14565, 2023a

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigat- ing hallucination in large multi-modal models via robust instruction tuning.arXiv preprint arXiv:2306.14565, 2023a. Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large visio...

    Pith/arXiv arXiv

  27. [2019]

    Self-introspective decoding: Alleviating hallucinations for large vision-language models.arXiv preprint arXiv:2408.02032,

    Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self-introspective decoding: Alleviating hallucinations for large vision-language models.arXiv preprint arXiv:2408.02032,

    Pith/arXiv arXiv

  28. [2020]

    Drew A Hudson and Christopher D Manning

    doi: 10.5281/zenodo.1212303. Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700–6709,

    doi:10.5281/zenodo.1212303

  29. [2023]

    Information flow routes: Automatically interpreting language models at scale.arXiv preprint arXiv:2403.00824,

    Javier Ferrando and Elena V oita. Information flow routes: Automatically interpreting language models at scale.arXiv preprint arXiv:2403.00824,

    Pith/arXiv arXiv

  30. [2024]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394,

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394,

    Pith/arXiv arXiv

  31. [2025]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455–14465, 2024a. Cong Chen, Mingyu Liu, Chenchen Jing, Yizhou Zhou, Fengyun Rao, Hao Chen,...

    Pith/arXiv arXiv