Pith. sign in

REVIEW 4 major objections 6 minor 48 references

CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CAI, a training-free method, mitigates object hallucination in large vision-language models by adding precomputed caption-query attention shifts to a handful of caption-sensitive attention heads at inference time.

desk verdict A cheap, training-free attention intervention that consistently reduces LVLM hallucination, but the stated visual mechanism is undercut by the shift vector's unmasked computation and the headline numbers are partly fitted. read the letter →

arxiv 2506.23590 v1 pith:YTJQQ2OJ submitted 2025-06-30 cs.CV

Qiming Li , Zekai Ye , Xiaocheng Feng , Weihong Zhong , Libo Qin , Ruihan Chen , Baohang Li , Kui Jiang
show 3 more authors
Yaowei Wang Ting Liu Bing Qin
This is my paper
classification cs.CV
keywords objecthallucinationlargevision-languagemodelsattentioninterventioncaption-sensitiveheadsinference-timemitigationtraining-freemethod
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large vision-language models routinely invent objects that are not in the image, and existing fixes either need expensive training data or slow down inference. This paper shows that the same models pay noticeably more attention to visual tokens when the prompt asks for a caption than when it asks a direct question, and that this gap can be reused as a steering signal. The proposed method, CAI, finds the attention heads that differ most between caption and non-caption queries, precomputes their output shift on a small set of pretraining samples, and adds that shift back at inference time. On POPE, MME, CHAIR, and MMHal-Bench, the intervention improves hallucination metrics across three open models with almost no added latency. The claim is that a fixed, image-agnostic shift toward the caption state is enough to restore fine-grained visual perception.

What carries the argument

The central object is the caption-sensitive attention head, defined as a head whose output at the last token differs strongly between caption and non-caption queries after text-related attention scores are masked. The method uses per-head binary classifiers to rank heads, then computes an attention output shift vector $S^{(l,h)}$ as the average of (caption output minus non-caption output) over a probing set. At inference this shift is added to the top-$K$ heads' outputs, scaled by a strength parameter $\alpha$, through the gated update $H^{l+1} = H^l + \sum_h (O^{(l,h)} + I^{(l,h)} \alpha S^{(l,h)}) W_o^l$. The shift is the load-bearing artifact: it encodes where the model's own visual attention goes when it is asked to describe rather than interrogate an image.

What would settle it

Collect a held-out set of images, compute the per-image caption-to-noncaption attention-output shift, and check whether hallucination improvements persist on images whose per-image shift points opposite to CAI's precomputed shift; if they do, the fixed steering direction is not the active ingredient.

Watch

Extended reading notes

Core claim

CAI's discovery is that caption queries put vision-language models into a reliably more visual state: 65.92% of attention heads and 30 of 32 layers increase their visual attention when the prompt is a caption. The paper treats this state as a target and steers non-caption inference toward it. Concretely, it trains one binary classifier per head to separate caption from non-caption query outputs, keeps the top-K heads by classification accuracy, and computes the average output difference between the two query types over 1,000 pretraining pairs. At inference, that difference is added to the selected heads' outputs before the output projection. The empirical claim is that this single precomputed shift makes LLaVA-1.5-7b, Qwen-VL-Chat, and LLaVA-NeXT hallucinate less on both discriminative and generative benchmarks while leaving other capabilities largely intact.

Load-bearing premise

The load-bearing premise is that the average attention-output difference between caption and non-caption queries, computed once on 1,000 pretraining samples, is a safe image-agnostic steering direction for every test image and query.

Editorial extensions

If this is right

  • On POPE, CAI raises accuracy by 3.30 to 6.58 points and F1 by 3.57 to 8.10 points over the vanilla model across LLaVA-1.5-7b, Qwen-VL-Chat, and LLaVA-NeXT.
  • On the MME hallucination subset, total scores increase by 50.0 to 76.2 points depending on the model, and on the full MME set LLaVA-1.5-7b improves on 13 of 14 tasks.
  • On MS-COCO CHAIR, sentence-level hallucination (CHAIRs) drops by 3.6 points and instance-level hallucination (CHAIRi) by 1.27 points, even when the prompt is itself a caption-style request.
  • On MMHal-Bench, the hallucination rate falls from 63.5% to 51.0% for LLaVA-1.5-7b and from 41.1% to 38.0% for Qwen-VL-Chat, while the informativeness score rises.
  • Inference cost stays near the vanilla model: time to first token changes from 99.8 ms to 102.2 ms and time per output token from 36.0 ms to 36.5 ms on LLaVA-1.5-7b, unlike contrastive decoding methods that multiply latency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves untested whether adapting the shift per image or per query would do better; if the pretraining-average shift is not the optimal direction, a lightweight per-image estimate could improve gains further.
  • The same 'expert query' pattern could transfer to other failure modes: find a query that reliably activates the desired behavior, identify the heads that separate it from a generic query, and steer those heads at inference.
  • Because CAI modifies attention outputs rather than logits, combining it with contrastive decoding is a natural next experiment the paper does not run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes CAI, a training-free, plug-and-play intervention for reducing object hallucination in large vision-language models. The authors observe that caption-style queries elicit stronger visual attention than non-caption queries in certain attention heads, and they exploit this by (1) searching over a small set of caption queries to find one with minimal attention shift, (2) training per-head SVM classifiers on masked attention outputs to identify caption-sensitive heads and computing average attention-output shift vectors from caption versus non-caption queries, and (3) adding scaled shift vectors to the selected heads during inference. The method is evaluated on POPE, MME, CHAIR, and MMHal-Bench using LLaVA-1.5-7b, Qwen-VL-Chat, and LLaVA-NeXT, reporting consistent improvements over several training-free baselines with minimal additional latency.

Significance. If the reported gains are reproducible, CAI would be a useful lightweight contribution to the hallucination-mitigation toolbox: it requires no LVLM fine-tuning, has a low inference overhead documented in Table 4, and is applied across three models and four benchmarks from a single set of pretrain-derived shift vectors. The paper's strengths are its clear three-stage formulation, the cross-model application of the same shift vectors, and the explicit latency comparison against contrastive-decoding baselines. However, the manuscript provides no code or random seeds, the two intervention hyperparameters are tuned on a test benchmark, and the mechanistic claim that the shift vector is vision-centric is undermined by the use of unmasked attention outputs in Eq. 9. The consistent direction of the gains is encouraging, but the current evidence is not yet statistically or methodologically strong enough for the SOTA claims made in the paper.

major comments (4)
  1. [§5.4, Appendix D, Table 1] The hyperparameters α and K are selected by maximizing accuracy on the MS-COCO Adversarial POPE test set in Figure 7 and Tables 5–6, and the same test set is then used to report the adversarial POPE row of Table 1; consequently, the headline adversarial numbers are fitted values rather than independent evaluations. This is load-bearing for the SOTA claim on POPE. Please report a validation-based selection procedure (for example, tuning on a held-out split or reporting the full grid over all settings) and add error bars or significance tests, since no repeated runs are reported anywhere in the paper.
  2. [§3.3, Eq. 9] The probing stage explicitly masks text-token attention in Eqs. 5–7 to isolate visual information, but the shift vector S^{(l,h)} in Eq. 9 is computed from the unmasked original attention outputs O_b^{(l,h)} and O'^{(l,h)}. Because caption and non-caption queries differ in wording and length, this average difference can encode language-prior components—such as attention to query tokens or generic sentence-structure differences—so the claim that S is a 'vision-centric optimization direction' is not yet supported. Please compute the shift from the masked outputs used for probing, or provide a control experiment (for example, length- and topic-matched non-caption queries, or a token-level decomposition of S) that isolates the visual contribution.
  3. [§4.2, Figure 4, Table 3] No error bars, confidence intervals, or significance tests are reported, and several claimed gains are small relative to the sample sizes: MMHal-Bench contains only 96 questions, and the VH-rate difference for Qwen-VL-Chat is 38.0 versus 38.4 for two of the baselines, while the CHAIR results are reported for a single model. Without repeated runs or an uncertainty estimate, the paper's central claim of consistent improvements is not statistically grounded. Please report multiple decoding seeds or bootstrap confidence intervals, or explicitly state that the gains are descriptive rather than statistically tested.
  4. [§4.2, Implementation Details] The paper states that the images used for probe and shift computation are 'outside the domain of the benchmark datasets,' but the probe samples come from the LLaVA pretraining dataset, which is built from image-caption corpora that include MS-COCO, while POPE and CHAIR evaluate on MS-COCO images. The claim of cross-domain discrepancy is therefore not justified without specifying the exact image splits (for example, COCO train versus val) and any overlap filtering. Please clarify the data sources and assess whether the COCO-based pretraining distribution materially overlaps the evaluation benchmarks.
minor comments (6)
  1. [§3.2, Eqs. 3–4] Please define the norm used to turn the attention-weight shift matrix into the scalar minimized in Eq. 4; as written, Eq. 4 appears to minimize a matrix-valued quantity over a finite index set.
  2. [§A.2 and §5.4] The main POPE experiments are described as using direct sampling decoding, but the ablation in Figure 7 and Appendix D is described as using greedy decoding; please clarify which decoding scheme is used for hyperparameter selection and for each reported result.
  3. [§1 and §3.3] The method is described as 'training-free,' but Stage 2 trains an SVM classifier per attention head; please qualify this as 'no LVLM fine-tuning' or otherwise clarify the terminology.
  4. [§3.4, Eq. 10] Please specify the dimension of S^{(l,h)} and state explicitly whether the same shift vector is added to every token position; the current notation suggests a last-token vector broadcast over all positions, which should be stated.
  5. [Throughout] Minor typographical issues include 'methodes' in §4.1, 'performence' in §4.2, and 'A verage' in Table 7; these should be corrected.
  6. [Table 3] Please use consistent baseline labels with the other tables ('Greedy' versus 'Regular') and state explicitly that the VH Rate column is a percentage.

Circularity Check

1 steps flagged · score 5.0 of 10

POPE Adversarial results are partly fitted: α and K are tuned on MS-COCO Adversarial POPE and then reported as SOTA on that same benchmark; the core shift-vector derivation is otherwise self-contained.

  1. fitted input called prediction [Section 5.4 (Implications of Hyperparameters), Appendix A.2, Table 1 (Adversarial POPE rows)]
    "CAI method primarily relies on two key hyperparameters: the intensity of intervention α and the number of refined attention heads K. We performed a series of ablation experiments using greedy decoding on the MS-COCO Adversarial POPE dataset."

    The paper's headline POPE results in Table 1, especially the Adversarial setting, are produced with α=1.5 and K=100. Appendix A.2 states that these values are used in the main experiments, while Section 5.4 and Appendix D select them through ablations on the MS-COCO Adversarial POPE dataset itself. Consequently, the reported accuracy and F1 on Adversarial POPE are hyperparameter-optimized numbers on the evaluation set, not out-of-sample predictions. Because α and K directly scale and gate the intervention added in Eq. 10, the best-case POPE claim is statistically forced by the same benchmark used to validate it. The shift-vector computation in Eq. 9 from 1,000 pretraining samples is not itself circular, but this particular benchmark result is partly fitted.

full rationale

The core CAI construction is not circular: the attention shift vector S in Eq. 9 is an average of caption-query minus non-caption-query attention outputs computed on 1,000 LLaVA pretraining samples, and the caption-sensitive heads are selected by binary classifiers trained on the same pretraining data. This yields an image-agnostic steering direction that is applied to held-out benchmark queries, so the central mechanism is externally grounded rather than defined by the evaluation benchmarks. No load-bearing self-citation or imported uniqueness theorem appears; citations to prior intervention work such as Li et al. 2024 are methodological, not premises that assume the conclusion. The one concrete circularity is the tuning of α and K on the MS-COCO Adversarial POPE dataset and then reporting the best configuration on that same dataset as a SOTA result in Table 1. This does not invalidate the method across all benchmarks, because CHAIR, MME, MMHal-Bench, and the Random and Popular POPE settings use the same fixed α/K and are less directly fitted, but the Adversarial POPE headline is a fitted value rather than an independent prediction. A separate correctness risk, not a circularity, is that Eq. 9 computes S from unmasked attention outputs even though the probing stage in Eqs. 5-6 masks text attention to isolate visual signal; as a result, the claimed vision-centric mechanism is not fully established by the paper's equations. That issue affects interpretation, not circularity, and does not change the fitted-hyperparameter finding.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the transferability of caption-query attention patterns to non-caption queries, plus the choice of alpha and K. No new physical entities are introduced. The probing uses 1000 pretrain samples, and hyperparameters are tuned on a test benchmark.

free parameters (2)
  • alpha (intervention intensity) = 1.5
    Tuned on MS-COCO Adversarial POPE (Appendix D, Table 5), and this value is used for all main results.
  • K (number of refined heads) = 100
    Tuned on MS-COCO Adversarial POPE (Appendix D, Table 6), and this value is used for all main results.
assumptions (4)
  • domain assumption Masking text-related attention scores to -inf during probing isolates visual information in attention outputs.
    Section 3.3, Eqs. 5-7. Assumes that removing text attention does not distort the visual signal used for head classification.
  • domain assumption Caption queries activate the model's inherent visual perception capability more strongly than non-caption queries, and this activation pattern can be transferred.
    Section 2 and 3. Central motivation; if false, the whole intervention direction is invalid.
  • domain assumption Classifiers trained on attention outputs from a small pretrain set generalize to identify caption-sensitive heads across models and datasets.
    Section 3.3. Uses 1000 samples from LLaVA pretrain to probe heads for Qwen-VL-Chat and LLaVA-NeXT.
  • domain assumption Adding the average output shift to top-K heads does not degrade other capabilities.
    Section 3.4, Eq. 10. Assumes the shift direction is safe and does not disrupt non-visual functions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models." pith.science (2026). https://pith.science/paper/YTJQQ2OJ

@misc{pith2026250623590,
  author       = {Pith},
  title        = {Pith review of: CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YTJQQ2OJ}},
  note         = {Machine review of arXiv:2506.23590}
}
read the original abstract

Although Large Vision-Language Models (LVLMs) have demonstrated powerful capabilities in interpreting visual information, they frequently produce content that deviates from visual information, leading to object hallucination. To tackle this, recent works mostly depend on expensive manual annotations and training cost, or significantly increase inference time. In this work, we observe that LVLMs' attention to visual information is significantly stronger when answering caption queries compared to non-caption queries. Inspired by this phenomenon, we propose Caption-sensitive Attention Intervention (CAI), a training-free, plug-and-play hallucination mitigation method that leverages the attention activation pattern in response to caption queries to enhance LVLMs' visual perception capability. Extensive experimental results across four benchmarks covering both discriminative and generative tasks, demonstrate that CAI achieves state-of-the-art (SOTA) hallucination mitigating performance only with minimal additional inference cost.

Figures

Figures reproduced from arXiv: 2506.23590 by the authors.

Figure 1
Figure 1. The visualization of attention weights at im [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A systematic quantitative analysis from head [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. An overview of the CAI method. Each square in the matrix represents the attention head output. Squares [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Main result of LLaVA-1.5-7b on MS-COCO CHAIR task. Smaller values of CHAIRi and CHAIRs indicate that the method demonstrates stronger halluci￾nation mitigation capabilities at instance and sentence levels. M ax_new_tokens is set to be 64. methods, CAI achieved superior…
Figure 6
Figure 6. Figure 6: The accuracies of classifiers. α: the intensity of intervention K: the number of refined heads P e r c e n t a g e(%) P e r c e n t a g e(%) [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Ablation study of α and K. model. By refining the output of these attention heads, CAI significantly enhances LVLMs’ visual perception capability and mitigate hallucination. 5.4 Implications of Hyperparameters CAI method primarily relies on two key hyper￾parameters: th…
Figure 8
Figure 8. Figure 8: LLaVA-1.5-7b demonstrated hallucination when addressing caption query of MS-COCO CHAIR dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Main results on the MME full set. CAI leads to the best enhancement in LVLMs’ perception capacities [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Caption query case of LLaVA-1.5-7b on MMHal-Bench. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Another caption query case of LLaVA-1.5-7b on MMHal-Bench. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Non-caption query case of LLaVA-1.5-7b on MMHal-Bench. [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Non-caption query case of LLaVA-1.5-7b on MMHal-Bench. [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Non-caption query case of LLaVA-1.5-7b on MMHal-Bench. [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 9 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Guang Dai, Ping Chen, and Shijian Lu. 2024. Agla: Mitigating object hallucinations in large vision-language models with assembly of global and local attention. arXiv preprint arXiv:2406.12718

    arXiv 2024

  4. [4]

    Kazi Hasan Ibn Arif, Sajib Acharjee Dip, Khizar Hussain, Lang Zhang, and Chris Thomas. 2025. Fixing imbalanced attention to mitigate in-context hallucination of large vision-language model. arXiv preprint arXiv:2501.12206

    arXiv 2025

  5. [5]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

    arXiv 2023

  6. [7]

    Jing Bi, Junjia Guo, Yunlong Tang, Lianggong Bruce Wen, Zhang Liu, and Chenliang Xu. 2024 b . Unveiling visual perception in language models: An attention head analysis approach. arXiv preprint arXiv:2412.18108

    arXiv 2024

  7. [8]

    Beitao Chen, Xinyu Lyu, Lianli Gao, Jingkuan Song, and Heng Tao Shen. 2024 a . Alleviating hallucinations in large vision-language models through hallucination-induced optimization. arXiv preprint arXiv:2405.15356

    arXiv 2024

  8. [9]

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. 2024 b . Halc: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425

    arXiv 2024

Show all 48 references
  1. [10]

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883

    2023 arXiv

  2. [11]

    Corinna Cortes. 1995. Support-vector networks. Machine Learning

    1995

  3. [12]

    Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. 2024. Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer V...

    2024

  4. [13]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. 2023. https://api.semanticscholar.org/CorpusID:259243928 Mme: A comprehensive evaluation benchmark for multimodal large la...

    2023 arXiv

  5. [14]

    Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Utkarsh Tyagi, Oriol Nieto, Zeyu Jin, and Dinesh Manocha. 2024. https://arxiv.org/abs/2405.15683 Vdgd: Mitigating lvlm hallucinations in cognitive prompts by bridging the visual perception gap . Preprint, arXiv:2405.15683

    2024 arXiv

  6. [15]

    Xuan Gong, Tianshi Ming, Xinpeng Wang, and Zhihua Wei. 2024. Damro: Dive into the attention mechanism of lvlm to reduce object hallucination. arXiv preprint arXiv:2410.04514

    2024

  7. [16]

    Anisha Gunjal, Jihan Yin, and Erhan Bas. 2024. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18135--18143

    2024

  8. [17]

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. 2024. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408.16500

    2024 arXiv

  9. [18]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232

    2023 arXiv

  10. [19]

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Con...

    2024

  11. [20]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  12. [21]

    Zhehan Kan, Ce Zhang, Zihan Liao, Yapeng Tian, Wenming Yang, Junyuan Xiao, Xu Li, Dongmei Jiang, Yaowei Wang, and Qingmin Liao. 2024. Catch: Complementary adaptive token-level contrastive decoding to mitigate hallucinations in lvlms. arXiv preprint arXiv:2411.12713

    2024 arXiv

  13. [22]

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1--10

    2018

  14. [23]

    Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Minjoon Seo. 2023. Volcano: mitigating multimodal hallucination through self-feedback guided revision. arXiv preprint arXiv:2311.07362

    2023 arXiv

  15. [24]

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  16. [25]

    Jiaming Li, Jiacheng Zhang, Zequn Jie, Lin Ma, and Guanbin Li. 2025. Mitigating hallucination for large vision language model by inter-modality correlation calibration decoding. arXiv preprint arXiv:2501.01926

    2025 arXiv

  17. [26]

    Kenneth Li, Oam Patel, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. 2024. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36

    2024

  18. [27]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355

    2023 arXiv

  19. [28]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, ...

    2014

  20. [29]

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations

    2023

  21. [30]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 a . Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296--26306

    2024

  22. [31]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024 b . https://llava-vl.github.io/blog/2024-01-30-llava-next/ Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  23. [32]

    Sheng Liu, Haotian Ye, and James Zou. 2024 c . Reducing hallucinations in vision-language models via latent space steering. arXiv preprint arXiv:2410.15778

    2024 arXiv

  24. [33]

    Shi Liu, Kecheng Zheng, and Wei Chen. 2024 d . Paying more attention to image: A training-free method for alleviating hallucination in lvlms. arXiv preprint arXiv:2407.21771

    2024 arXiv

  25. [34]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2024 e . Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216--233. Springer

    2024

  26. [35]

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arX...

    2024 arXiv

  27. [36]

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156

    2018 arXiv

  28. [37]

    Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, and Aman Chadha. 2024. A comprehensive survey of hallucination in large language, image, video and audio foundation models. Findings of the Association for Computational Linguistics: EMNLP 2024, pages 117...

    2024

  29. [38]

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2023. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525

    2023 arXiv

  30. [39]

    Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. 2024. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. arXiv preprint arXiv:2403.18715

    2024 arXiv

  31. [40]

    Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. 2024. Mitigating object hallucination via concentric causal attention. arXiv preprint arXiv:2410.15926

    2024 arXiv

  32. [41]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. 2024. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67(12):220105

    2024

  33. [42]

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. 2023. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704

    2023 arXiv

  34. [43]

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. 2024 a . Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Confere...

    2024

  35. [44]

    Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. 2024 b . Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17220

    2024

  36. [45]

    Linxi Zhao, Yihe Deng, Weitong Zhang, and Quanquan Gu. 2024. Mitigating object hallucination in large vision-language models via classifier-free guidance. arXiv preprint arXiv:2402.08680

    2024 arXiv

  37. [46]

    Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multi-dimensional evaluator for text generation. arXiv preprint arXiv:2210.07197

    2022 arXiv

  38. [47]

    Weihong Zhong, Xiaocheng Feng, Liang Zhao, Qiming Li, Lei Huang, Yuxuan Gu, Weitao Ma, Yuan Xu, and Bing Qin. 2024. https://arxiv.org/abs/2407.00569 Investigating and mitigating the multimodal hallucination snowballing in large vision-language models . Preprint, arXiv:2407.00569

    2024 arXiv

  39. [48]

    Guanyu Zhou, Yibo Yan, Xin Zou, Kun Wang, Aiwei Liu, and Xuming Hu. 2024. Mitigating modality prior-induced hallucinations in multimodal large language models via deciphering attention causality. arXiv preprint arXiv:2410.04780

    2024 arXiv

  40. [49]

    Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. 2024. Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding. arXiv preprint arXiv:2402.18476

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.