Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single transformer layer, with attention heads selected by a text-to-visual entropy ratio, can suppress hallucinated tokens in large vision-language models during one forward pass, avoiding the second query that contrastive decoding…

desk verdict Efficient single-pass contrastive decoding that mostly works; the TVER mechanism story needs rethinking before it is publishable. read the letter →

arxiv 2507.00898 v1 pith:TRZR7YWZ submitted 2025-07-01 cs.CV cs.CL

Zifu Wan , Ce Zhang , Silong Yong , Martin Q. Ma , Simon Stepputtis , Louis-Philippe Morency , Deva Ramanan , Katia Sycara
show 1 more author
Yaqi Xie
This is my paper · ORCID
classification cs.CVcs.CL
keywords hallucinationmitigationvision-languagemodelscontrastivedecodingattentionheadselectiontext-to-visualentropyratiotraining-freesingle-queryinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that hallucination in large vision-language models can be reduced without the second query that contrastive decoding methods need. Its method, ONLY, is training-free: during one forward pass it chooses a single transformer layer, computes a text-to-visual entropy ratio for each attention head, and zeroes the heads with ratio below the layer average. The resulting text-enhanced logits are then combined with the original logits, either additively or contrastively, depending on a per-token distance between the two distributions. Across three vision-language models and several benchmarks, ONLY reports lower hallucination and higher accuracy than the two-query baselines VCD and M3ID while adding only about 7 percent to inference time. If correct, this makes hallucination mitigation compatible with real-time deployment.

What carries the argument

The central object is the Text-to-Visual Entropy Ratio (TVER) per attention head. For head $i$ in layer $\ell$ it is $\mathrm{Entropy}(a^{T}_{\ell,i})/\mathrm{Entropy}(a^{V}_{\ell,i})$, where $a^{T}$ and $a^{V}$ are that head's attention probabilities restricted to textual and visual token positions. The paper's premise is that a head whose textual attention entropy is high relative to its visual entropy carries language-prior information rather than grounded visual evidence; zeroing heads below the layer average therefore produces logits that suppress predictable hallucinated language. The one-layer textual-enhanced attention, the residual re-entry into the final layer, and the adaptive collaborative/contrastive decoding rule are the mechanism that converts this proxy into a single-query intervention.

What would settle it

On a held-out benchmark with object distributions unlike COCO, compute the TVER-based head masks and check whether hallucination rates still drop; alternatively, replace the TVER ranking with a random head mask of the same size and measure POPE and CHAIR. If a random mask performs equally well, the entropy-ratio mechanism is not the cause of the gains.

Watch

Extended reading notes

Core claim

ONLY establishes that a single-layer intervention inside an already-running transformer is enough to shift generation away from language-prior hallucinations. For a chosen layer, the method computes the entropy of each attention head's attention weights restricted to text tokens and to visual tokens, takes the ratio $\mathrm{TVER} = H(a^{T})/H(a^{V})$, and masks to zero the heads with ratio below the layer average, producing a textual-enhanced multi-head attention output. Residual connections feed that output through the final layer to obtain text-enhanced logits, and the final distribution is an adaptive per-timestep combination: collaborative addition when the Manhattan distance between the original and enhanced distributions is below a threshold, and contrastive subtraction when the distance is large. The experiments report consistent gains over the two-query baselines VCD and M3ID, such as 3.14 percent higher accuracy on POPE and lower CHAIRS and CHAIRI on CHAIR, with an MME score of 635.55 against 598.33 for the next best baseline, at 1.07x latency.

Load-bearing premise

The load-bearing premise is that an attention head's text-to-visual entropy ratio is a trustworthy per-token measure of language bias, so that zeroing low-ratio heads removes hallucination pressure without damaging genuinely grounded visual content.

Editorial extensions

If this is right

  • Hallucination mitigation no longer requires a distorted second image or a text-only pass: a single forward pass with one layer's attention edited is sufficient, so the 2x or larger overhead of VCD and M3ID can be avoided in latency-sensitive systems.
  • Because the method is training-free, it can be applied to existing autoregressive vision-language models without fine-tuning, and the paper demonstrates this on LLaVA-1.5, InstructBLIP, and Qwen-VL, including a 13B variant.
  • The cost of intervention scales with one layer's attention computation rather than a full extra model run, matching the reported 1.07x inference-time increase and negligible memory overhead.
  • Improved results on MMBench, MM-Vet, and MMVP indicate that hallucination suppression and general multimodal understanding can improve together rather than trading off.
  • The head-selection mask depends on the current token's attention, so the intervention adapts dynamically to each decoding step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If TVER is a stable marker of language bias across layers and models, the same logic could choose the intervened layer or the number of masked heads dynamically per token, possibly reducing the cost below a full layer.
  • The correlation between TVER and hallucination is demonstrated on CHAIR and POPE, the same benchmarks used for the main evaluation; checking the correlation on distribution-shifted or rare-object images would test whether the proxy generalizes beyond COCO-style scenes.
  • The decoding threshold gamma is set differently for LLaVA-1.5 (0.2) and InstructBLIP/Qwen-VL (0.4); a rule that derives gamma from the distance statistic itself could remove a model-specific hyperparameter.
  • The paper lists video hallucination mitigation as future work; the same one-layer textual-enhancement principle is a natural candidate for video vision-language models, where the two-query cost of contrastive decoding is even more prohibitive.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes ONLY, a training-free decoding algorithm for large vision-language models that mitigates hallucinations with a single forward pass plus a one-layer intervention. At a chosen layer, attention heads with a below-average text-to-visual entropy ratio (TVER) are zeroed, and the resulting textual-enhanced output is carried through residual connections and the final MLP. The resulting logits are adaptively combined with the original logits, using either collaborative or contrastive decoding depending on a Manhattan-distance threshold. Experiments on LLaVA-1.5, InstructBLIP, and Qwen-VL across POPE, CHAIR, MME, MMBench, MM-Vet, MMVP, and LLaVA-Bench report consistent gains over VCD and M3ID at a reported 1.07x inference-time overhead.

Significance. If the results hold, ONLY is a practically valuable contribution to hallucination mitigation: it avoids the doubled inference cost of contrastive decoding methods while matching or exceeding their accuracy on several benchmarks. The paper's strengths include a wide evaluation across three backbones and multiple benchmarks, efficiency measurements on the same hardware, ablations of hyperparameters and layer choice, and a public code release. The main weaknesses are that several hyperparameters and the TVER heuristic are tuned and validated on the same benchmarks used for the headline evaluation, and that the most informative control for the head-selection mechanism (random head masking) is missing. These issues are addressable, but they currently limit confidence in the central claim.

major comments (5)
  1. [§3.3, Algorithm 1 vs Eq. (17)] Equation (17) defines the textual-enhanced probability as softmax(phi(Ĥ^L_t)), but Algorithm 1 (step 4, line 20) computes the TE logits as Linear(Ĥ^L_t + H^L_t), which adds the original final-layer hidden state. These two definitions give different distributions, and the paper does not state which one is implemented. Please correct the inconsistency and confirm that the implementation matches the stated equation; this is essential for reproducibility.
  2. [§4.1, §4.4, Supp. D] The decoding hyperparameters (α1, α2, β, γ) and the intervention layer index are selected by ablations conducted on POPE and CHAIR (Tables D5-D8 and Figure 7), and the TVER heuristic is motivated by correlations computed on the same two benchmarks (Figure 4 and §4.2). Thus the POPE and CHAIR improvements are at least partially the result of in-sample tuning. The paper should tune on a separate validation split or report the results with this limitation explicitly acknowledged.
  3. [Eqs. (8)-(11), Supp. A.1] The claim that TVER approximates the conditional-entropy ratio H(T|V)/H(V|T) is not supported: the quantities in Eq. (9) are entropies of attention weights renormalized separately over textual and visual subsets, not conditional entropies over the full attention distribution, so the monotonicity argument in Supp. A.1 does not apply. Moreover, because the textual and visual subsets differ greatly in size (for LLaVA-1.5, roughly 576 visual tokens versus a much smaller textual set; for InstructBLIP, 32 visual tokens), the ratio is strongly influenced by subset cardinality rather than by any information-theoretic property of the heads. Either provide a rigorous justification or reframe TVER as a purely empirical heuristic.
  4. [§4.4, Table 6] The ablation against other attention-modification strategies does not include the decisive control of masking the same number of heads chosen uniformly at random at the intervened layer. Without such a control, the observed gains could be attributed to the one-layer residual perturbation alone rather than to the TVER selection criterion. Please add a random-mask baseline matched in head count and layer.
  5. [Abstract vs. Tables 1-2] The abstract's claim that ONLY outperforms state-of-the-art by 3.14% on POPE and 1.6% on CHAIR cannot be recovered from the tables: the POPE tables show the largest single-row advantage over M3ID is 3.14% in one InstructBLIP/A-OKVQA cell, while many rows show smaller gains or a small loss (e.g., Table 1, Qwen-VL GQA Adversarial Acc 81.33 vs 81.90 for M3ID), and the CHAIR table shows no obvious 1.6% average advantage. Please state precisely which baseline and which aggregation the abstract numbers refer to, or replace them with a reproducible summary.
minor comments (6)
  1. [Supp. B.4] The heading 'Devision of Textual and Visual Tokens' should be 'Division'.
  2. [Table D6] The caption says 'varying values of α1' but the rows list α2 values; please correct.
  3. [Figure 3] The y-axis labels contain an extra space ('T extual Entropy', 'Visual Entropy'); please fix the labels.
  4. [Section 4.2, MME] The text reports an MME score of 634.67, but Table 3 and Table 5 report 635.55 and 635.6; please reconcile these numbers.
  5. [Eq. (5)] The notation fθ(yt|y<t) drops the conditioning on v and x used elsewhere; please make the conditioning consistent across equations.
  6. [Figure 5] The x-axis labels are not readable (they appear as unicode fragment strings); please render them as model names.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TVER head-selection and adaptive decoding are defined from attention entropies and logits, not from benchmark labels; out-of-sample gains on MME/MMBench/MM-Vet/MMVP support independence.

full rationale

The paper's derivation chain is: (i) define TVER as a ratio of entropies of attention subsets over text and visual tokens (Eqs. 7-11); (ii) select heads with above-average TVER (Eq. 12) to form a textual-enhanced MHA (Eq. 13); (iii) inject this into the last layer via residual connections (Eqs. 14-16) and combine with original logits through an adaptive contrastive/collaborative combination (Eq. 20). None of these steps defines the output in terms of the hallucination benchmarks, and none of the reported benchmark numbers is used as an input to the equations. The information-theoretic motivation in Supp. A.1 is approximate rather than a formal derivation, but this is a correctness/rigor concern, not circularity. The main evaluation on POPE and CHAIR uses hyperparameters (α1, α2, γ, β, layer index) selected via ablations on those same benchmarks (Fig. 7, Tables D5-D8), which limits the out-of-sample value of those specific numbers; however, this is benchmark tuning/overfitting, not a reduction of a prediction to a fitted parameter by construction, and the paper also reports consistent gains on MME, MMBench, MM-Vet, MMVP, and LLaVA-Bench, which were not used for tuning. No load-bearing self-citation or imported uniqueness theorem appears; self-citations [34, 41, 42, 43] are background or future-work references. Therefore no significant circularity is present.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The method introduces no new entities. It rests on four fitted hyperparameters, an unstated intervention layer choice, and a domain heuristic linking attention entropy to language bias; the conditional-entropy explanation in the supplement is an analogy, not a proof.

free parameters (5)
  • alpha_1 = 3
    Collaborative decoding weight in Eq. 20; chosen after ablation on POPE and CHAIR (Table D5).
  • alpha_2 = 1
    Contrastive decoding weight in Eq. 20; chosen after ablation on POPE and CHAIR (Table D6).
  • gamma = 0.2 for LLaVA-1.5; 0.4 for InstructBLIP and Qwen-VL
    Threshold controlling switching between collaborative and contrastive decoding; set per backbone via ablation (Table D8).
  • beta = 0.1
    Adaptive plausibility constraint truncation, taken from VCD and ablated on POPE and CHAIR (Table D7).
  • Intervention layer index = 0 (best in Fig. 7); not listed in Section 4.1
    The method requires choosing which transformer layer to intervene in; the ablation shows layer 0 best, but the main experiments do not state the layer used.
assumptions (3)
  • ad hoc to paper TVER is a reliable proxy for language bias, so masking heads with below-average TVER reduces hallucination.
    Introduced in Section 3.2, Eq. 12, supported only by the empirical correlation in Figure 4 computed on CHAIR and POPE; the supplement's conditional-entropy derivation (Supp A.1) is an analogy, not a formal proof.
  • ad hoc to paper Re-normalizing attention weights separately for textual and visual token subsets yields meaningful entropy values.
    Eqs. 9 and 10 define entropies after separate softmax operations; this normalization choice is not externally justified and changes the meaning of TVER.
  • ad hoc to paper A single extra attention computation at one layer plus the final MLP approximates the effect of a second contrastive query.
    This equivalence underlies the efficiency claim and is validated only empirically through benchmark results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models." pith.science (2026). https://pith.science/paper/TRZR7YWZ

@misc{pith2026250700898,
  author       = {Pith},
  title        = {Pith review of: ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TRZR7YWZ}},
  note         = {Machine review of arXiv:2507.00898}
}
read the original abstract

Recent Large Vision-Language Models (LVLMs) have introduced a new paradigm for understanding and reasoning about image input through textual responses. Although they have achieved remarkable performance across a range of multi-modal tasks, they face the persistent challenge of hallucination, which introduces practical weaknesses and raises concerns about their reliable deployment in real-world applications. Existing work has explored contrastive decoding approaches to mitigate this issue, where the output of the original LVLM is compared and contrasted with that of a perturbed version. However, these methods require two or more queries that slow down LVLM response generation, making them less suitable for real-time applications. To overcome this limitation, we propose ONLY, a training-free decoding approach that requires only a single query and a one-layer intervention during decoding, enabling efficient real-time deployment. Specifically, we enhance textual outputs by selectively amplifying crucial textual information using a text-to-visual entropy ratio for each token. Extensive experimental results demonstrate that our proposed ONLY consistently outperforms state-of-the-art methods across various benchmarks while requiring minimal implementation effort and computational cost. Code is available at https://github.com/zifuwan/ONLY.

Figures

Figures reproduced from arXiv: 2507.00898 by the authors.

Figure 1
Figure 1. Comparisons of accuracy and inference speed of mul￾tiple hallucination mitigation approaches. The size of bubbles stands for the GPU memory consumption. Our method effectively mitigates hallucination with only 0.07× extra time. of LVLMs in real-world applications where precise visual interpretation is essential [2, 3, 24]. Therefore, addressing hallucinations in LVLMs is crucial to ensuring their safe and effective … view at source ↗
Figure 2
Figure 2. Overview of our proposed ONLY. Our method retains the core decoding process of LVLMs but incorporates a textual-enhanced multi-head attention layer with a residual connection to the last layer’s output. This adjustment aims to produce an output with a greater focus on textual information. The resulting textual-enhanced logits are then adaptively decoded alongside the original output, employing either contrastive or … view at source ↗
Figure 3
Figure 3. Impact of applying diffusion noise on textual and visual attention entropy. We perform an analysis on all COCO samples from the POPE benchmark and observe that as distortion increases, textual entropy rises whereas visual entropy decreases. overhead. Specifically, inspired by information theory [30], we introduce an attention-head selection strategy guided by the text-to-visual entropy ratio. As illustrated in [PIT… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Text-to-visual entropy ratio is correlated with hal￾lucinations. (Left) Density plot of token-wise average textual-to￾visual entropy ratio and bar plot of average CHAIRI in each bin on the CHAIR benchmark; (Right) Density plots of token-level Manhattan distance between…
Figure 5
Figure 5. Figure 5: Results on MMVP [32]. We apply our approach to LLaVA-1.5 [22] and compare its performance against other hallucination mitigation methods. Method LLaVA-1.5 InstructBLIP Acc. ↑ Det. ↑ Acc. ↑ Det. ↑ Regular 6.07 6.20 5.26 5.53 Ours 7.00 7.13 6.60 6.73 VCD 4.60 5.13 4.87 5…
Figure 6
Figure 6. Figure 6: Case study on the LLaVA-Bench benchmark. We compare the responses generated by regular decoding and our method using LLaVA-1.5. GPT-4V-aided evaluation results are also provided alongside the responses. Hallucinated and accurate content is highlighted in red and blue. …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Disentangling Semantic Attention from Structural Bias in the Attention Manifold

    cs.CV 2026-07 conditional novelty 5.0 of 10

    SPAR removes a query-averaged structural bias from text-to-image attention and redistributes the reclaimed probability mass, reducing reported object and induced hallucinations in LLaVA models.

Reference graph

Works this paper leans on

44 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  2. [2]

    Hallucination of multimodal large language models: A survey

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930, 2024

    arXiv 2024

  3. [3]

    Detecting and evaluating medical hallu- cinations in large vision language models

    Jiawei Chen, Dingkang Yang, Tong Wu, Yue Jiang, Xiaolu Hou, Mingcheng Li, Shunli Wang, Dongling Xiao, Ke Li, and Lihua Zhang. Detecting and evaluating medical hallu- cinations in large vision language models. arXiv preprint arXiv:2406.10185, 2024

    arXiv 2024

  4. [4]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19–35. Springer, 2024

    work page 2024

  5. [5]

    Mitigating hallucination in visual language models with visual supervi- sion

    Zhiyang Chen, Yousong Zhu, Yufei Zhan, Zhaowen Li, Chaoyang Zhao, Jinqiao Wang, and Ming Tang. Mitigating hallucination in visual language models with visual supervi- sion. arXiv preprint arXiv:2311.16479, 2023

    arXiv 2023

  6. [6]

    HALC: Object hallucination reduc- tion via adaptive focal-contrast decoding

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. HALC: Object hallucination reduc- tion via adaptive focal-contrast decoding. In International Conference on Machine Learning, pages 7824–7846. PMLR, 2024

    work page 2024

  7. [7]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023

    work page 2023

  8. [8]

    Glass, and Pengcheng He

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by con- trasting layers improves factuality in large language models. In International Conference on Learning Representations , 2024

    work page 2024

Show all 44 references
  1. [9]

    Instructblip: towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: towards general-purpose vision-language models with instruction tuning. Advances in Neural Infor- mation Processing Systems, 36:49250–49267, 2023

    2023

  2. [10]

    Multi-modal hal- lucination control by visual information grounding

    Alessandro Favero, Luca Zancato, Matthew Trager, Sid- dharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hal- lucination control by visual information grounding. In Pro- ceedings of the IEEE/CVF Conference on Computer V...

    2024

  3. [11]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  4. [12]

    Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Confer...

    2024

  5. [13]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 6700–6709, 2019

    2019

  6. [14]

    Hallucination augmented contrastive learn- ing for multimodal large language model

    Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learn- ing for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2024

  7. [15]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9579–9589, 2024

    2024

  8. [16]

    Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogn...

    2024

  9. [17]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational Conference on Machine Learning, pages 19730– 19742. PMLR, 2023

    2023

  10. [18]

    Contrastive decoding: Open-ended text gener- ation as optimization

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Ja- son Eisner, Tatsunori B Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text gener- ation as optimization. In Proceedings of the Annual Meet- ing of the Association for Computational Ling...

    2023

  11. [19]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the Confer- ence on Empirical Methods in Natural Language Processing, pages 292–305, 2023

    2023

  12. [20]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740–755. Springer, 2014

    2014

  13. [21]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36:34892–34916, 2023

    2023

  14. [22]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  15. [23]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    2024

  16. [24]

    A survey on hallucination in large vision-language models

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiu- tian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253, 2024

    2024 arXiv

  17. [25]

    Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer, 2024

    2024

  18. [26]

    Alleviating hallucinations in large vision- language models through hallucination-induced optimiza- tion

    Xinyu Lyu, Beitao Chen, Lianli Gao, Hengtao Shen, and Jingkuan Song. Alleviating hallucinations in large vision- language models through hallucination-induced optimiza- tion. Advances in Neural Information Processing Systems , 37:122811–122832, 2024

    2024

  19. [27]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Con...

    2021

  20. [28]

    Object hallucination in image captioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 4035–4045, 2018

    2018

  21. [29]

    A-okvqa: A benchmark for visual question answering using world knowl- edge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowl- edge. In European Conference on Computer Vision , pages 146–162. Springer, 2022

    2022

  22. [30]

    A mathematical theory of communi- cation

    Claude E Shannon. A mathematical theory of communi- cation. The Bell system technical journal , 27(3):379–423, 1948

    1948

  23. [31]

    Aligning large multi- modal models with factually augmented rlhf

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multi- modal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525, 2023

    2023 arXiv

  24. [32]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024

    2024

  25. [33]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  26. [34]

    Instruct- part: Task-oriented part segmentation with instruction rea- soning

    Zifu Wan, Yaqi Xie, Ce Zhang, Zhiqiu Lin, Zihan Wang, Si- mon Stepputtis, Deva Ramanan, and Katia Sycara. Instruct- part: Task-oriented part segmentation with instruction rea- soning. arXiv preprint arXiv:2505.18291, 2025

    2025 arXiv

  27. [35]

    Mitigating hallucinations in large vision-language models with instruction contrastive decoding

    Xintong Wang, Jingheng Pan, Liang Ding, and Chris Bie- mann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. In Findings of the Association for Computational Linguistics ACL 2024, pages 15840–15853, 2024

    2024

  28. [36]

    Det- toolchain: A new prompting paradigm to unleash detection ability of mllm

    Yixuan Wu, Yizhou Wang, Shixiang Tang, Wenhao Wu, Tong He, Wanli Ouyang, Philip Torr, and Jian Wu. Det- toolchain: A new prompting paradigm to unleash detection ability of mllm. In European Conference on Computer Vi- sion, pages 164–182. Springer, 2024

    2024

  29. [37]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , ...

    2024

  30. [38]

    Woodpecker: Hallucination correction for multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:2310.16045 , 2023

    arXiv

  31. [39]

    MM-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. MM-vet: Evaluating large multimodal models for integrated capabilities. In International Conference on Machine Learn- ing, 2024

    2024

  32. [40]

    Contextual object detection with multi- modal large language models

    Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy. Contextual object detection with multi- modal large language models. International Journal of Com- puter Vision, pages 1–19, 2024

    2024

  33. [41]

    Incorpo- rating generative feedback for mitigating hallucinations in large vision-language models

    Ce Zhang, Zifu Wan, Zhehan Kan, Martin Q Ma, Si- mon Stepputtis, Deva Ramanan, Russ Salakhutdinov, Louis- Philippe Morency, Katia P Sycara, and Yaqi Xie. Incorpo- rating generative feedback for mitigating hallucinations in large vision-language models. In Workshop on Responsib...

    2024

  34. [42]

    Vscan: Rethinking visual token reduction for efficient large vision-language models

    Ce Zhang, Kaixin Ma, Tianqing Fang, Wenhao Yu, Hong- ming Zhang, Zhisong Zhang, Yaqi Xie, Katia Sycara, Haitao Mi, and Dong Yu. Vscan: Rethinking visual token reduction for efficient large vision-language models. arXiv preprint arXiv:2505.22654, 2025

    2025

  35. [43]

    Ma, Si- mon Stepputtis, Deva Ramanan, Russ Salakhutdinov, Louis- Philippe Morency, Katia P

    Ce Zhang, Zifu Wan, Zhehan Kan, Martin Q. Ma, Si- mon Stepputtis, Deva Ramanan, Russ Salakhutdinov, Louis- Philippe Morency, Katia P. Sycara, and Yaqi Xie. Self- correcting decoding with generative feedback for mitigating hallucinations in large vision-language models. In Inte...

    2025

  36. [44]

    Is there a {object} in the image?

    Jinrui Zhang, Teng Wang, Haigang Zhang, Ping Lu, and Feng Zheng. Reflective instruction tuning: Mitigating hal- lucinations in large vision-language models. In European Conference on Computer Vision, 2024. ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in L...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.