Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

PAINT: Paying Attention to INformed Tokens to Mitigate Hallucination in Large Vision-Language Model

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Selective boosting of local and summary visual tokens cuts LVLM hallucination by up to 62.3%.

desk verdict In-sample tuning of alpha/beta/N on the same 500-image test set makes the headline hallucination reductions unvalidated, though the local-vs-summary token idea is worth a proper test. read the letter →

arxiv 2501.12206 v3 pith:JFJXXYF4 submitted 2025-01-21 cs.CV cs.CL

Kazi Hasan Ibn Arif , Sajib Acharjee Dip , Khizar Hussain , Lang Zhang , Chris Thomas This is my paper · ORCID
classification cs.CVcs.CL
keywords hallucinationmitigationlargevision-languagemodelsself-attentioninterventionvisualtokensCLS-to-patchattentiontraining-freeLLaVA-1.5imagecaptioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PAINT claims that hallucination in large vision-language models tracks the weakening of attention to visual tokens as generation moves through deeper LLM layers, and that the right fix is selective rather than uniform reinforcement. The paper identifies two roles among ViT-produced visual tokens: local tokens, selected from early CLS-to-patch attention, carry grounded object details; summary tokens, selected from late layers, carry global scene semantics. Boosting these two groups with different attention margins in the LLM's self-attention lowers sentence-level hallucination on the MSCOCO validation set from 46.2% to 17.6% and instance-level hallucination from 13.8% to 4.0%, with a modest F1 drop from 75.9% to 71.8%. If the claim holds, PAINT is a plug-and-play, training-free intervention that improves faithfulness for existing LVLM checkpoints.

What carries the argument

The load-bearing mechanism is the CLS-to-patch attention map inside the Vision Transformer, read at two depths. Eq. (1) averages attention over heads in layer 1 and takes the top N% positions as local tokens; Eq. (2) does the same in layer 24 and takes the top N% as summary tokens. Eq. (3) then amplifies exactly those positions in the LLM's self-attention output, with separate margins α and β, at the intervention layer shown as Layer 2 in Figure 3. This mechanism converts an observed attention imbalance into a targeted re-weighting that requires no gradient updates or additional training.

What would settle it

Run PAINT on a second LVLM family, such as InstructBLIP or LLaMA-Adapter-v2, using the same MSCOCO subset and the same Layer-2 intervention; if CHAIRS and CHAIRI improvements vanish or require a different layer, the ViT-to-LLM token mapping is architecture-specific. A second check is to compare the selected local token patches against ground-truth object masks; low overlap would show the selector is not actually grounded.

Watch

Extended reading notes

Core claim

The paper's central discovery is that not all visual tokens deserve the same attention boost during decoding. Using the ViT's CLS-to-patch attention maps, PAINT selects the top N% of tokens in the first ViT layer as local tokens and the top N% in the last layer as summary tokens, then modifies LLM self-attention with Eq. (3): local tokens receive an added margin α·|Ā| and summary tokens receive β·|Ā|. The paper reports the best balance at α=0.7, β=0.4, N=25%, where CHAIRS drops from 46.2% to 17.6% and CHAIRI from 13.8% to 4.0%. This is the claim that selective boosting of informed tokens outperforms uniform visual-token boosting and does so while keeping most task accuracy.

Load-bearing premise

The method assumes that CLS-to-patch attention in the ViT reliably identifies which visual tokens carry grounded object information, and that boosting exactly those positions in one fixed LLM layer is what removes hallucination.

Editorial extensions

If this is right

  • The intervention is training-free: it changes only attention weights at inference time, so any existing LLaVA-1.5-style checkpoint can adopt it without retuning.
  • Because local and summary tokens need different margins, the paper's results imply that treating all visual tokens uniformly leaves hallucination reduction on the table.
  • The paper reports that adding spatial token selection can push CHAIRS down to 15.4%, showing further headroom beyond the main configuration.
  • The method is designed to transfer to other LVLMs with a ViT-plus-LLM architecture, including InstructBLIP, LLaMA-Adapter-v2, MiniGPT-4, and Shikra.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The two-margin design suggests a general recipe: partition visual tokens by their role in the ViT and give each role its own attention boost, a recipe that could extend to video or high-resolution inputs where local and summary roles are more pronounced.
  • Inference: One testable extension is to set α and β from the measured slope of attention decay across LLM layers, making the method adaptive per image rather than dependent on fixed hyperparameters.
  • Inference: If the mechanism generalizes, the method could combine with logit-level calibration, since attention re-weighting and logit refinement target different failure modes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes PAINT, a training-free method that reduces hallucination in large vision-language models by selectively amplifying self-attention to two groups of visual tokens, called local tokens and summary tokens, which are identified from CLS-to-patch attention maps in a Vision Transformer. The method is evaluated on 500 sampled MSCOCO instances using CHAIR and POPE metrics, reporting large reductions in hallucination (e.g., 61.9% sentence-level and 71.0% instance-level) relative to the PAI baseline. The central claim is that selective boosting with learned margins alpha and beta outperforms uniform attention boosting while preserving accuracy.

Significance. If the reported results were validated on out-of-sample data, PAINT would be a valuable contribution because it is a training-free, plug-and-play intervention with a conceptually interesting token taxonomy (local vs. summary). The paper includes a public code repository, which aids reproducibility. However, the current evaluation does not establish the central claim: the headline metrics are computed on the same 500-example sample used to select alpha, beta, and N, making them in-sample fits. The method also exhibits extreme sensitivity to hyperparameters, and the manuscript omits crucial implementation details about which layers and heads are modified.

major comments (4)
  1. [Sections 4.2 and 4.4] The 500-instance sample described in Section 4.2 is used both for selecting the hyperparameters alpha, beta, and N (Tables 2, 3, 4) and for computing the final results in Table 1. Because the headline reductions of 61.9% and 71.0% are obtained on the same examples that determine the operating point, they are in-sample fits rather than honest estimates of out-of-sample performance. A held-out test set, cross-validation, or a separate benchmark with hyperparameters fixed before evaluation is required to support the central claim.
  2. [Section 4.4.1, Table 2] The chosen operating point alpha=0.7 is not robustly separated from degenerate behavior: alpha=0.8 already reduces F1 to 51.1 and output length to 41.1, and alpha=0.9 collapses F1 to 8.7 with output length 10, making the low CHAIR values vacuous. This knife-edge sensitivity means the reported improvement at alpha=0.7 could be an artifact of selecting an extreme threshold without proper validation.
  3. [Section 3.3 and Figure 3] The method is underspecified: Figure 3 labels the modified attention layer as 'Layer 2', but Equations (1)-(3) and Section 3.3 do not state which token positions or which LLM layers are modified, and Section 4.3 introduces 'head-specific modulation' without defining it in Section 3. This prevents reproduction and leaves the central mechanism ambiguous, especially regarding the transferability of the layer choice across architectures.
  4. [Section 4.3 and Equation (6)] The abstract claims the method reduces hallucination 'while maintaining accuracy', but Table 1 shows F1 decreasing from 75.9 to 71.8, which is a meaningful drop. Furthermore, the Recall metric defined in Equation (6) is never reported in any table. The paper should report Recall, error bars from multiple runs, and a length-controlled analysis to rule out the possibility that reduced hallucination is simply due to shorter or different-length generated captions.
minor comments (6)
  1. [Abstract and Section 5] The abstract reports an improvement of 'up to 62.3%' while Section 5 reports 61.9% at the sentence level and 71.0% at the instance level; please reconcile these numbers and clarify the comparison baseline.
  2. [Section 4.1] The paper states that PAINT can be extended to InstructBLIP, LLaMA-Adapter-v2, MiniGPT4, and Shikra, but no experiments on those models are reported; please provide at least one cross-architecture validation or temper the claim.
  3. [Figures 2 and 3] The attention heatmaps in Figures 2 and 3 are difficult to read; please use larger fonts, a perceptually uniform colormap, and clear annotations.
  4. [References] References [14] and [15] appear to be the same paper (OPERA) published in two venues; please consolidate them to avoid duplicate citations.
  5. [Section 4.2] POPE is introduced as a benchmark in Section 4.2, but no POPE results are reported in any table; please either include the results or remove the mention.
  6. [Throughout] There are minor language issues, such as 'phenomena' in Section 1 and 'demonstrate' in the abstract, which should be corrected.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported hallucination reductions are in-sample optima: α, β, and N are tuned on the same 500-image evaluation set used to compute the headline 61.9% and 71.0% improvements.

  1. fitted input called prediction [Sections 4.2, 4.4.1-4.4.3, Table 1]
    "We sample 500 validation instances from MSCOCO 2014 to assess hallucination. ... The value of the margins as well as N are identified through our experiments. ... The optimal performance is achieved at β = 0.4, which balances hallucination reduction with overall model capability. ... The optimal setting is 25%, where amplified attention reduces CHAIRS to 17.6, CHAIRI to 4.0, and maintains a balanced F1 = 71.8, though with an increased length of 187.8. ..."

    The same 500 MSCOCO validation instances are used both to select α, β, and N and to report the headline reductions. Section 4.2 fixes the sample; Section 1 says the margins and N are 'identified through our experiments'; Sections 4.4.1-4.4.3 pick α=0.7, β=0.4, and N=25% by inspecting CHAIRS, CHAIRI, and F1 on that sample; Table 1 then reports exactly those values as PAINT's performance. No held-out split, cross-validation, or uncertainty estimate is described, so the 61.9% and 71.0% improvements are the in-sample optimum of a grid search on the evaluation set, not a prediction at a fixed configuration. This is the fitted-input-called-prediction pattern: fitted margins are presented as 'experimentally learned' and their evaluation-set metrics are presented as expected reduction.

full rationale

The paper's central empirical claim is that PAINT reduces hallucinations by 61.9% at the sentence level and 71.0% at the instance level while maintaining accuracy. The evaluation protocol, however, selects α, β, and N on the same 500 validation instances used to compute those headline numbers. Section 4.2 states the sample; Section 1 states that the margins and N are 'identified through our experiments'; Sections 4.4.1-4.4.3 choose α=0.7, β=0.4, and N=25% by inspecting CHAIRS, CHAIRI, and F1 on that sample; and Section 4.3/Table 1 reports exactly the resulting values as PAINT's performance. No held-out split, cross-validation, or uncertainty estimate is described. This makes the reported 61.9% and 71.0% improvements the in-sample optimum of a small grid search rather than an independent prediction at a fixed configuration. The knife-edge sensitivity in Tables 2-4 (e.g., α=0.8 drops F1 to 51.1 and α=0.9 collapses output length to 10) reinforces that the selected point is not robust. This is the fitted-input-called-prediction pattern: the fitted margins are renamed as learned, and their evaluation-set metrics are reported as the method's expected reduction. The token-taxonomy motivation has external support from Darcet et al. [7], so I do not score the self-citation in [2] as load-bearing. Consequently, the central numerical claim is partially circular, but the underlying intervention itself is not definitionally equivalent to its input; hence score 6 rather than 8 or 10.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method depends on three hyperparameters fitted to the evaluation set and on a qualitative attention-decay theory that is not quantitatively validated.

free parameters (3)
  • alpha (local token boost margin) = 0.7
    Selected by scanning alpha on the same 500-instance MSCOCO evaluation set (Table 2).
  • beta (summary token boost margin) = 0.4
    Selected by scanning beta on the same evaluation set (Table 3).
  • N (top token ratio) = 25%
    Selected by scanning the token ratio on the same evaluation set (Table 4).
assumptions (3)
  • domain assumption Hallucination arises from progressive weakening of visual-token attention in deeper LLM layers.
    Section 1 and Figure 1, an observation on LLaVA-1.5 that is generalized without evidence.
  • domain assumption CLS-to-patch attention in ViT layer 1 and layer 24 respectively identify local and summary tokens worth boosting.
    Section 3.1, Eqs. (1) and (2); the mapping to LLM token positions is not verified.
  • ad hoc to paper Boosting top-N% token attention by additive margins (Eq. 3) is a safe intervention that does not break generation.
    The boost is an arbitrary additive perturbation; its stability is shown to be fragile in Tables 2 and 3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PAINT: Paying Attention to INformed Tokens to Mitigate Hallucination in Large Vision-Language Model." pith.science (2026). https://pith.science/paper/JFJXXYF4

@misc{pith2026250112206,
  author       = {Pith},
  title        = {Pith review of: PAINT: Paying Attention to INformed Tokens to Mitigate Hallucination in Large Vision-Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JFJXXYF4}},
  note         = {Machine review of arXiv:2501.12206}
}
read the original abstract

Large Vision Language Models (LVLMs) have demonstrated remarkable capabilities in understanding and describing visual content, achieving state-of-the-art performance across various vision-language tasks. However, these models often generate descriptions containing objects or details that are absent in the input image, a phenomenon commonly known as hallucination. Our work investigates the key reasons behind this issue by analyzing the pattern of self-attention in transformer layers. We find that hallucinations often arise from the progressive weakening of attention weight to visual tokens in the deeper layers of the LLM. Some previous works naively boost the attention of all visual tokens to mitigate this issue, resulting in suboptimal hallucination reduction. To address this, we identify two critical sets of visual tokens that facilitate the transfer of visual information from the vision encoder to the LLM. Local tokens encode grounded information about objects present in an image, while summary tokens capture the overall aggregated representation of the image. Importantly, these two sets of tokens require different levels of weight enhancement. To this end, we propose \textbf{PAINT} (\textbf{P}aying \textbf{A}ttention to \textbf{IN}formed \textbf{T}okens), a plug-and-play framework that intervenes in the self-attention mechanism of the LLM, selectively boosting the attention weights of local and summary tokens with experimentally learned margins. Evaluation on the MSCOCO image captioning dataset demonstrate that our approach reduces hallucination rates by up to 62.3\% compared to baseline models while maintaining accuracy. Code is available at \href{https://github.com/hasanar1f/PAINT}{https://github.com/hasanar1f/PAINT}

Figures

Figures reproduced from arXiv: 2501.12206 by the authors.

Figure 1
Figure 1. Imbalanced self-attention weights observed in LLaVA [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. CLS-to-patch attention in ViT [8]. Initial layers (e.g., 0-1) highlight local patches containing fine-grained visual details (local tokens), while deeper layers (e.g., 22-23) emphasize sum￾mary tokens that aggregate global information (summary tokens). 3. Methodology We propose Paying Attention to INformed Tokens (PAINT), a hallucination mitigation method. Unlike previous meth￾ods that uniformly treat all visual tok… view at source ↗
Figure 3
Figure 3. PAINT hallucination mitigation method cal and summary tokens. We hypothesize that local and summary tokens require different levels of attention en￾hancement, rather than the uniform boosting applied in pre￾vious methods. The appropriate degree of enhancement re￾mains an open research question, which we explore through further experiments. To address this, we introduce two hy￾perparameters, α and β, to selectively b… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings

    cs.CV 2025-11 conditional novelty 5.0 of 10

    Injecting an average-pooled visual embedding into every text token improves hallucination-benchmark scores of Video-LLaVA by small single-digit amounts.

  2. CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    CAI reduces object hallucination in LVLMs by injecting caption-query attention patterns into selected attention heads at inference time.

Reference graph

Works this paper leans on

48 extracted references · 26 canonical work pages · cited by 2 Pith papers

  1. [1]

    Agla: Mitigating object hallucinations in large vision- language models with assembly of global and local attention,

    Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations in large vision- language models with assembly of global and local attention,

    work page

  2. [2]

    Nikolopou- los, Hans Vandierendonck, Deepu John, and Bo Ji

    Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S. Nikolopou- los, Hans Vandierendonck, Deepu John, and Bo Ji. Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models, 2024. 2

    work page 2024

  3. [3]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 1

    arXiv 2023

  4. [4]

    Shikra: Unleashing multi- modal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 3

    arXiv 2023

  5. [5]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march

    work page

  6. [6]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  7. [7]

    Vision transformers need registers, 2024

    Timoth ´ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, 2024. 2

    work page 2024

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. 1, 2

    work page 2021

Show all 48 references
  1. [9]

    Multi-modal hal- lucination control by visual information grounding, 2024

    Alessandro Favero, Luca Zancato, Matthew Trager, Sid- dharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hal- lucination control by visual information grounding, 2024. 2

    2024

  2. [10]

    Llama-adapter v2: Parameter-efficient vi- sual instruction model

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xi- angyu Yue, et al. Llama-adapter v2: Parameter-efficient vi- sual instruction model. arXiv preprint arXiv:2304.15010 ,

    arXiv

  3. [11]

    Cogagent: A visual lan- guage model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual lan- guage model for gui agents. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),...

    2024

  4. [12]

    A survey on evaluation of multimodal large language models, 2024

    Jiaxing Huang and Jingyi Zhang. A survey on evaluation of multimodal large language models, 2024. 1

    2024

  5. [13]

    Visual instruction tuning towards general- purpose multimodal model: A survey

    Jiaxing Huang, Jingyi Zhang, Kai Jiang, Han Qiu, and Shijian Lu. Visual instruction tuning towards general- purpose multimodal model: A survey. arXiv preprint arXiv:2312.16602, 2023. 1

    2023 arXiv

  6. [14]

    Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation, 2024

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation, 2024. 2

    2024

  7. [15]

    Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Confer...

    2024

  8. [16]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR,

  9. [17]

    Code: Contrasting self-generated description to combat hal- lucination in large multi-modal models, 2024

    Junho Kim, Hyunjun Kim, Yeonju Kim, and Yong Man Ro. Code: Contrasting self-generated description to combat hal- lucination in large multi-modal models, 2024. 2

    2024

  10. [18]

    Building and better understanding vision- language models: insights and future directions, 2024

    Hugo Laurenc ¸on, Andr´es Marafioti, Victor Sanh, and L ´eo Tronchon. Building and better understanding vision- language models: insights and future directions, 2024. 1

    2024

  11. [19]

    Reference-free hallucination de- tection for large vision-language models, 2024

    Qing Li, Chenyang Lyu, Jiahui Geng, Derui Zhu, Maxim Panov, and Fakhri Karray. Reference-free hallucination de- tection for large vision-language models, 2024. 1

    2024

  12. [20]

    Vlm-eval: A general evaluation on video large language models, 2023

    Shuailin Li, Yuang Zhang, Yucheng Zhao, Qiuyue Wang, Fan Jia, Yingfei Liu, and Tiancai Wang. Vlm-eval: A general evaluation on video large language models, 2023. 1

    2023

  13. [21]

    Evaluating object hallucination in large vision-language models, 2023

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models, 2023. 2

    2023

  14. [22]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 3

    2023 arXiv

  15. [23]

    Gpt- 4 enhanced multimodal grounding for autonomous driving: Leveraging cross-modal attention with large language mod- els

    Haicheng Liao, Huanming Shen, Zhenning Li, Chengyue Wang, Guofa Li, Yiming Bie, and Chengzhong Xu. Gpt- 4 enhanced multimodal grounding for autonomous driving: Leveraging cross-modal attention with large language mod- els. Communications in Transportation Research, 4:100116,

  16. [24]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceeding...

    2014

  17. [25]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural Information Processing Systems, pages 34892–34916. Curran Associates, Inc., 2023. 3

    2023

  18. [26]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1

    2024

  19. [27]

    A survey on hallucination in large vision-language models,

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiu- tian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models,

  20. [28]

    Paying more atten- tion to image: A training-free method for alleviating halluci- nation in lvlms, 2024

    Shi Liu, Kecheng Zheng, and Wei Chen. Paying more atten- tion to image: A training-free method for alleviating halluci- nation in lvlms, 2024. 2, 3

    2024

  21. [29]

    Negative object presence evaluation (nope) to measure object hallucination in vision-language models

    Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. Negative object presence evaluation (nope) to measure object hallucination in vision-language models. arXiv preprint arXiv:2310.05338, 2023. 1

    2023 arXiv

  22. [30]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023. 1

    arXiv

  23. [31]

    Vista-llama: Reducing hallucination in video language models via equal distance to visual tokens

    Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reducing hallucination in video language models via equal distance to visual tokens. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13151–13160, 2024. 2

    2024

  24. [32]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  25. [33]

    Object hallucination in image cap- tioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image cap- tioning. arXiv preprint arXiv:1809.02156, 2018. 3

    2018 arXiv

  26. [34]

    Arık, and Tomas Pfister

    Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan ¨O. Arık, and Tomas Pfister. Mitigating object hallucination via data augmented contrastive tuning,

  27. [35]

    Visual text meets low-level vision: A comprehen- sive survey on visual text processing, 2024

    Yan Shu, Weichao Zeng, Zhenhang Li, Fangmin Zhao, and Yu Zhou. Visual text meets low-level vision: A comprehen- sive survey on visual text processing, 2024. 1

    2024

  28. [36]

    Mitigating entity-level hallu- cination in large language models, 2024

    Weihang Su, Yichen Tang, Qingyao Ai, Changyue Wang, Zhijing Wu, and Yiqun Liu. Mitigating entity-level hallu- cination in large language models, 2024. 2

    2024

  29. [37]

    Evaluation and analysis of hal- lucination in large vision-language models

    Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, et al. Evaluation and analysis of hal- lucination in large vision-language models. arXiv preprint arXiv:2308.15126, 2023. 1

    2023 arXiv

  30. [38]

    Visionllm: Large language model is also an open-ended decoder for vision-centric tasks, 2023

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks, 2023. 1

    2023

  31. [39]

    Huggingface’s transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chau- mond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R´emi Louf, Morgan Funtowicz, et al. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. 3

    1910 arXiv

  32. [40]

    Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models

    Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265, 2023. 1

    2023 arXiv

  33. [41]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023. 1

    2023 arXiv

  34. [42]

    Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data, 2024

    Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wen- tao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data, 2024. 2

    2024

  35. [43]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 1

    2023

  36. [44]

    Vision-language models for vision tasks: A survey

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5625–5644, 2024. 1

    2024

  37. [45]

    Opt: Open pre-trained trans- former language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained trans- former language models. arXiv preprint arXiv:2205.01068,

    arXiv

  38. [46]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 3

    2023 arXiv

  39. [47]

    Ibd: Alleviating hallucinations in large vision- language models via image-biased decoding, 2024

    Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating hallucinations in large vision- language models via image-biased decoding, 2024. 2

    2024

  40. [2023]

    org/blog/2023-03-30-vicuna, 3(5),

    URL https://lmsys. org/blog/2023-03-30-vicuna, 3(5),

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.