Pith. sign in

REVIEW 4 major objections 7 minor 65 references

A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models

T0 review · 4 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Visual hallucination in LLaVA-like models comes mainly from the vision encoder and the projector, not the language model.

desk verdict Systematic component analysis and two new benchmarks are real contributions, but the headline 'LLM is faithful' claim is contradicted by the paper's own relation results. read the letter →

arxiv 2505.01958 v1 pith:Q5Z6V7MI submitted 2025-05-04 cs.CV cs.CL

Liqiang Jing , Guiming Hardy Chen , Ehsan Aghazadeh , Xin Eric Wang , Xinya Du This is my paper · ORCID
classification cs.CVcs.CL
keywords visualobjecthallucinationlargevision-languagemodelsLLaVAvisionencoderprojectoralignmentbenchmarkscontrastivecognition-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large vision-language models such as LLaVA hallucinate objects, attributes, and relations when answering questions about images. This paper tries to locate where those hallucinations originate by taking the model apart into its three components — the language model, the CLIP vision encoder, and the projector that connects them — and testing each one in isolation. It concludes that the language model is largely faithful when it receives the image content as text, that the vision encoder itself misperceives objects, and that the projector preserves visual information but aligns it poorly with the language model's embedding space. If this diagnosis is right, hallucination is primarily a perception and cross-modal alignment problem, and the field's emphasis on decoding strategies and language-model corrections addresses a downstream symptom rather than the root cause.

What carries the argument

The load-bearing mechanism is component-wise decomposition of a LLaVA-like model, with each component probed by a dedicated diagnostic. The language decoder is isolated by text substitution: replace the image with ground-truth objects, attributes, and relations, then measure question-answering accuracy. The vision encoder is isolated by CLIP text-image matching on yes/no templates. The projector is probed twice: V-information-based linear probing (comparing pre- and post-projection features on classification) to test information preservation, and cosine similarity between projected image features and caption embeddings to test visual-textual alignment. The proposed mitigations are CLIP post-training with synthetic hard negatives, fine-grained perception-based instruction tuning, and a contrastive alignment loss for the projector with three instantiations.

What would settle it

Run the text-substitution experiment with the full ground-truth Visual Genome annotation for each image rather than a filtered subset: if LLaVA's accuracy on QA-VisualGenome remains far below the reported 63–68% attribute and relation accuracy even when exhaustive visual facts are supplied as text, the claim that the language model is faithful would be falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is a three-part localization of visual object hallucination in LLaVA-like models. First, when a model is given a complete textual description of an image — objects for POPE, objects plus attributes and relations for QA-VisualGenome — the underlying LLM answers almost perfectly (up to 99.67% on POPE random), so the language model is not the hallucination source. Second, CLIP, acting as the vision encoder, makes substantial errors on the simple text-image matching task "There is a {object} in the image," scoring only 83.33% on POPE random, which indicates perception-level hallucination. Third, linear probing shows the projector retains essentially all visual information (post-projection accuracy drops less than 2% on CIFAR and ImageNet), but the cosine similarity between projected image features and the embeddings of their own captions is only 0.03–0.06, indicating a failure to align the visual and textual spaces. The paper then proposes component-specific fixes and shows they reduce hallucination on perception-based benchmarks and, for the contrastive alignment variants, on a new cognition-based benchmark.

Load-bearing premise

The argument that the LLM is faithful rests on the assumption that a textual list of objects, attributes, and relations is a complete substitute for the image; if that list omits or misrepresents visual content, the low error with text input does not prove that the LLM is not a hallucination source.

Editorial extensions

If this is right

  • Hallucination-mitigation efforts in LLaVA-like models should prioritize the vision encoder and the projector; decoding-time or LLM-only fixes address a secondary symptom.
  • Fine-grained CLIP post-training or fine-grained instruction tuning yields measurable gains on POPE, POPE-NoCaps, and QA-VisualGenome without adding parameters at inference time.
  • Adding a contrastive alignment stage for the projector (about 12 minutes of training in the paper's setup) improves a cognition-based hallucination benchmark, suggesting that better alignment helps the model draw on world knowledge stored in the LLM.
  • Attribute and relation hallucinations are harder to cure by projector alignment alone, implying that better visual representations are the bottleneck for fine-grained perception.
  • The new QA-VisualGenome and QA-FB15k benchmarks extend hallucination evaluation beyond object existence to attributes, relations, and knowledge-dependent cognition.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the vision-encoder locus holds, hallucination rates should scale with CLIP's per-category matching error; a testable prediction is that models with improved CLIP fine-grained discrimination will hallucinate less on exactly the object categories where CLIP currently confuses present and absent objects.
  • The low cosine similarity between projected visual features and caption embeddings may mean that projection is not performing alignment in a simple cosine-similarity geometry; the authors do not explore whether the autoregressive loss itself is doing the alignment implicitly, in which case cosine similarity is the wrong diagnostic.
  • The text-substitution experiment could be reused as a general audit tool: for any new vision-language model, the gap between image-input accuracy and caption-input accuracy localizes hallucination to perception versus generation without needing component-level retraining.
  • The QA-FB15k results hint at a connection between modality alignment and factual reasoning that the paper does not develop: aligning the projector better may unlock knowledge already present in the language model, which could be probed directly by measuring knowledge-query accuracy under varying alignment strengths.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper presents a component-level analysis of hallucination in LLaVA-like large vision-language models. It claims three findings: (1) the LLM generates faithful content when image information is supplied as text, (2) the CLIP vision backbone produces perception-level hallucinations, and (3) the projector preserves visual information but fails to align visual and textual spaces. Based on these diagnoses, it proposes targeted mitigations: fine-grained contrastive tuning of CLIP, fine-grained perception-based visual instruction tuning, and three contrastive projector-alignment objectives. It also introduces two benchmarks, QA-VisualGenome for attribute/relation hallucinations and QA-FB15k for cognition-based hallucinations, and evaluates the proposed methods on POPE, POPE-NoCaps, QA-VisualGenome, QA-FB15k, Amber, and LLaVA-Bench.

Significance. If the component-level diagnosis is correct, the paper makes a useful contribution by shifting attention from decoding-side or LLM-side hallucination fixes to perception and alignment problems, and it provides concrete training objectives for those components. The two new benchmarks also fill real gaps in attribute/relation and cognition-based hallucination evaluation. The paper's strengths are its breadth: it probes each component, proposes several mitigation variants, and evaluates on multiple benchmark suites. However, the central 'LLM is faithful' result is not established because the text-substitution probe relies on incomplete Visual Genome scene graphs, and the paper's own Table 1 relation split contradicts the claim at one model scale. The projector-alignment claim also lacks direct causal evidence, and the reported mitigation gains are small and unreplicated. The contribution is promising but needs reanalysis and additional experiments before the central claims can be accepted.

major comments (4)
  1. [§2.2, Conjecture 1; Table 1] The claim that the LLM is faithful when visual information is replaced by text is not established by the reported experiments. For QA-VisualGenome, the text input is a Visual Genome scene-graph listing, but no completeness check against the image is provided, and Visual Genome annotations are known to be sparse. Table 1 shows that text-only Vicuna-7B_LLaVA scores 63.2 on the relation split while image-based LLaVA-7B scores 67.57, whereas at the 13B scale the ordering reverses (84.30 versus 56.40). Either the text surrogate is an incomplete or biased proxy, which invalidates the isolation of the LLM, or the LLM itself contributes to relational hallucination at the 7B scale; both possibilities contradict the strong 'LLM is faithful' conclusion, which is load-bearing for the paper's other findings.
  2. [§2.4, Conjecture 3; Table 3] The projector information-preservation claim is not fully supported by Table 3. The table reports only LLaVA-13B, yet the text says that 'the performance percentage drop of post-projection features is less than 2%, indicating that the visual features are well preserved by the projectors in both models.' No 7B probing results are shown even though LLaVA-7B is the model used in all later mitigation experiments, and no variance or repeated-seed statistics are reported. The claim that the projector is not a source of information loss is therefore currently supported for only one model scale.
  3. [§2.4, Table 4; §4.2, Table 6] The conclusion that low cosine similarity between projected image features and caption embeddings causes hallucination is a functional claim that the paper does not test directly. Cosine similarity in an anisotropic embedding space can be low even when features are perfectly usable for next-token prediction, so the appropriate control is to compare against random caption embeddings or to show that increasing this similarity reduces hallucination. The mitigation experiments in Table 6 do not provide that evidence: contrastive alignment variants improve POPE accuracy by at most about 1.1 points, and on QA-VisualGenome they often decrease attribute accuracy (e.g., from 64.67 to 60.95 in the first Int. Align. row). Section 4.2 itself concedes that object hallucination may not be directly related to alignment, which undercuts the causal interpretation of Table 4.
  4. [§4.1, Table 5] The claims that w-ECLIP and w-FineIns reduce hallucination are based on very small accuracy and F1 differences with no error bars or significance testing. For example, on POPE Random, LLaVA-7B has Acc 87.42/F1 86.36 and w-ECLIP has Acc 87.80/F1 86.87; on QA-VisualGenome Relation, w-ECLIP declines from 67.57 to 67.00. Without multiple seeds or confidence intervals, these differences are within plausible run-to-run variation, so the mitigation claim is not yet demonstrated.
minor comments (7)
  1. [Table 1] Several entries concatenate two numbers without a space (e.g., '79.5080.79', '100.00100.00', '99.6799.67', '87.9089.15'); please fix the formatting for readability.
  2. [§3.2.1, Eq. (7)] Equation (7) writes L_itg = -p(R|I,T); as written this is not a loss. It should presumably be -log p(R|I,T).
  3. [§2.2] The phrase 'the LLM after the pertaining and instruction tuning' should read 'after the pre-training and instruction tuning.'
  4. [§2.4] The notation H_V(Y) is used in the V-information definitions but never defined; please define it as the minimal V-entropy of Y without conditioning, or clarify the notation.
  5. [Table 4] The caption says caption embeddings are processed by the Vicuna tokenizer, but the paper does not specify how the caption embedding is computed (e.g., mean-pooled token embeddings, last hidden state) or what 'Image Res. (575, 488)' denotes; please clarify.
  6. [Table 6] The two 'Int. Align.' rows are distinguished only by an unlabeled symbol; please add a readable key such as 'trainable λ' and 'fixed λ' that is consistent with Section 3.2.1.
  7. [§4.1] The text says w-ECLIP is 'comparable or even better' than w-FineIns, but Table 5 shows mixed ordering across columns; please state the comparison more precisely.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the component-level diagnosis is empirical and the mitigation gains are evaluated on separate hallucination benchmarks, with no fitted input renamed as a prediction.

full rationale

I find no step that reduces by construction to its own input. Section 2.2 tests Conjecture 1 by feeding the LLM ground-truth text descriptions (objects/attributes/relations) in place of images; the model's accuracy is an empirical quantity, not an artifact of the probe design, and Table 1 shows non-trivial failures (e.g., Vicuna-7B_LLaVA relation accuracy 63.2 vs image-based LLaVA-7B 67.57). Section 2.3 tests Conjecture 2 by direct CLIP image-text matching on object templates. Section 2.4 tests Conjecture 3 by linear probing pre/post-projector features on CIFAR/ImageNet, and Conjecture 4 by cosine similarity between projected image and caption embeddings. None of these quantities is a fitted parameter, and none is defined in terms of the paper's proposed mitigation. The mitigation methods (w-ECLIP, w-FineIns, contrastive projector alignment) are evaluated on held-out hallucination benchmarks (POPE, POPE-NoCaps, QA-VisualGenome, QA-FB15k, Amber, LLaVA-Bench), not on the diagnostic probes themselves. Self-citations (e.g., Jing and Du 2024; Jing et al. 2024) appear only in related work and are not load-bearing. The closest concern, that the Section 2.2 text surrogate may not fully substitute for visual input, is an experimental-validity question rather than a circularity: the text input is not derived from model outputs, and the accuracy numbers are not forced by the construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central diagnostic claims rest on three proxy measurements: caption substitution for visual input, CLIP template matching for perception fidelity, and cosine similarity for cross-modal alignment. None of these is validated against an external ground truth, so they are the main assumptions a replicator must buy.

free parameters (5)
  • lambda (contrastive alignment weight) = initialized to 5; learnable in one setting, fixed in another
    Hand-chosen initial weight for the projector alignment loss in Section 3.2.1; the central mitigation results may depend on it.
  • lambda_1, lambda_2 (margin loss weights) = both set to 1
    Hand-set weights for the CLIP margin losses in Equations 4 and 6; ablation in Appendix F shows modest sensitivity.
  • tau_1, tau_2 (margin thresholds) = not specified
    Margin thresholds in Equations 4 and 5 for the contrastive loss; values are not reported, affecting reproducibility.
  • beta (temperature) = not specified
    Temperature in the image-text contrastive loss in Equation 1; not reported in the paper.
  • number of inserted or removed objects = one to three per caption
    Design choice for generating negative CLIP training examples in Section 3.1.
assumptions (5)
  • domain assumption Linear probing accuracy difference is a valid estimate of V-information loss through the projector.
    Used in Section 2.4 to conclude the projector preserves visual features; the conclusion depends on probe family and task choice.
  • domain assumption Cosine similarity between a projected image feature and the caption text embedding measures cross-modal alignment quality.
    Used in Conjecture 4 to conclude the projector misaligns modalities; the authors cite prior work but do not establish a threshold.
  • domain assumption Feeding the LLM text descriptions of image content isolates the language model's contribution to hallucination.
    Central to Finding 1 in Section 2.2; requires the text list to be a faithful and complete substitute for the image.
  • domain assumption CLIP text-image matching accuracy with simple templates measures perception-level object hallucination.
    Used in Section 2.3 to attribute hallucinations to the vision encoder; ignores possible prompt artifacts.
  • domain assumption GPT-4 generated negative captions and region captions are correct and useful training signals.
    Used to build w-ECLIP and w-FineIns data in Section 3.1; errors in synthetic data would propagate into the trained models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models." pith.science (2026). https://pith.science/paper/Q5Z6V7MI

@misc{pith2026250501958,
  author       = {Pith},
  title        = {Pith review of: A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q5Z6V7MI}},
  note         = {Machine review of arXiv:2505.01958}
}
read the original abstract

Large Vision-Language Models (LVLMs) demonstrate remarkable capabilities in multimodal tasks, but visual object hallucination remains a persistent issue. It refers to scenarios where models generate inaccurate visual object-related information based on the query input, potentially leading to misinformation and concerns about safety and reliability. Previous works focus on the evaluation and mitigation of visual hallucinations, but the underlying causes have not been comprehensively investigated. In this paper, we analyze each component of LLaVA-like LVLMs -- the large language model, the vision backbone, and the projector -- to identify potential sources of error and their impact. Based on our observations, we propose methods to mitigate hallucination for each problematic component. Additionally, we developed two hallucination benchmarks: QA-VisualGenome, which emphasizes attribute and relation hallucinations, and QA-FB15k, which focuses on cognition-based hallucinations.

Figures

Figures reproduced from arXiv: 2505.01958 by the authors.

Figure 1
Figure 1. An overview of our paper. We first investigate [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Tuning CLIP with fine-grained data (left) and [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The illustration of the hallucinated case for [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 12 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Harsh Agrawal, Peter Anderson, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. https://doi.org/10.1109/ICCV.2019.00904 nocaps: novel object captioning at scale . In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019 , pa...

    arXiv 2019

  4. [4]

    Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Ping Chen, Xiaoqin Zhang, and Shijian Lu. 2025. http://arxiv.org/abs/2406.12718 Mitigating object hallucinations in large vision-language models with assembly of global and local attention

    arXiv 2025

  5. [5]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: visual question answering. In IEEE International Conference on Computer Vision , pages 2425--2433. IEEE Computer Society

    work page 2015

  6. [6]

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Yitzhak Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. 2023. Openflamingo: An open-source framework for training large autoregressive vision-language models. CoRR...

    arXiv 2023

  7. [7]

    Hudson, Ehsan Adeli, Russ B

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ B. Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin...

    arXiv 2021

  8. [8]

    Antoine Bordes, Nicolas Usunier, Alberto Garc \' a - Dur \' a n, Jason Weston, and Oksana Yakhnenko. 2013. https://proceedings.neurips.cc/paper/2013/hash/1cecc7a77928ca8133fa24680a88d2f9-Abstract.html Translating embeddings for modeling multi-relational data . In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Inform...

    2013

Show all 65 references
  1. [9]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    2020 arXiv

  2. [10]

    Yue Chang, Liqiang Jing, Xiaopeng Zhang, and Yue Zhang. 2024. https://doi.org/10.48550/ARXIV.2409.16494 A unified hallucination mitigation framework for large vision-language models . CoRR, abs/2409.16494

    2024 doi arXiv

  3. [11]

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. 2024 a . Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684

    2024 arXiv

  4. [12]

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024 b . https://doi.org/10.1007/978-3-031-73004-7\_2 An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models . In Computer Visi...

    2024 doi

  5. [13]

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. 2024 c . http://arxiv.org/abs/2403.00425 Halc: Object hallucination reduction via adaptive focal-contrast decoding

    2024 arXiv

  6. [14]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. vicuna: An opensource chatbot impressing gpt-4 with 90

    2023

  7. [15]

    Glass, and Pengcheng He

    Yung - Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. 2024. https://openreview.net/forum?id=Th6NyL07na Dola: Decoding by contrasting layers improves factuality in large language models . In The Twelfth International Conference on Learning Repr...

    2024

  8. [16]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. CoRR, abs/2305.06500

    2023 arXiv

  9. [17]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248--255. Ieee

    2009

  10. [18]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognit...

    2021

  11. [19]

    Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. 2023. Multimodal-gpt: A vision and language model for dialogue with humans. CoRR, abs/2305.04790

    2023 arXiv

  12. [20]

    Anisha Gunjal, Jihan Yin, and Erhan Bas. 2023. Detecting and preventing hallucinations in large vision language models. arXiv preprint arXiv:2308.06394

    2023 arXiv

  13. [21]

    John Hewitt, Kawin Ethayarajh, Percy Liang, and Christopher D Manning. 2021. Conditional probing: measuring usable information beyond a baseline. arXiv preprint arXiv:2109.09234

    2021 arXiv

  14. [22]

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2023. https://doi.org/10.48550/ARXIV.2311.17911 OPERA: alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allo...

    2023 doi arXiv

  15. [23]

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024 a . http://arxiv.org/abs/2311.17911 Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation

    2024 arXiv

  16. [24]

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024 b . Deciphering cross-modal alignment in large vision-language models with modality integration rate. arXiv preprint arXiv:2410.07167

    2024 arXiv

  17. [25]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1--38

    2023

  18. [26]

    Liqiang Jing and Xinya Du. 2024. https://doi.org/10.48550/ARXIV.2404.05046 FGAIF: aligning large vision-language models with fine-grained AI feedback . CoRR, abs/2404.05046

    2024 doi arXiv

  19. [27]

    Liqiang Jing, Ruosen Li, Yunmo Chen, and Xinya Du. 2024. http://arxiv.org/abs/2311.01477 Faithscore: Fine-grained evaluations of hallucinations in large vision-language models

    2024 arXiv

  20. [28]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li - Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei - Fei. 2017. https://doi.org/10.1007/S11263-016-0981-7 Visual genome: Connecting language and v...

    2017 doi

  21. [29]

    Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images

    2009

  22. [30]

    http://www.cs.toronto.edu/ kriz/cifar.html Cifar-100 (canadian institute for advanced research)

    Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. http://www.cs.toronto.edu/ kriz/cifar.html Cifar-100 (canadian institute for advanced research)

  23. [31]

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2023. https://doi.org/10.48550/ARXIV.2311.16922 Mitigating object hallucinations in large vision-language models through visual contrastive decoding . CoRR, abs/2311.16922

    2023 doi arXiv

  24. [32]

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. 2023 a . Otter: A multi-modal model with in-context instruction tuning. CoRR, abs/2305.03726

    2023 arXiv

  25. [33]

    Vi \' e gas, Hanspeter Pfister, and Martin Wattenberg

    Kenneth Li, Oam Patel, Fernanda B. Vi \' e gas, Hanspeter Pfister, and Martin Wattenberg. 2023 b . http://papers.nips.cc/paper\_files/paper/2023/hash/81b8390039b7302c909cb769f8b6cd93-Abstract-Conference.html Inference-time intervention: Eliciting truthful answers from a langua...

    2023

  26. [34]

    Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. 2023 c . https://doi.org/10.48550/ARXIV.2312.10665 Silkie: Preference distillation for large visual language models . CoRR, abs/2312.10665

    2023 doi arXiv

  27. [35]

    Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, and Qi Liu. 2023 d . https://doi.org/10.48550/ARXIV.2306.04387 M\( ^ 3 \)it: A large-scale dataset towards multi-modal multilingual instruction tuni...

    2023 doi arXiv

  28. [36]

    Qing Li, Jiahui Geng, Derui Zhu, Zongxiong Chen, Kun Song, Lei Ma, and Fakhri Karray. 2025. https://doi.org/10.1609/AAAI.V39I26.34954 Internal activation revision: Safeguarding vision language models without parameter update . In AAAI-25, Sponsored by the Association for the A...

    2025 doi

  29. [37]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji - Rong Wen. 2023 e . https://doi.org/10.18653/V1/2023.EMNLP-MAIN.20 Evaluating object hallucination in large vision-language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Langua...

    2023 doi

  30. [38]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \' a r, and C

    Tsung - Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \' a r, and C. Lawrence Zitnick. 2014. Microsoft COCO: common objects in context. In ECCV, volume 8693 of Lecture Notes in Computer Science, pages 740--755. Springer

    2014

  31. [39]

    Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2023 a . Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modali...

    2023 arXiv

  32. [42]

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2024 a . http://arxiv.org/abs/2306.14565 Mitigating hallucination in large multi-modal models via robust instruction tuning

    2024 arXiv

  33. [43]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023 d . Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744

    2023 arXiv

  34. [44]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 e . Visual instruction tuning. CoRR, abs/2304.08485

    2023 arXiv

  35. [45]

    Yufang Liu, Tao Ji, Changzhi Sun, Yuanbin Wu, and Aimin Zhou. 2024 b . https://aclanthology.org/2024.emnlp-main.1016 Investigating and mitigating object hallucinations in pretrained vision-language (CLIP) models . In Proceedings of the 2024 Conference on Empirical Methods in N...

    2024

  36. [46]

    Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. 2023. http://arxiv.org/abs/2310.05338 Negative object presence evaluation (nope) to measure object hallucination in vision-language models

    2023 arXiv

  37. [47]

    Jiaying Lu, Jinmeng Rao, Kezhen Chen, Xiaoyuan Guo, Yawen Zhang, Baochen Sun, Carl Yang, and Jie Yang. 2023. Evaluation and mitigation of agnosia in multimodal large language models. CoRR, abs/2309.04041

    2023 arXiv

  38. [48]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen - tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. CoRR, abs/2305.14251

    2023 arXiv

  39. [49]

    OpenAI. 2022. Chatgpt blog post

    2022

  40. [50]

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. In EMNLP, pages 4035--4045. ACL

    2018

  41. [51]

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang - Yan Gui, Yu - Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. 2023 a . https://doi.org/10.48550/ARXIV.2309.14525 Aligning large multimodal models with factually augment...

    2023 doi arXiv

  42. [52]

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Yikang Shen, Chuang Gan, Liang - Yan Gui, Yu - Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. 2023 b . Aligning large multimodal models with factually augmented RLHF . CoRR, abs/2309.14525

    2023 arXiv

  43. [53]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie - Anne Lachaux, Timoth \' e e Lacroix, Baptiste Rozi \` e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aur \' e lien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and effici...

    2023 arXiv

  44. [54]

    Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. 2023. https://doi.org/10.48550/ARXIV.2311.07397 An llm-free multi-dimensional benchmark for mllms hallucination evaluation . CoRR, abs/2311.07397

    2023 doi arXiv

  45. [55]

    Tan Wang, Jianqiang Huang, Hanwang Zhang, and Qianru Sun. 2020. https://doi.org/10.1109/CVPR42600.2020.01077 Visual commonsense R-CNN . In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 , pages 10757--10767. C...

    2020

  46. [56]

    Jinfeng Wei and Xiaofeng Zhang. 2024. https://doi.org/10.1145/3664647.3681076 DOPRA: decoding over-accumulation penalization and re-allocation in specific weighting layer . In Proceedings of the 32nd ACM International Conference on Multimedia, MM 2024, Melbourne, VIC, Australi...

    2024

  47. [57]

    Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. 2024. http://papers.nips.cc/paper\_files/paper/2024/hash/a76ed4a8ef522c823d73925e7fff16d4-Abstract-Conference.html Mitigating object hallucination via concentric causal attention . In Advances in Neural Information Processing S...

    2024

  48. [58]

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. 2023. mplug-owl: Modularization empowers large language models with mult...

    2023 arXiv

  49. [59]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. 2023. Woodpecker: Hallucination correction for multimodal large language models. CoRR, abs/2310.16045

    2023 arXiv

  50. [60]

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai - Tao Zheng, Maosong Sun, and Tat - Seng Chua. 2023. https://doi.org/10.48550/ARXIV.2312.00849 RLHF-V: towards trustworthy mllms via behavior alignment from fine-grained correctional...

    2023 doi arXiv

  51. [61]

    Zihao Yue, Liang Zhang, and Qin Jin. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.633 Less is more: Mitigating multimodal hallucination from an EOS decision perspective . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: L...

    2024 doi

  52. [62]

    Xiaofeng Zhang, Yihao Quan, Chaochen Gu, Chen Shen, Xiaosong Yuan, Shaotian Yan, Hao Cheng, Kaijie Wu, and Jieping Ye. 2024 a . https://doi.org/10.48550/ARXIV.2411.09968 Seeing clearly by layer two: Enhancing attention heads to alleviate hallucination in lvlms . CoRR, abs/2411.09968

    2024 doi arXiv

  53. [63]

    Yue Zhang, Liqiang Jing, and Vibhav Gogate. 2025. https://doi.org/10.1609/AAAI.V39I24.34792 Defeasible visual entailment: Benchmark, evaluator, and reward-driven optimization . In AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25...

    2025 doi

  54. [64]

    Yue Zhang, Jingxuan Zuo, and Liqiang Jing. 2024 b . Fine-grained and explainable factuality evaluation for multimodal summarization. arXiv preprint arXiv:2402.11414

    2024

  55. [65]

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. 2024. https://doi.org/10.48550/ARXIV.2402.11411 Aligning modalities in vision large language models via preference fine-tuning . CoRR, abs/2402.11411

    2024 doi arXiv

  56. [66]

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2023. https://doi.org/10.48550/ARXIV.2310.00754 Analyzing and mitigating object hallucination in large vision-language models . CoRR, abs/2310.00754

    2023 doi arXiv

  57. [67]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. CoRR, abs/2304.10592

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.