Pith. sign in

REVIEW 7 cited by

Mitigating Hallucination in Visual Language Models with Visual Supervision

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.16479 v1 pith:ETPXKQOG submitted 2023-11-27 cs.CV

Zhiyang Chen , Yousong Zhu , Yufei Zhan , Zhaowen Li , Chaoyang Zhao , Jinqiao Wang , Ming Tang This is my paper · ORCID
classification cs.CV
keywords hallucinationdetailedimagelvlmsmodelsvisiongenerateloss
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large vision-language models (LVLMs) suffer from hallucination a lot, generating responses that apparently contradict to the image content occasionally. The key problem lies in its weak ability to comprehend detailed content in a multi-modal context, which can be mainly attributed to two factors in training data and loss function. The vision instruction dataset primarily focuses on global description, and the auto-regressive loss function favors text modeling rather than image understanding. In this paper, we bring more detailed vision annotations and more discriminative vision models to facilitate the training of LVLMs, so that they can generate more precise responses without encounter hallucination. On one hand, we generate image-text pairs with detailed relationship annotations in panoptic scene graph dataset (PSG). These conversations pay more attention on detailed facts in the image, encouraging the model to answer questions based on multi-modal contexts. On the other hand, we integrate SAM and mask prediction loss as auxiliary supervision, forcing the LVLMs to have the capacity to identify context-related objects, so that they can generate more accurate responses, mitigating hallucination. Moreover, to provide a deeper evaluation on the hallucination in LVLMs, we propose a new benchmark, RAH-Bench. It divides vision hallucination into three different types that contradicts the image with wrong categories, attributes or relations, and introduces False Positive Rate as detailed sub-metric for each type. In this benchmark, our approach demonstrates an +8.4% enhancement compared to original LLaVA and achieves widespread performance improvements across other models.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling

    cs.CV 2025-07 conditional novelty 6.0 of 10

    INTER is a training-free logit-correction method that adds Harsanyi interaction scores to selected keyword tokens, lowering hallucination on six LVLM benchmarks.

  2. ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A single-layer, single-query intervention that amplifies attention heads with high text-to-visual entropy reduces hallucination in LVLMs at about 1.07x the inference time of regular decoding.

  3. MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM

    cs.CV 2025-05 conditional novelty 6.0 of 10

    MIRAGE is a benchmark that separates reasoning hallucinations from perception errors in multimodal LLMs, and Logos is a curriculum reinforcement fine-tuning method that reduces logical hallucinations.

  4. Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Ensemble Decoding reduces object hallucination in large vision-language models by ensembling logits from attention-weighted image sub-images.

  5. Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models

    cs.CV 2025-02 conditional novelty 6.0 of 10

    DeGF reduces hallucinations in vision-language models by generating an image from the model's own response and using the divergence between predictions on original and generated images to switch between complementary ...

  6. Dual-Stage Value-Guided Inference with Margin-Based Reward Adjustment for Fast and Faithful VLM Captioning

    cs.CV 2025-06 reject novelty 5.0 of 10

    A two-stage, value-guided decoding strategy with a margin-based reward adjustment is claimed to yield more faithful, detailed VLM captions at about a quarter of VisVM's inference cost.

  7. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools