REVIEW 14 cited by
Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Vision-Language Models (LVLMs) are increasingly adept at generating contextually detailed and coherent responses from visual inputs. However, their application in multimodal decision-making and open-ended generation is hindered by a notable rate of hallucinations, where generated text inaccurately represents the visual contents. To address this issue, this paper introduces the Instruction Contrastive Decoding (ICD) method, a novel approach designed to reduce hallucinations during LVLM inference. Our method is inspired by our observation that what we call disturbance instructions significantly exacerbate hallucinations in multimodal fusion modules. ICD contrasts distributions from standard and instruction disturbance, thereby increasing alignment uncertainty and effectively subtracting hallucinated concepts from the original distribution. Through comprehensive experiments on discriminative benchmarks (POPE and MME) and a generative benchmark (LLaVa-Bench), we demonstrate that ICD significantly mitigates both object-level and attribute-level hallucinations. Moreover, our method not only addresses hallucinations but also significantly enhances the general perception and recognition capabilities of LVLMs.
Forward citations
Cited by 14 Pith papers
-
ChartCap: Mitigating Hallucination of Dense Chart Captioning
A new 565K-pair chart-caption dataset with schema-based dense captions and a reference-free visual consistency metric improves VLM captioning and reduces hallucination.
-
TPCD: Tone-Pressure Contrastive Decoding and the Label-Free Gating Bottleneck in Vision-Language Models
Subtracting a VLM's high-pressure 'commitment' logits from its neutral logits (TPCD) suppresses prompt-induced hallucination on the tone-matters benchmark, but only with routing gates that remain benchmark-specific an...
-
Measuring Epistemic Humility in Multimodal Large Language Models
A new 22,831-question visual benchmark shows that major multimodal LLMs struggle to reject false answer options, often scoring near random when abstaining is the only correct response.
-
Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation
A dual-level attention intervention that boosts salient visual-token attention and suppresses text/system attention during decoding reduces hallucination rates in LLaVA, MiniGPT-4, and mPLUG-Owl2 on POPE and CHAIR.
-
Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Models
EVRB is a three-part inference-time method that prunes ambiguous visual tokens, divides the model's output distribution by a text-only prior, and triggers early stopping to reduce hallucination in LVLMs.
-
Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding
Ensemble Decoding reduces object hallucination in large vision-language models by ensembling logits from attention-weighted image sub-images.
-
Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs
On a new extended needle-in-a-haystack benchmark, explicit anti-hallucination prompts and dispersed fact placement cause some long-context LLMs to over-refuse or collapse in accuracy, while others remain robust.
-
Energy-Guided Decoding for Object Hallucination Mitigation
An energy-guided, training-free decoding rule that chooses the layer with minimal energy reduces object hallucination and yes-bias on several benchmarks.
-
CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models
CAI reduces object hallucination in LVLMs by injecting caption-query attention patterns into selected attention heads at inference time.
-
Dual-Stage Value-Guided Inference with Margin-Based Reward Adjustment for Fast and Faithful VLM Captioning
A two-stage, value-guided decoding strategy with a margin-based reward adjustment is claimed to yield more faithful, detailed VLM captions at about a quarter of VisVM's inference cost.
-
Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Models
A causal intervention architecture with confounder dictionaries is applied to LLaVA, producing modest hallucination reductions on POPE and CHAIR but with methodological caveats.
-
Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding
FarSight adds upper-triangular negative biases to the causal mask to absorb outlier-token attention, reducing hallucinations in MLLMs without training.
-
Adaptive Sparse Softmax: An Effective and Efficient Softmax Variant
The preprint's abstract claims a sparse softmax variant that masks non-competitive classes and accelerates training, but the provided body contains an unrelated chart-captioning paper and none of the claimed method.
-
MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Models
MCA-LLaVA reindexes image tokens by sums of mirrored 2D coordinates so instruction tokens attend across the whole image, reducing hallucination on POPE, CHAIR, and MME.
Discussion (0). Sign in to comment.