REVIEW 11 cited by
Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
While Large Vision-Language Models (LVLMs) have rapidly advanced in recent years, the prevalent issue known as the `hallucination' problem has emerged as a significant bottleneck, hindering their real-world deployments. Existing methods mitigate this issue mainly from two perspectives: One approach leverages extra knowledge like robust instruction tuning LVLMs with curated datasets or employing auxiliary analysis networks, which inevitable incur additional costs. Another approach, known as contrastive decoding, induces hallucinations by manually disturbing the vision or instruction raw inputs and mitigates them by contrasting the outputs of the disturbed and original LVLMs. However, these approaches rely on empirical holistic input disturbances and double the inference cost. To avoid these issues, we propose a simple yet effective method named Self-Introspective Decoding (SID). Our empirical investigation reveals that pretrained LVLMs can introspectively assess the importance of vision tokens based on preceding vision and text (both instruction and generated) tokens. We develop the Context and Text-aware Token Selection (CT2S) strategy, which preserves only unimportant vision tokens after early layers of LVLMs to adaptively amplify text-informed hallucination during the auto-regressive decoding. This approach ensures that multimodal knowledge absorbed in the early layers induces multimodal contextual rather than aimless hallucinations. Subsequently, the original token logits subtract the amplified vision-and-text association hallucinations, guiding LVLMs decoding faithfully. Extensive experiments illustrate SID generates less-hallucination and higher-quality texts across various metrics, without extra knowledge and much additional computation burdens.
Forward citations
Cited by 11 Pith papers
-
Zoom-IQA: Image Quality Assessment with Reliable Region-Aware Reasoning
Zoom-IQA lets a vision-language model iteratively crop and zoom into image regions before giving a quality score, improving reasoning and restoration guidance over single-pass IQA models.
-
DICA: Dual-Indicator Guided Contrastive Alignment in Multimodal Large Language Models
Monitoring attention entropy and image-output correlation during decoding, then applying targeted contrastive corrections, reduces hallucination in multimodal LLMs without retraining.
-
SAVAA: Mitigating Hallucinations in LVLMs via Step-wise Adaptive Visual Attention Amplification
Adaptively scaling visual attention boosting per token, guided by a combined entropy-and-visual-grounding risk score, reduces hallucinations in LVLMs more than fixed boosting.
-
MACD: Model-Aware Contrastive Decoding via Counterfactual Data
MACD reduces Video-LLM hallucination by masking model-identified critical objects/frames via gradient ascent and using the masked video as a contrastive decoding reference.
-
Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation
Tracking positive shifts in visual attention over information-rich query words yields a saliency map that, when used to boost visual and query attention during decoding, reduces object hallucination on CHAIR, POPE, an...
-
Two Causes, Not One: Rethinking Omission and Fabrication Hallucinations in MLLMs
Omission and fabrication hallucinations in MLLMs are claimed to have distinct causes, and the new VPFC method reduces omissions without adding fabrications.
-
INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling
INTER is a training-free logit-correction method that adds Harsanyi interaction scores to selected keyword tokens, lowering hallucination on six LVLM benchmarks.
-
Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation
A dual-level attention intervention that boosts salient visual-token attention and suppresses text/system attention during decoding reduces hallucination rates in LLaVA, MiniGPT-4, and mPLUG-Owl2 on POPE and CHAIR.
-
ROAD: Responsibility-Oriented Reward Design for Reinforcement Learning in Autonomous Driving
A responsibility-aware crash penalty, built from a traffic-law knowledge graph and a vision-language blame classifier, improves MetaDrive success rates and shifts reported collision blame away from the ego vehicle.
-
Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Models
EVRB is a three-part inference-time method that prunes ambiguous visual tokens, divides the model's output distribution by a text-only prior, and triggers early stopping to reduce hallucination in LVLMs.
-
Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding
FarSight adds upper-triangular negative biases to the causal mask to absorb outlier-token attention, reducing hallucinations in MLLMs without training.
Discussion (0). Continue with ORCID to comment.