REVIEW 7 cited by
Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Object hallucination has been an Achilles' heel which hinders the broader applications of large vision-language models (LVLMs). Object hallucination refers to the phenomenon that the LVLMs claim non-existent objects in the image. To mitigate the object hallucinations, instruction tuning and external model-based detection methods have been proposed, which either require large-scare computational resources or depend on the detection result of external models. However, there remains an under-explored field to utilize the LVLM itself to alleviate object hallucinations. In this work, we adopt the intuition that the LVLM tends to respond logically consistently for existent objects but inconsistently for hallucinated objects. Therefore, we propose a Logical Closed Loop-based framework for Object Hallucination Detection and Mitigation, namely LogicCheckGPT. In specific, we devise logical consistency probing to raise questions with logical correlations, inquiring about attributes from objects and vice versa. Whether their responses can form a logical closed loop serves as an indicator of object hallucination. As a plug-and-play method, it can be seamlessly applied to all existing LVLMs. Comprehensive experiments conducted on three benchmarks across four LVLMs have demonstrated significant improvements brought by our method, indicating its effectiveness and generality.
Forward citations
Cited by 7 Pith papers
-
Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theory
Object hallucinations in MLLMs track multi-head spatial inconsistency and temporal visual-attention fade; AFIP corrects both via cross-head enrichment and gated historical reinjection, reducing CHAIR/POPE rates training-free.
-
DICA: Dual-Indicator Guided Contrastive Alignment in Multimodal Large Language Models
Monitoring attention entropy and image-output correlation during decoding, then applying targeted contrastive corrections, reduces hallucination in multimodal LLMs without retraining.
-
Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
Modality bias, an imbalanced attention to text or image during hallucinated outputs, is shown to be mitigated by a training-free attention intervention plus contrastive decoding.
-
Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding
Ensemble Decoding reduces object hallucination in large vision-language models by ensembling logits from attention-weighted image sub-images.
-
OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucination
OmniDPO extends direct preference optimization with audio-video alignment and modality-degradation preference pairs to reduce omni-modal hallucination.
-
Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding
FarSight adds upper-triangular negative biases to the causal mask to absorb outlier-token attention, reducing hallucinations in MLLMs without training.
-
Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.
Discussion (0). Continue with ORCID to comment.