REVIEW 5 cited by
ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Despite the recent breakthroughs achieved by Large Vision Language Models (LVLMs) in understanding and responding to complex visual-textual contexts, their inherent hallucination tendencies limit their practical application in real-world scenarios that demand high levels of precision. Existing methods typically either fine-tune the LVLMs using additional data, which incurs extra costs in manual annotation and computational resources or perform comparisons at the decoding stage, which may eliminate useful language priors for reasoning while introducing inference time overhead. Therefore, we propose ICT, a lightweight, training-free method that calculates an intervention direction to shift the model's focus towards different levels of visual information, enhancing its attention to high-level and fine-grained visual details. During the forward pass stage, the intervention is applied to the attention heads that encode the overall image information and the fine-grained object details, effectively mitigating the phenomenon of overly language priors, and thereby alleviating hallucinations. Extensive experiments demonstrate that ICT achieves strong performance with a small amount of data and generalizes well across different datasets and models. Our code will be public.
Forward citations
Cited by 5 Pith papers
-
Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability
Large multimodal models mostly fail to proactively detect flawed textual premises, and their performance depends on error type and on how they weight text versus images.
-
Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
Modality bias, an imbalanced attention to text or image during hallucinated outputs, is shown to be mitigated by a training-free attention intervention plus contrastive decoding.
-
GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs
GrAInS uses Integrated Gradients to identify the most influential tokens, then builds layer-wise steering vectors that improve truthfulness, reduce hallucination, and preserve general capabilities in LLMs and VLMs.
-
CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention
An inference-time attention-shift intervention aligns non-English queries' cross-modal attention with English, cutting multilingual object hallucination in LVLMs on POPE and MME.
-
Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations
VaLSe uses attention-based visual contribution maps to steer an LVLM's latent features toward visually grounded content, reducing object hallucinations on several benchmarks while exposing flaws in CHAIR-style evaluation.
Discussion (0). Continue with ORCID to comment.