Pith. sign in

REVIEW 8 cited by

Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.04780 v2 pith:QQIO2SAV submitted 2024-10-07 cs.CV

Guanyu Zhou , Yibo Yan , Xin Zou , Kun Wang , Aiwei Liu , Xuming Hu This is my paper · ORCID
classification cs.CV
keywords attentionlanguagemultimodalbiasescausallargemodalitypriors
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Multimodal Large Language Models (MLLMs) have emerged as a central focus in both industry and academia, but often suffer from biases introduced by visual and language priors, which can lead to multimodal hallucination. These biases arise from the visual encoder and the Large Language Model (LLM) backbone, affecting the attention mechanism responsible for aligning multimodal inputs. Existing decoding-based mitigation methods focus on statistical correlations and overlook the causal relationships between attention mechanisms and model output, limiting their effectiveness in addressing these biases. To tackle this issue, we propose a causal inference framework termed CausalMM that applies structural causal modeling to MLLMs, treating modality priors as a confounder between attention mechanisms and output. Specifically, by employing backdoor adjustment and counterfactual reasoning at both the visual and language attention levels, our method mitigates the negative effects of modality priors and enhances the alignment of MLLM's inputs and outputs, with a maximum score improvement of 65.3% on 6 VLind-Bench indicators and 164 points on MME Benchmark compared to conventional methods. Extensive experiments validate the effectiveness of our approach while being a plug-and-play solution. Our code is available at: https://github.com/The-Martyr/CausalMM

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    Large multimodal models mostly fail to proactively detect flawed textual premises, and their performance depends on error type and on how they weight text versus images.

  2. IKOD: Mitigating Visual Attention Degradation in Large Vision-Language Models

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    IKOD reduces hallucination in vision-language models by merging KV states to derive image-focused shorter-sequence logits and combining them with normal decoding, without training.

  3. Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    EVRB is a three-part inference-time method that prunes ambiguous visual tokens, divides the model's output distribution by a text-only prior, and triggers early stopping to reduce hallucination in LVLMs.

  4. Pierce the Mists, Greet the Sky: Decipher Knowledge Overshadowing via Knowledge Circuit Analysis

    cs.CL 2025-05 conditional novelty 6.0 of 10

    PhantomCircuit traces knowledge overshadowing to attention circuits during training and prunes circuit edges to recover the overshadowed answer.

  5. Disentangling Semantic Attention from Structural Bias in the Attention Manifold

    cs.CV 2026-07 conditional novelty 5.0 of 10

    SPAR removes a query-averaged structural bias from text-to-image attention and redistributes the reclaimed probability mass, reducing reported object and induced hallucinations in LLaVA models.

  6. CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    CAI reduces object hallucination in LVLMs by injecting caption-query attention patterns into selected attention heads at inference time.

  7. CAFES: A Collaborative Multi-Agent Framework for Multi-Granular Multimodal Essay Scoring

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A student-teacher multi-agent pipeline with positive-only feedback improves QWK agreement with human essay scores by 21% on a multimodal benchmark, with gains concentrated in traits where baselines were weakest.

  8. NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR

    eess.AS 2026-04 unverdicted novelty 4.0 of 10

    NIM4-ASR delivers SOTA ASR performance on public benchmarks using a 2.3B-parameter LLM with multi-stage training, real-time streaming, and million-scale hotword customization via RAG.

Pith tools