REVIEW 9 cited by
Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities for capturing and reasoning over multimodal inputs. However, these models are prone to parametric knowledge conflicts, which arise from inconsistencies of represented knowledge between their vision and language components. In this paper, we formally define the problem of $\textbf{cross-modality parametric knowledge conflict}$ and present a systematic approach to detect, interpret, and mitigate them. We introduce a pipeline that identifies conflicts between visual and textual answers, showing a persistently high conflict rate across modalities in recent LVLMs regardless of the model size. We further investigate how these conflicts interfere with the inference process and propose a contrastive metric to discern the conflicting samples from the others. Building on these insights, we develop a novel dynamic contrastive decoding method that removes undesirable logits inferred from the less confident modality components based on answer confidence. For models that do not provide logits, we also introduce two prompt-based strategies to mitigate the conflicts. Our methods achieve promising improvements in accuracy on both the ViQuAE and InfoSeek datasets. Specifically, using LLaVA-34B, our proposed dynamic contrastive decoding improves an average accuracy of 2.24%.
Forward citations
Cited by 9 Pith papers
-
ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Models
A new taxonomy and 1,500-item dataset, ENTRAP-VL, lets researchers measure whether vision-language models are entrained by textual and visual context separately.
-
Linguistic Context Recodes Visual Representations in Vision-Language Models
Goal-directed language prompts make VLMs add a transferable goal-relevant marker to selected image objects and amplify those objects' queried attributes in later layers, and both effects causally influence answers.
-
How Do Vision-Language Models Process Conflicting Information Across Modalities?
Vision-language models answer from whichever modality is encoded more saliently in their final-layer representations, and specific attention heads can be manipulated to shift that preference.
-
Is Extending Modality The Right Path Towards Omni-Modality?
Fine-tuning LLMs on extra modalities improves some knowledge tasks but degrades reasoning and instruction-following; weighted model merging preserves language ability better than training one model on all modalities.
-
mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation
A systematic empirical study finds that for multimodal RAG, EVA-CLIP retrieval, listwise LVLM reranking, and feeding only the top-ranked document works best, with a self-reflection agent adding further gains.
-
Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models
MMKC-Bench provides a human-verified benchmark of multimodal knowledge conflicts and shows that current LMMs prefer internal parametric knowledge over external evidence.
-
Robust Multimodal Large Language Models Against Modality Conflict
A new benchmark, MMMC, triggers hallucinations in all tested multimodal models, and reinforcement learning on it reduces such hallucinations more than prompt engineering or supervised fine-tuning.
-
MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Models
MCA-LLaVA reindexes image tokens by sums of mirrored 2D coordinates so instruction tokens attend across the whole image, reducing hallucination on POPE, CHAIR, and MME.
-
Foundation Models for Astrophysics
Astronomical 'foundation models' largely reuse transformers and self-supervised pretraining, but evidence of transfer to new instruments, populations, or tasks remains rare; the paper argues such evidence, not archite...
Discussion (0). Sign in to comment.