REVIEW 14 cited by
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Is vision good enough for language? Recent advancements in multimodal models primarily stem from the powerful reasoning abilities of large language models (LLMs). However, the visual component typically depends only on the instance-level contrastive language-image pre-training (CLIP). Our research reveals that the visual capabilities in recent multimodal LLMs (MLLMs) still exhibit systematic shortcomings. To understand the roots of these errors, we explore the gap between the visual embedding space of CLIP and vision-only self-supervised learning. We identify ''CLIP-blind pairs'' - images that CLIP perceives as similar despite their clear visual differences. With these pairs, we construct the Multimodal Visual Patterns (MMVP) benchmark. MMVP exposes areas where state-of-the-art systems, including GPT-4V, struggle with straightforward questions across nine basic visual patterns, often providing incorrect answers and hallucinated explanations. We further evaluate various CLIP-based vision-and-language models and found a notable correlation between visual patterns that challenge CLIP models and those problematic for multimodal LLMs. As an initial effort to address these issues, we propose a Mixture of Features (MoF) approach, demonstrating that integrating vision self-supervised learning features with MLLMs can significantly enhance their visual grounding capabilities. Together, our research suggests visual representation learning remains an open challenge, and accurate visual grounding is crucial for future successful multimodal systems.
Forward citations
Cited by 14 Pith papers
-
Revisiting Model Stitching In the Foundation Model Era
Final-feature matching makes heterogeneous VFMs reliably stitchable; stitched models fuse complementary strengths and enable efficient multi-VFM Stitch Trees for multimodal LLMs.
-
Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulation
BadSem shows that semantic mismatches between images and text can serve as stealthy backdoor triggers for VLMs, achieving near-perfect attack success with low poisoning rates.
-
The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping
Trace-grounded parametric profiling of three synthetic counting tasks shows current video-language models only count reliably at low event counts and low rates, and final-answer accuracy masks poor timestamp-level eve...
-
VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation
Counterfactual present/removed teacher views attribute visually supported corrections and reconstruct student-anchored distillation targets that beat source-mixed multimodal OPD.
-
DELTAVID: Enhancing Fine-Grained Spatiotemporal Perception with Cross-Video Differences
Rule-reward training on controllable cross-video differences (Grounding + MCQ) improves Video MLLM local spatiotemporal evidence localization and transfers to general video QA benchmarks.
-
Cognitive Pivot Points and Visual Anchoring: Unveiling and Rectifying Hallucinations in Multimodal Reasoning Models
Multimodal reasoning models hallucinate at high-entropy cognitive bifurcation points due to loss of visual semantic anchoring, and the V-STAR training paradigm with HVAR rewards and FRM reflection mitigates this by re...
-
Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Tasks
A new contrastive real-image benchmark shows most vision-language models fail spatial relation tasks, while chain-of-thought reasoning models approach human-level accuracy.
-
LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model
RL training on preference-labeled critic data transforms a 7B vision-language model into both a stronger critic and a stronger generative policy, improving average benchmark accuracy by 5.7% and enabling self-critique...
-
A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarks
Blind text-only likelihood models match or exceed CLIP on many compositionality benchmarks because positives and negatives differ systematically in length, plausibility, or image style.
-
Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations
STARE is a 4K-task benchmark showing multimodal LLMs perform near random chance on multi-step spatial simulation tasks such as cube net folding and tangrams, despite strong 2D transformation results.
-
Humanoid World Models: Open World Foundation Models for Humanoid Robotics
Masked-transformers trained on humanoid video forecast future frames with better FID than flow-matching models, and parameter sharing cut model size 33-53% with minimal quality loss.
-
AMIA: Automatic Masking and Joint Intention Analysis Makes LVLMs Robust Jailbreak Defenders
AMIA is an inference-only jailbreak defense that masks text-irrelevant image patches and prompts single-pass intention analysis, lifting average defense success on LVLMs from 52.4% to 81.7%.
-
Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations
VaLSe uses attention-based visual contribution maps to steer an LVLM's latent features toward visually grounded content, reducing object hallucinations on several benchmarks while exposing flaws in CHAIR-style evaluation.
-
MoDA: Modulation Adapter for Fine-Grained Visual Grounding in Instructional MLLMs
An instruction-conditioned channel gate on pre-aligned visual tokens improves grounding in LLaVA-style models on most reported benchmarks, but the mechanism is undercut by the paper's own ablation and the abstract ove...
Discussion (0). Continue with ORCID to comment.