REVIEW 3 cited by
How Far Are We from Intelligent Visual Deductive Reasoning?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Vision-Language Models (VLMs) have recently demonstrated incredible strides on diverse vision language tasks. We dig into vision-based deductive reasoning, a more sophisticated but less explored realm, and find previously unexposed blindspots in the current SOTA VLMs. Specifically, we leverage Raven's Progressive Matrices (RPMs), to assess VLMs' abilities to perform multi-hop relational and deductive reasoning relying solely on visual clues. We perform comprehensive evaluations of several popular VLMs employing standard strategies such as in-context learning, self-consistency, and Chain-of-thoughts (CoT) on three diverse datasets, including the Mensa IQ test, IntelligenceTest, and RAVEN. The results reveal that despite the impressive capabilities of LLMs in text-based reasoning, we are still far from achieving comparable proficiency in visual deductive reasoning. We found that certain standard strategies that are effective when applied to LLMs do not seamlessly translate to the challenges presented by visual reasoning tasks. A detailed analysis reveals that VLMs struggle to solve these tasks mainly because they are unable to perceive and comprehend multiple, confounding abstract patterns in RPM examples.
Forward citations
Cited by 3 Pith papers
-
Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Tasks
A new contrastive real-image benchmark shows most vision-language models fail spatial relation tasks, while chain-of-thought reasoning models approach human-level accuracy.
-
Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Task
MultiStAR decomposes RAVEN abstract reasoning into five staged sub-tasks and MSEval scores partial progress, showing MLLMs handle basic perception but fail at rule deduction.
-
Disentangling Perception and Reasoning in Multimodal LLMs via Reward Design
On AlgoPuzzleVQA, visual perception—not algorithmic reasoning—is the main bottleneck: giving Claude models clean text representations of images improves accuracy by 23.6–26.7 points, while GRPO reward design on Qwen-2...
Discussion (0). Continue with ORCID to comment.