REVIEW 10 cited by
VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Current multimodal benchmarks often conflate reasoning with domain-specific knowledge, making it difficult to isolate and evaluate general reasoning abilities in non-expert settings. To address this, we introduce VisualPuzzles, a benchmark that targets visual reasoning while deliberately minimizing reliance on specialized knowledge. VisualPuzzles consists of diverse questions spanning five categories: algorithmic, analogical, deductive, inductive, and spatial reasoning. One major source of our questions is manually translated logical reasoning questions from the Chinese Civil Service Examination. Experiments show that VisualPuzzles requires significantly less intensive domain-specific knowledge and more complex reasoning compared to benchmarks like MMMU, enabling us to better evaluate genuine multimodal reasoning. Evaluations show that state-of-the-art multimodal large language models consistently lag behind human performance on VisualPuzzles, and that strong performance on knowledge-intensive benchmarks does not necessarily translate to success on reasoning-focused, knowledge-light tasks. Additionally, reasoning enhancements such as scaling up inference compute (with "thinking" modes) yield inconsistent gains across models and task types, and we observe no clear correlation between model size and performance. We also found that models exhibit different reasoning and answering patterns on VisualPuzzles compared to benchmarks with heavier emphasis on knowledge. VisualPuzzles offers a clearer lens through which to evaluate reasoning capabilities beyond factual recall and domain knowledge.
Forward citations
Cited by 10 Pith papers
-
LUT: Latent Utility Training for Visual Reasoning
LUT trains visual latent reasoning from plain VQA data by selecting teacher trajectories with an answer-information-gain score and reweighting latent steps via attention attribution during RL.
-
Beacon: Knowing When and How to Perform Agentic Visual Reasoning
Beacon improves agentic visual reasoning by teaching models when tools are necessary and how to use them for net gains, via necessity-aware rewards and hint-guided RL.
-
Trace: A Taxonomy-Guided Environment for Multidomain Visual Reasoning
RLVR training on 64,000 procedurally generated Trace instances improves Qwen2.5-VL macro-average on 24 external visual reasoning benchmarks by 3.51 points at 3B and 4.06 points at 7B.
-
PACE: A Proxy for Agentic Capability Evaluation
PACE builds proxy benchmarks from non-agentic instances via relevance and global selection plus regression to predict agentic scores with MAE under 4%, Spearman correlation above 0.80, and 85% ranking accuracy at unde...
-
MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs
MapTab is a new multimodal benchmark with 328 images and nearly 200k queries that shows current MLLMs have substantial difficulty with multi-criteria route planning when visual and tabular information must be combined.
-
VisualSphinx: Large-Scale Synthetic Vision Logic Puzzles for RL
LLM-derived rules, genetic expansion, and program-drawn images produce 660K visual logic puzzles; GRPO training on them lifts a VLM's accuracy on those puzzles and modestly on MathVista.
-
MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs
A new 1,188-question multimodal benchmark covering deductive, inductive, and abductive reasoning shows that leading MLLMs score around 60% and are especially weak at abductive reasoning.
-
Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Models
Current vision-language models fall far short of humans on spatial reasoning, especially when they must generate answers directly instead of choosing from options.
-
Limits of Spatial Imagery Reasoning in Frontier LLM Models
An external 3D imagery tool does not fix frontier MLLM mental-rotation performance (≤62.5%), revealing missing visual-spatial primitives rather than a pure working-memory limit.
-
PyVision: Agentic Vision with Dynamic Tooling
Giving multimodal LLMs a loop in which they generate, execute, and refine Python code improves their performance on visual reasoning benchmarks.
Discussion (0). Sign in to comment.