REVIEW 23 cited by
Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The ability to organically reason over and with both text and images is a pillar of human intelligence, yet the ability of Multimodal Large Language Models (MLLMs) to perform such multimodal reasoning remains under-explored. Existing benchmarks often emphasize text-dominant reasoning or rely on shallow visual cues, failing to adequately assess integrated visual and textual reasoning. We introduce EMMA (Enhanced MultiModal reAsoning), a benchmark targeting organic multimodal reasoning across mathematics, physics, chemistry, and coding. EMMA tasks demand advanced cross-modal reasoning that cannot be addressed by reasoning independently in each modality, offering an enhanced test suite for MLLMs' reasoning capabilities. Our evaluation of state-of-the-art MLLMs on EMMA reveals significant limitations in handling complex multimodal and multi-step reasoning tasks, even with advanced techniques like Chain-of-Thought prompting and test-time compute scaling underperforming. These findings underscore the need for improved multimodal architectures and training paradigms to close the gap between human and model reasoning in multimodality.
Forward citations
Cited by 23 Pith papers
-
Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuning
Point-RFT uses point-grounded chain-of-thought with GRPO reinforcement to improve chart reasoning, reporting 90.04% on ChartQA, though internal tables and the OOD setup weaken the claim.
-
Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems
A new spoken math benchmark, Spoken-MQA, shows that current speech-based AI models reason poorly from spoken math input, especially for arithmetic and knowledge-heavy problems.
-
PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
Frontier MLLMs remain far from mastering atomic visual perception: none reach 60% on a failure-derived, perception-only benchmark of ten capabilities.
-
SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopy
A new benchmark of 2,703 automatically generated multimodal questions for scanning probe microscopy, plus a modified F1 metric that penalizes over-selection and labels model 'personalities'.
-
MentisOculi: Revealing the Limits of Reasoning with Mental Imagery
Visual thoughts — latent tokens, interleaved images, or video rollouts — do not currently improve multi-step reasoning over text-only baselines in frontier models.
-
LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model
RL training on preference-labeled critic data transforms a 7B vision-language model into both a stronger critic and a stronger generative policy, improving average benchmark accuracy by 5.7% and enabling self-critique...
-
KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contexts
KRETA, a 2,577-item Korean text-rich VQA benchmark, shows vision-language models recognize Korean text well but lag in multi-step reasoning, especially in open-source models.
-
SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models
SEAM measures VLM reasoning consistency across modalities using paired semantically equivalent textual and visual notations, and finds systematic vision-language imbalance.
-
VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoning
VL-Cogito, trained with progressive curriculum RL, online difficulty weighting, and dynamic length rewards, matches or beats prior reasoning MLLMs on ten multimodal benchmarks.
-
MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning
State-of-the-art multimodal language models perform at or near random chance on MARBLE, a new hard benchmark for spatial reasoning and planning.
-
ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs
A verifiable RL proxy task that asks VLMs to locate a single injected hallucination in a 200-word caption improves visual perception and transfers to math and abstract reasoning benchmarks.
-
Seeing is Not Reasoning: MVPBench for Graph-based Evaluation of Multi-path Visual Physical CoT
A new multi-image benchmark and graph-based scoring method show that MLLMs produce weak, poorly-grounded chains of thought on visual physics tasks, and that RL post-training can degrade spatial reasoning.
-
MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs
A new 1,188-question multimodal benchmark covering deductive, inductive, and abductive reasoning shows that leading MLLMs score around 60% and are especially weak at abductive reasoning.
-
Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model
RL-trained VLMs generalize compositionally far better than SFT-trained ones on synthetic geometry and spatial tasks, but cross-modal combination remains weak, and a caption-before-thinking plus progress-reward recipe ...
-
FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow
FullFront adds a three-task benchmark for webpage design, perception, and code generation, and finds top MLLMs still fail at fine-grained layout and interaction implementation.
-
PhyX: Does Your Model Have the "Wits" for Physical Reasoning?
PhyX is a new 3,000-question visual physics benchmark; the best AI model tested scores 45.8 percent, well below the 75.6 to 78.9 percent of a small human student sample.
-
lmgame-Bench: How Good are LLMs at Playing Games?
lmgame-Bench turns six classic games into a scaffolded LLM evaluation suite, ranks 13 models, detects contamination, and reports RL transfer from Sokoban or Tetris to unseen games and planning tasks.
-
ProLaViT: Learning Progressive Latent Visual Thoughts in Structured Latent Space
Progressive multi-step latent visual thoughts, endogenously distilled from a model's own encoder on synthetic trajectories and regularized by distance-weighted diversity, improve MLLM visual reasoning accuracy and efficiency.
-
Skywork-R1V3 Technical Report
A 38B open-source VLM reaches 76.0% on MMMU using RL post-training and connector-only tuning, with a critical-token entropy metric for checkpoint selection.
-
Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models
Vision-language models improve little, often not at all, when given demonstrations, even when demonstrations contain explicit reasoning steps.
-
CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning Capabilities of VLMs
The paper releases CSVQA, a Chinese multimodal STEM benchmark of 1,378 questions with human explanations, and reports that the best tested VLM, o1, reaches only 49.6% accuracy.
-
Wait, We Don't Need to "Wait"! Removing Thinking Tokens Improves Reasoning Efficiency
Suppressing "Wait"-like reflection tokens at decode time reduces reasoning token counts by 27-51% across five R1-style model families, with mixed accuracy effects.
-
Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Models
A survey-style position paper claims that reinforcement fine-tuning powers reasoning in multimodal LLMs, summarizing over a hundred recent works and proposing five future research directions.
Discussion (0). Sign in to comment.