Pith. sign in

REVIEW 16 cited by

DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.00836 v2 pith:CPBAEWDT submitted 2024-10-29 cs.CV cs.AIcs.CL

Chengke Zou , Xingang Guo , Rui Yang , Junyu Zhang , Bin Hu , Huan Zhang This is my paper
classification cs.CVcs.AIcs.CL
keywords vlmsreasoningmathematicaldynamathvisualmodelsquestionsrobustness
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The rapid advancements in Vision-Language Models (VLMs) have shown great potential in tackling mathematical reasoning tasks that involve visual context. Unlike humans who can reliably apply solution steps to similar problems with minor modifications, we found that SOTA VLMs like GPT-4o can consistently fail in these scenarios, revealing limitations in their mathematical reasoning capabilities. In this paper, we investigate the mathematical reasoning robustness in VLMs and evaluate how well these models perform under different variants of the same question, such as changes in visual numerical values or function graphs. While several vision-based math benchmarks have been developed to assess VLMs' problem-solving capabilities, these benchmarks contain only static sets of problems and cannot easily evaluate mathematical reasoning robustness. To fill this gap, we introduce DynaMath, a dynamic visual math benchmark designed for in-depth assessment of VLMs. DynaMath includes 501 high-quality, multi-topic seed questions, each represented as a Python program. Those programs are carefully designed and annotated to enable the automatic generation of a much larger set of concrete questions, including many different types of visual and textual variations. DynaMath allows us to evaluate the generalization ability of VLMs, by assessing their performance under varying input conditions of a seed question. We evaluated 14 SOTA VLMs with 5,010 generated concrete questions. Our results show that the worst-case model accuracy, defined as the percentage of correctly answered seed questions in all 10 variants, is significantly lower than the average-case accuracy. Our analysis emphasizes the need to study the robustness of VLMs' reasoning abilities, and DynaMath provides valuable insights to guide the development of more reliable models for mathematical reasoning.

Discussion (0). Sign in to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Model Guides You How to Draw: Adaptive Visual Gating for Unified Multimodal Reasoning

    cs.CV 2026-07 conditional novelty 6.5 of 10

    Early Generation Intent and Visual Fidelity signals let AdaViG abort low-utility intermediate image generations in UMM math CoT, improving accuracy up to 5.7% and cutting visual FLOPs 25–91%.

  2. CodePercept: Code-Grounded Visual STEM Perception for MLLMs

    cs.CV 2026-03 conditional novelty 6.5 of 10

    Perception, not reasoning, is the main bottleneck for MLLM STEM visual reasoning, and training on executable reconstruction code measurably fixes it.

  3. MIDAL: A Dataset of Math Image Descriptions for Accessible Learning

    cs.CV 2026-08 unverdicted novelty 6.0 of 10

    MIDAL provides 2,020 described math images to train vision-language models for accessible math image descriptions and improved math reasoning.

  4. What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs

    cs.CL 2026-06 conditional novelty 6.0 of 10

    VLM accuracy can be predicted from a scalar capability score derived from LLM text benchmarks plus multimodal data volume via a fitted transfer-absorption scaling law.

  5. Kwai Keye-VL 1.5 Technical Report

    cs.CV 2025-09 conditional novelty 6.0 of 10

    Keye-VL-1.5 combines similarity-based Slow-Fast video token allocation with progressive context extension and iterative RL, reporting leading video-understanding results among 8B-scale multimodal models.

  6. R-4B: Incentivizing General-Purpose Auto-Thinking Capability in MLLMs via Bi-Mode Annealing and Reinforce Learning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    R-4B uses bi-mode annealing plus a GRPO variant with forced thinking and non-thinking rollouts so a 4B multimodal model adaptively picks when to reason, reportedly matching 16B thinking models on several reasoning benchmarks.

  7. GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoning

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A generative multimodal process reward model that produces step-level critiques and corrections improves average math accuracy for six multimodal LLMs by 2.9 to 5.9 points under a refinement-based Best-of-N strategy.

  8. Position: Reasoning After Perception Means Reasoning Without Vision

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Reasoning in text space cannot recover visual information that was collapsed during perception, so multimodal models need architectures that reason within the visual representation.

  9. BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset

    cs.CL 2025-07 conditional novelty 6.0 of 10

    BMMR provides a 110k-question bilingual, multimodal, college-level dataset across 300 subjects where state-of-the-art models score at most about 50%.

  10. Kwai Keye-VL Technical Report

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Kwai Keye-VL shows that a five-mode chain-of-thought cold-start plus mix-mode reinforcement learning can push an 8B multimodal model to strong short-video and general vision-language performance.

  11. MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI

    cs.AI 2025-06 conditional novelty 6.0 of 10

    MMReason is an open-ended multimodal reasoning benchmark that filters out guessable and memorized questions and scores model answers both by final answer and by intermediate steps.

  12. TimeCausality: Evaluating the Causal Ability in Time Dimension for Vision Language Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Introduces a 700-pair benchmark for temporal causal reasoning in VLMs, revealing large open-source vs. closed-source gaps and strong position bias.

  13. Can Vision-Language Models Solve Visual Math Equations?

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Vision-language models solve text equations but fail on picture equations because they cannot reliably count repeated icons; counting, not recognition or reasoning, is the limiting step.

  14. MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Models

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    A new benchmark of 2,000 K-12 math questions photographed in real scenes finds that multimodal LLMs perform significantly worse on authentic handheld images than on clean benchmarks.

  15. WeThink: Toward General-purpose Vision-Language Reasoning via Reinforcement Learning

    cs.CV 2025-06 conditional novelty 5.0 of 10

    WeThink, a 120K-image QA dataset with AI-generated reasoning paths, combined with hybrid-reward reinforcement learning, improves a 7B vision-language model across 14 benchmarks.

  16. MiMo-VL Technical Report

    cs.CL 2025-06 conditional novelty 5.0 of 10

    MiMo-VL-7B-RL, a 7B open-source vision-language model, reports state-of-the-art results on 35 of 40 benchmarks and a 59.4 OlympiadBench score, with the report crediting long-CoT pretraining data and mixed on-policy RL.

Pith tools