Pith. sign in

REVIEW 4 major objections 3 minor 1 cited by

MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI

T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MMReason, a new open-ended multimodal benchmark, claims existing MLLMs score far lower on multi-step reasoning than MCQ tests suggest, with the best model, GPT-4o, at 25.7%.

desk verdict The filtering story is solid and the benchmark deserves refereeing, but the GPT-4o step-scoring is unvalidated and the intermediate-step numbers are not yet trustworthy. read the letter →

arxiv 2506.23563 v1 pith:OBTD72IR submitted 2025-06-30 cs.AI cs.CLcs.CV

Huanjin Yao , Jiaxing Huang , Yawen Qiu , Michael K. Chen , Wenzheng Liu , Wei Zhang , Wenjie Zeng , Xikun Zhang
show 4 more authors
Jingyi Zhang Yuxin Song Wenhao Wu Dacheng Tao
This is my paper
classification cs.AIcs.CLcs.CV
keywords multimodallargelanguagemodelsmulti-stepreasoningbenchmarkopen-endedquestionsguessabilitymemorizationintermediatestepevaluationternaryscoring
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MMReason is a benchmark built to measure whether multimodal large language models can actually reason through multi-step problems, rather than guess or memorize. The paper argues that current multimodal benchmarks, dominated by multiple-choice questions, let models arrive at correct answers without valid reasoning, either by guessing among options or by recalling leaked training data. MMReason removes answer choices, filters out questions that any of four strong models can answer from text alone, and scores each intermediate reasoning step against a human-annotated reference solution. On the resulting 1,384 questions, the best evaluated model, GPT-4o, reaches only 25.7% final-answer accuracy, suggesting that state-of-the-art multimodal reasoning is much weaker than MCQ leaderboards imply. The paper's case matters because if true, progress toward AGI needs to be measured by step-level reasoning on open-ended multimodal problems, not by option-picking accuracy.

What carries the argument

The load-bearing mechanism is the combination of two filters and one grader. First, multiple-choice questions with unique answers are rewritten in open-ended form to block guessing. Second, a multi-model voting procedure runs four strong MLLMs on text-only versions of each question for two rounds, deleting any question answered correctly without the image; this targets memorization and enforces visual relevance. Third, GPT-4o decomposes each model response into steps and scores each step as correct (1), unverifiable (0.5), or incorrect (0), using the annotated reference solution, then averages the step scores. The 'unverifiable' category is what makes the grader usable when a response follows a valid alternative solution that the reference does not cover.

What would settle it

Run a human agreement study on a random sample of MMReason responses: have several human raters assign the same correct/unverifiable/incorrect labels to each step using the same reference solutions, and compare the averaged intermediate scores with GPT-4o's labels. If human raters frequently disagree with the model judge, or if the ranking of models by step score reverses under human grading, the benchmark's central claim that it precisely and reliably evaluates intermediate reasoning is not supported.

Watch

Extended reading notes

Core claim

The central discovery is that a carefully filtered set of 1,384 open-ended multimodal questions, spanning six disciplines and several difficulty tiers, is sufficient to expose a large gap between appearance and reality in MLLM reasoning. After converting unique-answer multiple-choice items to open-ended format and removing text-solvable or memorized instances via iterative multi-model voting, the authors find that no model answers more than a quarter of questions correctly: GPT-4o leads with 25.7%, followed by Claude-3.7V Sonnet at 25.1% and Gemini-1.5 Pro at 24.9%. They also report intermediate-step scores from a reference-based ternary grader, with GPT-4o scoring 42.1% on steps, and an error analysis attributing most of GPT-4o's failures to reasoning and perception errors. The paper reads these results as evidence that current MLLMs have not yet acquired robust long-chain multimodal reasoning.

Load-bearing premise

The benchmark's intermediate-step scores all depend on GPT-4o being a trustworthy step extractor and grader; if GPT-4o's step labels are biased, the claimed advantage over final-answer-only evaluation collapses.

Editorial extensions

If this is right

  • If MMReason measures what it claims, then published MCQ-based accuracy numbers for MLLMs overstate their genuine multi-step reasoning ability.
  • The 25.7% ceiling for GPT-4o provides a concrete reference point for future model development on open-ended multimodal reasoning.
  • The discipline-wise breakdown shows engineering questions are the hardest across nearly all models, while math and health questions are relatively easier.
  • The filtering results, with text-only accuracy dropping from about 6-13% before filtering to under 1% after filtering, indicate the retained benchmark is genuinely vision-dependent.
  • The intermediate-step scores being higher than final-answer accuracy for closed-source models suggests these models often reason plausibly but fail at the final step or answer extraction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the filtering step removes every question that any of the four chosen models can answer from text, the benchmark's difficulty is partially a moving target; as those models improve, the retained set may shift and require periodic re-filtering.
  • Editorial inference: the three-level scoring could be extended to a 'partially correct' scale or to multiple reference solutions, which would make the intermediate-step scores more informative for training and error analysis.
  • Editorial inference: a direct testable extension is to use MMReason's step scores as a training signal or as a diagnostic to distinguish perception failures from reasoning failures, which the paper's small error analysis suggests is feasible.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper introduces MMReason, a multimodal multi-step reasoning benchmark consisting of 1,384 open-ended questions across six disciplines, with questions reformulated from existing benchmarks and newly collected from the web. The construction pipeline converts multiple-choice questions into open-ended formats, filters out memorized or visually irrelevant items using a multi-model text-only voting procedure (K=4 models, T=2 rounds), and annotates questions with step-by-step reference solutions. The evaluation uses GPT-4o both to extract final answers and to segment responses into steps, assigning ternary labels (correct/unverifiable/incorrect) that are averaged into an intermediate-step score. The main empirical claim is that state-of-the-art MLLMs perform poorly on this benchmark, with the best model (GPT-4o) achieving only 25.7% final-answer accuracy.

Significance. If the construction and evaluation methods are valid, MMReason would be a valuable benchmark: the filtering evidence in Table 5 is the strongest part, showing that text-only accuracy drops to 0.65-0.78% while visual accuracy remains 11.7-25.7% and the multimodal relevance rate rises above 94%, which concretely demonstrates that the retained questions require visual input. The benchmark also covers a broader disciplinary and difficulty range than many existing multimodal reasoning benchmarks. However, the novelty claims regarding intermediate-step evaluation are not yet supported because the GPT-4o-based ternary scoring mechanism lacks validation; the final-answer accuracy results are plausible and interesting, but the paper's stated advantage over final-answer-only benchmarks depends on the unvalidated grader.

major comments (4)
  1. [Section 4.1 and 4.3] The paper does not specify how many of the 1,384 benchmark questions have reference step-by-step solutions. Section 4.1 states that 'a subset of questions from our newly collected data' is manually annotated, which implies that not all questions in MMReason have reference solutions. If the intermediate-step scores in Table 3 are computed only on that subset, the paper must report the subset size and confirm that all models are scored on exactly the same questions; otherwise the intermediate-score column is not comparable with the final-answer column and the model rankings may be inconsistent across metrics.
  2. [Section 4.3] The reference-based ternary scoring mechanism rests entirely on GPT-4o for both segmenting responses into steps and assigning correct/unverifiable/incorrect labels. The paper provides no human agreement study, no second judge, no disclosed size of any validation subset, and no distribution or error analysis of the ternary labels. This is load-bearing because the word 'precisely' in the central claim and the entire intermediate-step column of Table 3 depend on the reliability of this grader. The concern is compounded by the fact that GPT-4o is also the top-scoring model on the benchmark, so a grader bias toward GPT-4o's response style could inflate its intermediate score; a second judge or a human-annotated subset would be needed to rule this out.
  3. [Section 4.3, Eq. (1)] The intermediate score S_inter = (1/N) sum Score(s_n) is not independent of how GPT-4o chooses to segment a response. Two equally valid segmentations of the same solution can yield different values of N and different step-level labels, so the metric is not well-defined unless segmentation stability is reported. In addition, the 'unverifiable' category is always scored 0.5, which can reward a model for writing an uncheckable step rather than a correct one; the paper gives no evidence that this neutral category is applied consistently across responses or across models.
  4. [Figure 3] The qualitative example in Figure 3 raises concerns about grader leniency: the wrong response, whose Step 4 performs an incorrect substitution (using 115,200 instead of 58,500) and which yields a wrong final answer, nevertheless receives an intermediate score of 0.7. The paper does not discuss how such partial-credit cases are calibrated, how often they occur, or whether the threshold for a 'passing' intermediate score is meaningful. An error analysis of the grader itself—not just of the evaluated models—is necessary to interpret the intermediate-step scores in Table 3.
minor comments (3)
  1. [Section 1 and Table 3] Section 1 reports GPT-4o's intermediate-step score as 'xx.x%', but Table 3 lists 42.1; this unresolved placeholder must be corrected.
  2. [Section 5.2.1 and Table 3] There are typos in model names: 'Calude-3.7V Soonet' should be 'Claude-3.7V Sonnet', and Table 3's 'LLaV A-OneVision' and 'LLaV A-CoT' should be 'LLaVA-OneVision' and 'LLaVA-CoT'.
  3. [Section 5.3 and Figure 4] The error analysis in Figure 4 is based on only 50 randomly selected incorrect responses from GPT-4o; the paper should state the sampling procedure and the confidence interval or stability of the reported distribution, especially because 'Reasoning Error' and 'Perceptual Error' are close in proportion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; MMReason's results are empirical benchmark measurements rather than derivations from fitted inputs.

full rationale

MMReason is a benchmark-construction and evaluation paper, so the derivation-chain circularity failure mode largely does not apply: Table 3 reports measured final-answer accuracies and intermediate-step scores, not quantities derived from fitted parameters. The multi-model voting filter (Algorithm 1) removes questions that are answerable without images, and Table 5's post-filter text-only accuracy is a transparent consequence of that construction rather than a hidden prediction, and the paper does not present it as an emergent result. The reference-based ternary scoring in Section 4.3 uses GPT-4o as a grader, which creates an evaluation-reliability concern (no human agreement study is reported), but it is not circular: Equation (1) averages independently produced step scores, and no output of the scoring is an input to the benchmark's construction. Citations to the authors' prior work (e.g., Mulberry, R1-VL) are related-work references and are not load-bearing premises for the benchmark's validity. Therefore no specific reduction of a claimed result to its own inputs can be exhibited.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No natural-science free parameters are present. The tuning choices K=4, T=2, removal threshold >0, and the 1/0.5/0 scoring weights are hand-selected and unreported in sensitivity analysis; they directly shape the benchmark. The axioms are the unvalidated identification claims: text-only success equals memorization, and GPT-4o step labels equal human judgments.

free parameters (4)
  • Filtering model count K = 4
    Section 5.1 chooses four MLLMs for the voting filter; no sensitivity analysis explores a different K, so the final 1,384 questions depend on this hand-selected set.
  • Voting rounds T = 2
    Algorithm 1 and Section 5.1 set T=2; the number of removed questions would likely change with T.
  • Removal threshold = 0 (remove if any model answers text-only version correctly)
    Algorithm 1 removes a question if Count(qj) > 0, meaning if any filtering model answers it from text alone; an alternative threshold or a human audit could retain different questions.
  • Ternary step score weights = 1, 0.5, 0 for correct, unverifiable, incorrect steps
    Section 4.3 assigns these weights to step labels; equal weighting is arbitrary and is not validated against human grading.
assumptions (4)
  • domain assumption If a filtering model answers the text-only version of a question correctly, the question is memorized or visually irrelevant.
    Invoked in Algorithm 1 and Section 4.2; a legitimate text-based reasoning problem could be removed because strong language-only reasoning suffices, so the filter may select for questions that are hard from text rather than genuinely multimodal.
  • domain assumption GPT-4o, with access to reference solutions, reliably decomposes responses and assigns ternary step scores.
    Section 4.3 defines intermediate scoring entirely through GPT-4o; no human agreement or calibration study is reported, so the central 'reliable intermediate reasoning' claim depends on an unvalidated judge.
  • domain assumption GPT-4o reliably extracts final answers from free-form responses.
    Section 4.3 uses GPT-4o to extract and judge final answers; no extraction accuracy or human verification is reported.
  • domain assumption Reformulating uniquely answerable multiple-choice questions into open-ended form removes guessing shortcuts without changing difficulty.
    Section 4.1; wording changes could alter difficulty or introduce ambiguity, and no post-reformulation human audit is described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI." pith.science (2026). https://pith.science/paper/OBTD72IR

@misc{pith2026250623563,
  author       = {Pith},
  title        = {Pith review of: MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OBTD72IR}},
  note         = {Machine review of arXiv:2506.23563}
}
read the original abstract

Reasoning plays a crucial role in advancing Multimodal Large Language Models (MLLMs) toward Artificial General Intelligence. However, existing MLLM benchmarks often fall short in precisely and comprehensively evaluating long-chain reasoning abilities from three key aspects: (1) lack of difficulty and diversity, (2) susceptibility to guessability and memorization, (3) inadequate assessment of intermediate reasoning steps. To fill this gap, we introduce MMReason, a new benchmark designed to precisely and comprehensively evaluate MLLM long-chain reasoning capability with diverse, open-ended, challenging questions. First, we curate challenging questions requiring multi-step reasoning from various fields (i.e., 6 disciplines) and multiple difficulty levels (i.e., from pre-university to university, and from foundational to competition tiers). Second, these questions are reformulated into an open-ended format and filtered using a multi-model voting technique to eliminate shortcut cases related to guessing and memorization, ensuring robust reasoning evaluations. Third, we annotate the questions with detailed step-by-step solutions, and design a reference-based ternary scoring mechanism to reliably assess intermediate reasoning steps. With MMReason, we benchmark popular leading MLLMs and provide an in-depth analysis of their reasoning capabilities. We hope MMReason will serve as a valuable resource for advancing MLLM reasoning research. Code will be available at https://github.com/HJYao00/MMReason.

Figures

Figures reproduced from arXiv: 2506.23563 by the authors.

Figure 1
Figure 1. Existing benchmarks are susceptible to (a) guessability, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Examples of Guessability (Left) and Memorization (Right) issues. In the left figure, the model arrives at an incorrect numerical answer through multi-step reasoning but ultimately guesses the correct option by chance. In the right figure, we remove the choices from the original question before inputting it into MLLMs, yet the model still generates an answer choice, indicating the presence of memorization issues in e… view at source ↗
Figure 3
Figure 3. Qualitative analysis. MPO-8B&78B [32], Qwen2.5-VL-7B&72B[2]. We use VLMEvalKit [6] to evaluate these models on MMReason. 5.2. Experimental Results 5.2.1. Main Results In this section, we comprehensively compare the reasoning capabilities of various MLLMs in both intermediate steps and final answers on MMReason [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Error distribution over 50 incorrect responses from GPT [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results of incorrect reasoning. ure indicate that the model misunderstands “Capitalized in￾terests”, leading to incorrect reasoning. This demonstrates that MMReason can precisely evaluate the model’s reason￾ing ability. For intermediate step evaluation, the…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

51 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425– 2433, 2015. 3

    2015

  2. [2]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao- hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Jun- yang Lin. Qwen2.5-vl technical repor...

    arXiv 2025

  3. [3]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330,

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330,

    arXiv

  4. [4]

    R1-v: Reinforcing super generalization ability in vision- language models with less than $3

    Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in vision- language models with less than $3. https://github. com/Deep-Agent/R1-V , 2025. Accessed: 2025-02-02. 3

    2025

  5. [5]

    M 3cot: A novel benchmark for multi- domain multi-step multi-modal chain-of-thought

    Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M 3cot: A novel benchmark for multi- domain multi-step multi-modal chain-of-thought. In Proc. of ACL, 2024. 2, 3

    work page 2024

  6. [6]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia , pages 11198–11201, 2024. 7

    2024

  7. [7]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  8. [8]

    Mme: A compre- hensive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A compre- hensive evaluation benchmark for multimodal large language models, 2024. 2, 3

    work page 2024

Show all 51 references
  1. [9]

    Hallusionbench: an advanced diagnos- tic suite for entangled language hallucination and visual il- lusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnos- tic suite for entangled language hallucination and visual il- lusion in large vision-language models. In P...

    2024

  2. [10]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 1, 3

    2025 arXiv

  3. [11]

    Olympiadbench: A chal- lenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A chal- lenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:240...

    2024 arXiv

  4. [12]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749 ,

    arXiv

  5. [13]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 3

    2019

  6. [14]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 3, 6, 7, 8

    2024 arXiv

  7. [15]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richard- son, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. 1, 3

    2024 arXiv

  8. [16]

    Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency

    Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, et al. Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency. arXiv preprint arXiv:2502.09621, 2025. 3

    2025 arXiv

  9. [17]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 3, 6

    2024 arXiv

  10. [18]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 2, 3

    2023 arXiv

  11. [19]

    From system 1 to system 2: A survey of reasoning large language models

    Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. From system 1 to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502.17419, 2025. 3

    2025 arXiv

  12. [20]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. 1

    2024 arXiv

  13. [21]

    X-reasoner: Towards generalizable reasoning across modalities and domains

    Qianchu Liu, Sheng Zhang, Guanghui Qin, Timothy Os- sowski, Yu Gu, Ying Jin, Sid Kiblawi, Sam Preston, Mu Wei, Paul V ozila, et al. X-reasoner: Towards generalizable reasoning across modalities and domains. arXiv preprint arXiv:2505.03981, 2025. 3

    2025 arXiv

  14. [22]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vi- sion, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vi- sion, pages 216–233. Springer, 2024. 2, 3

    2024

  15. [23]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel 10 Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023. 1, 2, 3, 4, 6

    arXiv

  16. [24]

    Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022. 3

    2022 arXiv

  17. [25]

    Visual cot: Advancing multi-modal language models with a com- prehensive dataset and benchmark for chain-of-thought rea- soning

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuo- fan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a com- prehensive dataset and benchmark for chain-of-thought rea- soning. Advances in Neural Information Processing Sys...

    2024

  18. [26]

    Vlm-r1: A stable and generaliz- able r1-style large vision-language model

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-r1: A stable and generaliz- able r1-style large vision-language model. arXiv preprint arXiv:2504.07615, 2025. 3

    2025 arXiv

  19. [27]

    Claude 3.7 sonnet, 2025

    Anthropic Team. Claude 3.7 sonnet, 2025. 3, 6, 7

    2025

  20. [28]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text. arXiv preprint arXiv:2403.05530, 2024. 3, 6, 7

    2024 arXiv

  21. [29]

    Kimi k1.5: Scaling reinforcement learning with llms

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1.5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025. 3

    2025 arXiv

  22. [30]

    Qvq: To see the world with wisdom, 2024

    Qwen Team. Qvq: To see the world with wisdom, 2024. 1, 3, 6

    2024

  23. [31]

    Qwq: Reflect deeply on the boundaries of the unknown, 2024

    Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, 2024. 1, 3

    2024

  24. [32]

    Enhancing the reason- ing ability of multimodal large language models via mixed preference optimization

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. Enhancing the reason- ing ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442,

    arXiv

  25. [33]

    Open-r1-video

    Xiaodong Wang and Peixi Peng. Open-r1-video. https: //github.com/Wang- Xiaodong1899/Open- R1- Video, 2025. 3

    2025

  26. [34]

    Vi- suothink: Empowering lvlm reasoning with multimodal tree search

    Yikun Wang, Siyin Wang, Qinyuan Cheng, Zhaoye Fei, Liang Ding, Qipeng Guo, Dacheng Tao, and Xipeng Qiu. Vi- suothink: Empowering lvlm reasoning with multimodal tree search. arXiv preprint arXiv:2504.09130, 2025. 3

    2025 arXiv

  27. [35]

    Charxiv: Charting gaps in realistic chart understanding in multimodal llms

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sad- hika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. Advances in Neural In- formation Processing Systems, 37:113569–1136...

    2025

  28. [36]

    Boosting mul- timodal reasoning with mcts-automated structured thinking

    Jinyang Wu, Mingkuan Feng, Shuai Zhang, Ruihan Jin, Feihu Che, Zengqi Wen, and Jianhua Tao. Boosting mul- timodal reasoning with mcts-automated structured thinking. arXiv preprint arXiv:2502.02339, 2025. 3

    2025

  29. [37]

    Deepseek-vl2: Mixture-of- experts vision-language models for advanced multimodal understanding

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of- experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024. 3, 6

    2024 arXiv

  30. [38]

    Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning

    Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502.14768, 2025. 1

    2025 arXiv

  31. [39]

    Llava-o1: Let vision language models reason step- by-step

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-o1: Let vision language models reason step- by-step. arXiv preprint arXiv:2411.10440, 2024. 1, 3, 6

    2024 arXiv

  32. [40]

    Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search

    Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search. arXiv preprint arXiv:2412.18319 , 2024. ...

    2024 arXiv

  33. [41]

    R1-sharevl: Incentivizing reasoning capability of multimodal large language models via share-grpo

    Huanjin Yao, Qixiang Yin, Jingyi Zhang, Min Yang, Yibo Wang, Wenhao Wu, Fei Su, Li Shen, Minghui Qiu, Dacheng Tao, et al. R1-sharevl: Incentivizing reasoning capability of multimodal large language models via share-grpo. arXiv preprint arXiv:2505.16673, 2025. 3

    2025 arXiv

  34. [42]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 3, 6

    2024 arXiv

  35. [43]

    Mmt-bench: A comprehensive multimodal bench- mark for evaluating large vision-language models towards multitask agi

    Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprehensive multimodal bench- mark for evaluating large vision-language models towards multitask agi. arXiv preprint arXiv:2404.16006, 2024. 3

    2024 arXiv

  36. [44]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 2

    2023 arXiv

  37. [45]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In Proceedings of the IEEE/CVF Conference on...

    2024

  38. [46]

    Mmmu-pro: A more robust multi- discipline multimodal understanding benchmark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi- discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024. 2, 3

    2024 arXiv

  39. [47]

    Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. Mm1. 5: Methods, analysis & insights from multimodal llm fine-tuning. arXiv preprint arXiv:2409.20566, 2024. 3

    2024 arXiv

  40. [48]

    R1-vl: Learn- ing to reason with multimodal large language models via step-wise group relative policy optimization

    Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learn- ing to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.12937, 2025. 3 11

    2025 arXiv

  41. [49]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Sprin...

    2024

  42. [50]

    Improve vision language model chain-of- thought reasoning

    Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, and Yiming Yang. Improve vision language model chain-of- thought reasoning. arXiv preprint arXiv:2410.16198, 2024. 1, 3

    2024 arXiv

  43. [51]

    Dynamath: A dynamic visual bench- mark for evaluating mathematical reasoning robustness of vision language models

    Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual bench- mark for evaluating mathematical reasoning robustness of vision language models. arXiv preprint arXiv:2411.00836,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.