REVIEW 5 major objections 5 minor 3 cited by
RBench-V: A Primary Assessment for Visual Reasoning Models with Multi-modal Outputs
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper's benchmark RBench-V asks models to draw while reasoning; the best model scores 25.8 percent to humans' 82.3 percent.
desk verdict A genuinely new benchmark for multi-modal output reasoning with a striking model-human gap, but the central construct is not yet validated because the evaluation protocol never actually requires or scores image generation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the benchmark itself: 803 hand-selected question-answer pairs spanning math (176), physics (157), counting (195), and games (275), of which 763 have multimodal inputs and 40 are text-only, split into 356 multiple-choice and 447 open-ended items. The selection criterion is that solving a question should require producing new visual content—drawing a geometric figure, adding auxiliary lines, tracing a trajectory, connecting dots—rather than merely interpreting the input. The evaluation protocol measures reasoning through multi-modal outputs by using a unified LLM-as-a-judge framework with GPT-4o scoring each model's final top-1 text answer.
What would settle it
A direct test would be to run o3 and the other top models on RBench-V with all image-generation ability disabled while still allowing the input image to be viewed, then compare accuracy with the reported 25.8 percent. If a text-only-reasoning model scores the same, or if a version of o3 forbidden from drawing shows no drop, then the benchmark is not actually measuring multi-modal output reasoning.
Extended reading notes
Core claim
The paper's central claim is that RBench-V measures a capability existing benchmarks miss: vision-indispensable reasoning with multi-modal outputs, or multi-modal chain-of-thought. While MMLU and MMMU supply multimodal inputs and demand text answers, RBench-V questions are designed so that the solver must generate novel images, construct auxiliary lines, trace light rays or maze paths, or mark counted objects en route to an answer. The evaluation reports a decisive gap: the best model, o3, scores 25.8 percent overall, far below the human expert score of 82.3 percent, and the best open-source model scores 10.6 percent. The paper also finds that larger models, omni-models with joint text-image decoding, and long text-only reasoning models show little or no improvement on the benchmark, and it documents cases where o3 solves geometry algebraically rather than by drawing, a route it calls a 'multi-modal reasoning shortcut'. It concludes that current foundation models struggle to generate and integrate multi-modal outputs in visual thinking.
Load-bearing premise
The benchmark's validity depends on the 803 questions truly being solvable only by generating or modifying an image during reasoning, yet the evaluation checks only final text answers and never verifies that any image was produced.
Editorial extensions
If this is right
- If RBench-V measures what it claims, omni-models that decode images as well as text do not automatically gain visual reasoning ability: Qwen2.5VL-7B and Qwen2.5-Omni-7B score at similar levels, as do MiniCPM-V-2.6 and MiniCPM-o-2.6.
- Scaling model size is not sufficient: raising Qwen2.5VL from 7B to 72B, or scaling InternVL and LLaVA-OneVision similarly, produces no clear accuracy gain on RBench-V.
- Long text-only chain-of-thought models barely outperform their non-thinking counterparts, so the missing capability lies in generating multimodal outputs rather than in spending more tokens on text reasoning.
- Reporting accuracy after removing math questions, where models can substitute algebra for drawing, widens the human-model gap and is proposed as a cleaner signal of true multimodal reasoning.
- o3's large lead over prior models is read as evidence both that RBench-V tracks genuine progress and that the field remains far from human-level visual reasoning.
Reading between the lines
- A stricter protocol that requires the model to actually emit an image (an annotated drawing, a traced path) before the final answer would test whether the low scores come from an inability to draw or from a failure to engage the drawing modality; the current protocol judges only final text.
- One testable extension is to give models an external drawing tool and allow them to feed their own generated image back as input; if accuracy rises sharply, the bottleneck is generating the visual step rather than visual perception itself.
- Because the paper identifies algebraic shortcuts in math, a focused pure-drawing subset could be repurposed as a training signal: reinforcement learning that rewards correctly drawn auxiliary structures might close the gap faster than scale, which the paper's own scaling results predict.
- The same question-design principle could extend beyond still images to video or audio outputs, which the paper names as future work.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces RBench-V, a benchmark of 803 hand-curated questions across mathematics, physics, counting, and games, intended to assess what the authors call 'vision-indispensable reasoning' that requires multi-modal outputs, such as drawing auxiliary lines, tracing paths, or constructing images. The authors evaluate a broad range of open- and closed-source models, including o3, Gemini 2.5 Pro, GPT-4o, and Qwen2.5-VL, and report that the best model, o3, achieves 25.8% accuracy versus 82.3% for human experts. They also report a 'w/o math' score to control for algebraic shortcuts and argue that current models fail because they cannot generate or manipulate images during reasoning. Data and code are released publicly.
Significance. If the construct were properly validated, RBench-V would fill a genuine gap by evaluating multi-modal chain-of-thought output rather than input understanding alone. The paper's strengths include a broad model suite, a human baseline, public data and code, and an honest acknowledgment that math items can be solved via text-only algebraic reasoning. However, the evaluation protocol never requires or scores an image output, and the judge is GPT-4o, which is itself one of the evaluated models. These issues mean that the headline accuracy numbers are informative measurements of final-answer performance, but they do not yet support the paper's specific central claim about multi-modal output capability or about models' inability to use drawing as a thinking tool.
major comments (5)
- [Section 4 (evaluation protocol)] The central claim that RBench-V measures multi-modal output reasoning is not supported by the evaluation protocol, because models are never required to emit an image and no image output is scored. Section 4 states that a unified LLM-as-a-Judge framework with GPT-4o is used and Top-1 accuracy is reported, which judges only the final text answer. The connect-the-dots example in Section 4.3, where o3 describes the diagram instead of connecting the dots, is scored incorrect, but this outcome is equally consistent with a failure in visual understanding or in text-only reasoning. To support the central claim, the protocol should elicit and score image outputs, or otherwise demonstrate per-item that text-only reasoning cannot produce the correct answer.
- [Section 4 (LLM-as-a-Judge)] GPT-4o is used as the judge while also being one of the evaluated models, and no human agreement is reported. This creates an uncontrolled potential bias in the model rankings and in the human-versus-model comparison. Please report inter-rater agreement on a sample (e.g., Cohen's kappa) against human graders, and consider using a judge model that is not in the evaluated set, or human grading for open-ended items. In addition, Table 3 reports no confidence intervals or significance tests; several adjacent scores, such as 10.0 and 10.6 for InternVL-3-38B and Qwen2.5VL-72B, are likely within sampling noise, so the ranking claims are not statistically supported.
- [Section 3.1 and Section 4.4 (construct validity)] The evidence that drawing is necessary for solving RBench-V items is limited to the design principle and to Table 2's win rates, which are based on only 30 sampled items per benchmark and on subjective ratings by models and experts. High win rates show that RBench-V items are judged as more drawing-intensive than MMLU or MMMU items, not that drawing is required for a correct answer. Moreover, Section 4.4 concedes that math items can be solved by coordinate algebra and therefore reports scores 'w/o math,' but no analogous check is provided for counting, physics, or games. Without category-level controls, such as comparing a text-only reasoning condition against a drawing-allowed condition, the phrase 'vision-indispensable' is not established.
- [Section 4.3 and Table 3 (math shortcut)] The 'w/o math' result is a post-hoc exclusion of 176 of 803 questions, and the headline 25.8% accuracy therefore mixes items with and without the alleged multi-modal requirement. If RBench-V is meant to assess multi-modal output, the math category as currently designed fails for models that can use algebraic shortcuts. The paper should either redesign math items to require geometric construction, or present RBench-V as a mixed benchmark that measures both multi-modal and text-only strategies, with the 'w/o math' score as the primary evidence for the specific multi-modal claim. As written, the 25.8% figure is not a clean measure of multi-modal output ability.
- [Section 4.2 and Table 3 (human baseline)] The human expert score of 82.3% lacks essential reporting details: the number of experts, the number of questions assigned per expert, time limits, whether drawing was allowed or required, and the grading protocol are not specified. Without this information, the headline human-model gap cannot be interpreted as a comparison of multi-modal output capability. The authors should also state whether the same rubric and judge were used for human answers and model answers, since differences in grading criteria could account for part of the gap.
minor comments (5)
- [Figure 2 caption] In the caption of Figure 2, 'Rench' should be 'R-Bench'.
- [Abstract and Section 1] There are several typos: 'vision-indisperential' in the Section 1 bullet list should be 'vision-indispensable', and 'humam' in Section 4 should be 'human'.
- [Section 4.2] The deployment details for vLLM and VLMEvalKit are not given (versions, batch sizes, or any non-default generation parameters), which limits reproducibility of the open-source model numbers.
- [Section 3.2] The statement that RBench-V includes 40 text-only questions sits awkwardly with the 'vision-indispensable' framing; please clarify whether these items can be solved without visual input and whether they are included in the 'w/o math' analysis.
- [Figure 3] The red lines in Figure 3 illustrate an idealized reasoning trace rather than an actual model output; the caption should make this explicit to avoid implying that any evaluated model produced those drawings.
Circularity Check
No derivational circularity; minor self-citation only; construct-validity caveat is not circularity.
full rationale
RBench-V's reported scores are empirical measurements, not quantities derived from the benchmark's definition. The paper fits no parameters, uses no equations whose outputs are preset by their inputs, and makes no prediction that is definitionally equal to its evidence. The low model accuracies and the 82.3% human score in Table 3 are independent empirical observations obtained through a fixed evaluation protocol, so the central claim (models underperform humans on RBench-V) does not reduce to the construction of the benchmark. The only self-citation is the companion R-Bench paper (Guo et al. 2025) in Section 2.2, where it is listed among existing text-input benchmarks; that citation is not load-bearing for RBench-V's design, its question curation, or its conclusions. The paper does contain a self-referential validation statement in Section 4.4 ('Notably, the results also validate the design of RBench-V itself'), but this is an interpretive loop rather than an algebraic or definitional reduction: the scores would still be empirical even if the validation claim is weak. The more substantive concern raised by the manuscript is construct validity, not circularity: the protocol records only a final text answer judged by GPT-4o and never requires or scores an image output, and the analysis of o3's connect-the-dots failure ('merely attempting to describe the points in the diagram, rather than actually connecting them') is treated as evidence against multi-modal output capability without directly testing that capability. That is a validity gap, not a derivational circularity, and per the hard rules it does not elevate the circularity score. The score of 2 reflects the minor non-load-bearing self-citation and the self-validation phrasing, while explicitly not treating the construct-validity issue as circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption Questions in RBench-V can only be solved by generating or modifying images during reasoning.
- domain assumption GPT-4o can reliably judge open-ended visual reasoning answers.
- standard math Standard domain knowledge in geometry, physics, and counting is taken as background.
Cite this review
Pith. "Pith review of RBench-V: A Primary Assessment for Visual Reasoning Models with Multi-modal Outputs." pith.science (2026). https://pith.science/paper/7QOHRFXO
@misc{pith2026250516770,
author = {Pith},
title = {Pith review of: RBench-V: A Primary Assessment for Visual Reasoning Models with Multi-modal Outputs},
year = {2026},
howpublished = {\url{https://pith.science/paper/7QOHRFXO}},
note = {Machine review of arXiv:2505.16770}
}
read the original abstract
The rapid advancement of native multi-modal models and omni-models, exemplified by GPT-4o, Gemini, and o3, with their capability to process and generate content across modalities such as text and images, marks a significant milestone in the evolution of intelligence. Systematic evaluation of their multi-modal output capabilities in visual thinking processes (also known as multi-modal chain of thought, M-CoT) becomes critically important. However, existing benchmarks for evaluating multi-modal models primarily focus on assessing multi-modal inputs and text-only reasoning while neglecting the importance of reasoning through multi-modal outputs. In this paper, we present a benchmark, dubbed RBench-V, designed to assess models' vision-indispensable reasoning abilities. To construct RBench-V, we carefully hand-pick 803 questions covering math, physics, counting, and games. Unlike previous benchmarks that typically specify certain input modalities, RBench-V presents problems centered on multi-modal outputs, which require image manipulation such as generating novel images and constructing auxiliary lines to support the reasoning process. We evaluate numerous open- and closed-source models on RBench-V, including o3, Gemini 2.5 Pro, Qwen2.5-VL, etc. Even the best-performing model, o3, achieves only 25.8% accuracy on RBench-V, far below the human score of 82.3%, highlighting that current models struggle to leverage multi-modal reasoning. Data and code are available at https://evalmodels.github.io/rbenchv
Figures
Figures from the paper (1 more)
Forward citations
Cited by 3 Pith papers
-
SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents
SciAgent-8B, fine-tuned on trajectories synthesized from a tool dependency graph, outperforms Qwen3-VL-235B-Instruct on SciAgentBench, a new 259-task benchmark for multi-step scientific tool-use.
-
R-4B: Incentivizing General-Purpose Auto-Thinking Capability in MLLMs via Bi-Mode Annealing and Reinforce Learning
R-4B uses bi-mode annealing plus a GRPO variant with forced thinking and non-thinking rollouts so a 4B multimodal model adaptively picks when to reason, reportedly matching 16B thinking models on several reasoning benchmarks.
-
Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task
An interleaved temporal-spatial tool-use framework ('STAR') improves GPT-4o's VideoQA accuracy by 8.2% on VideoMME and 4.6% on LongVideoBench.
Reference graph
Works this paper leans on
-
[1]
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219,
-
[4]
Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,
-
[5]
DeepSeek. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,
-
[6]
Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models.arXiv preprint arXiv:2410.07985,
-
[8]
Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,
Google, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,
-
[9]
Google, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530,
-
[10]
URL https: //arxiv.org/abs/2505.02018. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations,
-
[12]
Pytorch image quality: Metrics for image quality assessment.arXiv preprint arXiv:2208.14818,
Sergey Kastryulin, Jamil Zakirov, Denis Prokopenko, and Dmitry V Dylov. Pytorch image quality: Metrics for image quality assessment.arXiv preprint arXiv:2208.14818,
Show all 24 references
-
[13]
Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326,
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326,
-
[15]
Doubao-1.5-pro
Seed. Doubao-1.5-pro. https: // seed. bytedance. com/ en/ special/ doubao\ _1\ _5\ _pro/, 2025a. Seed. Seed1.5-thinking: Advancing superb reasoning models with reinforcement learning, 2025b. URLhttps://arxiv.org/abs/2504.13914. StepFun. Step-r1-v-mini.https://www.stepfun.com/c...
-
[16]
Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971,
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971,
-
[17]
Measuring multimodal mathematical reasoning with math-vision dataset.arXiv preprint arXiv:2402.14804, 2024a
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset.arXiv preprint arXiv:2402.14804, 2024a. Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang...
-
[18]
Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.arXiv preprint arXiv:2406.01574, 2024c
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.arXiv preprint arXiv:2406.01574, 2024c. Zhiyu Wu, Xiaoka...
-
[19]
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report.arXiv preprint arXiv:2503.20215,
-
[20]
Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800,
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800,
-
[21]
Descriptive image quality assessment in the wild.arXiv preprint arXiv:2405.18842,
Zhiyuan You, Jinjin Gu, Zheyuan Li, Xin Cai, Kaiwen Zhu, Chao Dong, and Tianfan Xue. Descriptive image quality assessment in the wild.arXiv preprint arXiv:2405.18842,
-
[22]
Mmmu: A massive multi-discipline multimodal under- standing and reasoning benchmark for expert agi
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal under- standing and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on ...
-
[24]
Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479,
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479,
-
[1991]
Gemini 2.5: Our most intelligent ai model.https://blog.google/technology/google- deepmind/gemini-model-thinking-updates-march-2025/#gemini-2-5-thinking,
Google. Gemini 2.5: Our most intelligent ai model.https://blog.google/technology/google- deepmind/gemini-model-thinking-updates-march-2025/#gemini-2-5-thinking,
2025
-
[2017]
Mistral 7b.arXiv preprint arXiv:2310.06825,
10 Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b.arXiv preprint arXiv:2310.06825,
-
[2022]
Minigpt-4: En- hancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592,
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592,
-
[2023]
Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923,
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923,
-
[2024]
Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255,
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255,
-
[2025]
Qwen technical report.arXiv preprint arXiv:2309.16609,
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.