Pith. sign in

REVIEW 5 major objections 5 minor 118 references

Seeing is Not Reasoning: MVPBench for Graph-based Evaluation of Multi-path Visual Physical CoT

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Leading multimodal models cannot reliably execute multi-step, visually grounded physical reasoning, and common RL post-training often makes spatial reasoning worse.

desk verdict Useful benchmark, but the headline findings rest on an unvalidated GPT-4o judge and a completeness assumption that need to be fixed before the strong claims stand. read the letter →

arxiv 2505.24182 v1 pith:MXUCZHBE submitted 2025-05-30 cs.CV cs.AI

Zhuobai Dong , Junchao Yi , Ziyuan Zheng , Haochen Han , Xiangxi Zheng , Alex Jinpeng Wang , Fangming Liu , Linjie Li This is my paper
classification cs.CVcs.AI
keywords visualphysicalreasoningchain-of-thoughtmultimodallargelanguagemodelsgraph-basedevaluationmulti-pathreinforcementlearningpost-trainingspatialbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to test whether multimodal large language models can genuinely reason about physical scenes image by image, rather than merely describe them. It introduces MVPBench, a benchmark of 1,211 tasks that present multiple related images and require a final answer plus a step-by-step reasoning chain anchored to the visual evidence, with several human-annotated valid chains per task. On this benchmark, even the strongest tested models show low reasoning accuracy and weak image-text alignment, and reasoning errors dominate over perception errors in most subsets. The graph-based chain-of-thought metrics reveal that models often fail to follow valid logical paths, and RL-based post-training, commonly believed to improve visual reasoning, frequently lowers spatial-reasoning scores. If the paper is right, current alignment practices do not produce physically grounded reasoning, and reward design needs to become vision-centric.

What carries the argument

The load-bearing evaluation machinery is a graph-based chain-of-thought consistency metric. For each item, human-annotated key steps are nodes of a directed graph whose edges encode valid logical transitions, and multiple valid paths are recorded for each task. A model's generated reasoning is parsed into steps, embedded as edges in the same graph, and scored by Path Validity Rate (the share of model edges present in the reference graph), Path Coverage Score (the normalized longest matched sub-path), and their harmonic mean, the CoT Match Score, with adjusted versions that penalize over-generation. Quality is measured by Step Accuracy Score, Key Step Coverage, and a weighted CoT Reasoning Score, while efficiency is measured by Step Relevance Score and Reflection Validity Rate. The graph formalism is what lets the benchmark separate visually grounded multi-step reasoning from fluent but ungrounded narration.

What would settle it

Fine-tune a base model with reinforcement learning whose reward explicitly scores step-level grounding against the MVPBench reference graphs; if that model outscores its untrained base on spatial-reasoning metrics, the paper's conclusion that RL post-training harms spatial reasoning fails for that reward family.

Watch

Extended reading notes

Core claim

The central claim is that seeing is not reasoning: current multimodal models can perceive objects in images but cannot carry out coherent multi-step visual physical reasoning. Concretely, MVPBench asks models to answer physics questions from interleaved image sequences under chain-of-thought prompting, and the strongest closed model reaches a CoT Reasoning Score of about 75 in the multi-image setting while most open models score lower, with the Spatial Relations subset the hardest. The paper's error analysis attributes most failures to reasoning steps rather than visual perception, except in spatial-relation tasks where perception errors are far more common. The paper further reports that multi-image input helps almost all models, by up to 21 percentage points, and that RL-based post-training, tested with several open models and reward families, can reduce visual-physics and spatial-reasoning scores relative to their base counterparts.

Load-bearing premise

The load-bearing premise is that the human-annotated key-step graphs are a complete inventory of valid reasoning paths; if annotators missed legitimate alternative chains, or if the automated step parser fails, the diversity and quality scores understate model ability and the headline conclusion is weakened.

Editorial extensions

If this is right

  • High scores on conventional visual question answering tell little about whether a model reasons from images; performance must be judged at the level of individual reasoning steps.
  • Multi-image input is not optional for physical reasoning: giving the full image sequence raised reasoning scores by up to 21 points, so single-frame evaluations understate both ability and failure.
  • RL-based post-training that optimizes final-answer or conversational fluency can degrade spatial reasoning, so alignment rewards should be redesigned to include step-level visual grounding.
  • Spatial relations, not physics knowledge, are the current bottleneck, with perception errors playing a much larger role there than in physics experiments and problems.
  • Current multimodal models should not be treated as trustworthy physical-world reasoners for embodied or science applications without step-level visual verification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the reference graph is treated as the complete set of valid paths, the diversity metrics may penalize legitimate chains that annotators did not list; crowd-sourcing many more valid paths per item and re-measuring would test whether the headline understates model ability.
  • The paper's 'RL harms spatial reasoning' result is conditional on the tested reward designs; it predicts that RL with explicit step-level visual-grounding rewards would reverse the drop, and MVPBench provides the measurement tool to check that.
  • The graph formalism transfers directly to other multi-step visual domains, such as medical image interpretation or driving scene understanding, where the same question of whether a model's text is anchored to what it sees arises.
  • The strong multi-image gains suggest a possible scaling relationship between the number of temporally spaced images and measured reasoning quality, which the paper does not vary systematically.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces MVPBench, a benchmark of 1,211 carefully curated visual-physics examples with multi-image inputs and multi-path chain-of-thought (CoT) annotations across four domains: physics experiments, physics problems, spatial relations, and dynamic prediction. The authors propose a graph-based evaluation suite with correctness metrics (SAS, KSC, CRS), diversity metrics (PVR, PCS, CMS), and efficiency metrics (SRS, RVR), and report results on open- and closed-source MLLMs under single- and multi-image settings. The headline findings are that current MLLMs perform poorly on visually grounded physical reasoning, multi-image input helps most models, and RL-based post-training often degrades spatial reasoning and visual-physics scores.

Significance. If the evaluation methodology is validated, MVPBench fills a real gap: it combines real-world visual physics, multi-image inputs, and multi-path CoT annotations, and the graph-based metrics go beyond final-answer accuracy. The authors release data and code, report human diversity performance (Appendix B.1), and use human final-answer labels for the correctness component, which is a genuine strength. However, the step-level metrics that carry most of the paper's conclusions are produced by GPT-4o as an unvalidated judge, and no text-only baseline is provided, so the central claims are currently conditional on the reliability of that judge and on the completeness of the human-annotated reference graphs.

major comments (5)
  1. [Section 4.1, Section 4.3, Appendix E] SAS, KSC, SRS, and RVR are all computed from GPT-4o judgments on step decomposition, correctness, relevance, and reflection validity, but no inter-annotator agreement or human rescoring is reported for these judgments. Limitation (iii) in Appendix E acknowledges dependence on proprietary models but does not validate that the judgments are correct. Since GPT-4o is also one of the evaluated systems, a systematic preference for certain step granularity, phrasing, or reasoning style could differentially affect models, especially post-trained checkpoints. Please add a validation study: human rescoring of a sample of model outputs across multiple models, reporting Cohen's kappa or similar, and a sensitivity analysis with an alternative judge.
  2. [Section 5 and Table 2] The abstract claims that models exhibit 'weak image-text alignment' and that MVPBench minimizes text-prior shortcuts, but no text-only baseline is reported. The experiments only compare single-image versus multi-image inputs; neither condition removes the images. Without a text-only or image-blind control, the claim that failures are due to visual reasoning rather than language priors is unsupported. Please add a text-only baseline (e.g., the same questions without images, or with captions only) and report the accuracy gap.
  3. [Section 4.2] PVR and PCS assume that the human-annotated reference graph is a complete inventory of valid reasoning paths: any model edge not matching the graph is penalized, and PCS uses the longest matched sub-path. If annotators missed legitimate alternative chains, or if step parsing by GPT-4o fails, the diversity scores understate model ability. The average of 2.67 reference paths per sample is reported, but no completeness validation is provided. Please have human experts judge a sample of unmatched model paths as valid or invalid, and report the proportion of valid paths currently counted as errors.
  4. [Section 4.2] The adjusted path coverage score is defined as PCS * exp(-alpha * (Np/Ngt - 1)), where alpha controls the over-generation penalty, but the value of alpha is never reported anywhere in the manuscript. Since Table 2 and Table 3 report PCS and CMS numbers that depend on this adjustment, the results are not reproducible. Please report the alpha value used for all experiments, and include a sensitivity analysis over reasonable alpha ranges.
  5. [Figures 5-6 and Table 8] The claim that RL post-training harms spatial reasoning is based on comparisons where base and post-trained models are evaluated with different generation settings. For example, Table 8 shows InternVL2.5-78B uses max_new_tokens=2048 while InternVL2.5-78B-MPO uses 1024, and do_sample differs across models. Since SAS, KSC, SRS, and diversity metrics depend on response length and sampling behavior, the reported degradations may be artifacts of the evaluation configuration rather than of post-training. Please control generation hyperparameters across matched model pairs and report bootstrap confidence intervals or paired significance tests for the post-training comparisons.
minor comments (5)
  1. [Section 5, first paragraph] The text says diversity is assessed via PVR and RCS, but the metric introduced in Section 4.2 is PCS (Path Coverage Score); please correct the abbreviation.
  2. [Table 2 caption] The caption states RVR is set to 100 for models lacking reflection ability, but the table reports values such as 99.52 for OpenAI o3 and 99.32 for InternVL2.5-78B-MPO; please clarify how RVR is computed for models with no detected reflection steps.
  3. [Appendix C.2] There is a typo in 'Chinses Gaokao physics questions'; this should read 'Chinese Gaokao'.
  4. [Appendix H.2, Table 8] For API-based models such as GPT-4o, o3, and Claude, the table lists torch.dtype, which is not a meaningful generation setting for those models; please remove or mark those entries as not applicable.
  5. [Section 4.3] The sentence 'SRS, similar to SCS, is defined as...' references a metric 'SCS' that is not defined elsewhere; it should likely refer to SAS or should be rephrased.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all metrics are scored against human-annotated references; GPT-4o-as-judge is a validation concern, not a self-referential reduction.

full rationale

This paper is a benchmark construction and empirical evaluation, not a formal derivation; no claimed theorem or prediction is obtained from a parameter fitted to the same target. The metrics SAS, KSC, SRS, and RVR (Sections 4.1 and 4.3) are operationalized as GPT-4o judgments against human-annotated key steps and references, and PVR/PCS (Section 4.2) compare model paths with a reference graph constructed from human annotations. Those references are external to the evaluated models, so the scores are not defined in terms of the outputs they score. The final-answer component Correct(s_A) is a human label. The closest concern is evaluator bias: GPT-4o is both a principal evaluated subject and the judge, and the paper itself flags dependence on proprietary MLLM outputs in Limitations (iii). This is a validity threat but not circularity; no equation reduces a result to its input, and the self-citations (e.g., [30], [43]) are background related-work, not load-bearing for the benchmark's conclusions. The limited human performance check (Appendix B.1) is narrow and does not validate step-level judgments, but again that is a missing-validation issue, not a circular derivation. Therefore no circular step is exhibited.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The benchmark's scores depend on several unvalidated assumptions, most importantly the correctness and completeness of annotator key steps and the reliability of the GPT-4o judge. Two hand-set alpha weights enter the headline metrics without sensitivity analysis.

free parameters (2)
  • alpha_CRS = 0.7
    Weight in CRS = alpha*SAS + (1-alpha)*Correct, set by default in Section 4.1; no sensitivity analysis is reported.
  • alpha_overgeneration = None reported
    Penalty exponent in PCS_adj = PCS*exp(-alpha*(Np/Ngt - 1)) in Section 4.2; the value used in experiments is not stated.
assumptions (4)
  • domain assumption Annotated key steps are the correct and complete reasoning decomposition for each item.
    All SAS/KSC/PVR/PCS scores treat the human-annotated key steps as ground truth; Appendix C.1 describes double annotation but no agreement statistics are reported.
  • domain assumption GPT-4o reliably partitions model outputs and judges step correctness and relevance.
    Section 4.1 and Appendix G rely on GPT-4o as evaluator; no human validation of the judge is reported.
  • domain assumption Reference graph matching captures physical reasoning validity.
    Section 4.2 maps model paths to annotated graphs; a model chain that is physically correct but not in the reference graph scores as unmatchable.
  • domain assumption Visual evidence in the provided images is sufficient to determine the answer without external text priors.
    The benchmark claims to minimize text-prior shortcuts, but no text-only baseline verifies this; Physics Problems are word problems coupled to images.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Seeing is Not Reasoning: MVPBench for Graph-based Evaluation of Multi-path Visual Physical CoT." pith.science (2026). https://pith.science/paper/MXUCZHBE

@misc{pith2026250524182,
  author       = {Pith},
  title        = {Pith review of: Seeing is Not Reasoning: MVPBench for Graph-based Evaluation of Multi-path Visual Physical CoT},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MXUCZHBE}},
  note         = {Machine review of arXiv:2505.24182}
}
read the original abstract

Understanding the physical world - governed by laws of motion, spatial relations, and causality - poses a fundamental challenge for multimodal large language models (MLLMs). While recent advances such as OpenAI o3 and GPT-4o demonstrate impressive perceptual and reasoning capabilities, our investigation reveals these models struggle profoundly with visual physical reasoning, failing to grasp basic physical laws, spatial interactions, and causal effects in complex scenes. More importantly, they often fail to follow coherent reasoning chains grounded in visual evidence, especially when multiple steps are needed to arrive at the correct answer. To rigorously evaluate this capability, we introduce MVPBench, a curated benchmark designed to rigorously evaluate visual physical reasoning through the lens of visual chain-of-thought (CoT). Each example features interleaved multi-image inputs and demands not only the correct final answer but also a coherent, step-by-step reasoning path grounded in evolving visual cues. This setup mirrors how humans reason through real-world physical processes over time. To ensure fine-grained evaluation, we introduce a graph-based CoT consistency metric that verifies whether the reasoning path of model adheres to valid physical logic. Additionally, we minimize shortcut exploitation from text priors, encouraging models to rely on visual understanding. Experimental results reveal a concerning trend: even cutting-edge MLLMs exhibit poor visual reasoning accuracy and weak image-text alignment in physical domains. Surprisingly, RL-based post-training alignment - commonly believed to improve visual reasoning performance - often harms spatial reasoning, suggesting a need to rethink current fine-tuning practices.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

118 extracted references · 36 canonical work pages

  1. [1]

    Origins of physical knowledge.Psychological Review, 99(4):605–632, 1992

    Elizabeth S Spelke and Katherine Breinlinger. Origins of physical knowledge.Psychological Review, 99(4):605–632, 1992

    1992

  2. [2]

    Infants’ physical world.Current Directions in Psychological Science, 13(3):89–94, 2004

    Renee Baillargeon. Infants’ physical world.Current Directions in Psychological Science, 13(3):89–94, 2004

    2004

  3. [3]

    A theory of causal learning in children: Causal maps and bayes nets.Psychological Review, 111(1):3–32, 2004

    Alison Gopnik, Clark Glymour, David M Sobel, Laura E Schulz, Tamar Kushnir, and David Danks. A theory of causal learning in children: Causal maps and bayes nets.Psychological Review, 111(1):3–32, 2004

    2004

  4. [4]

    Building machines that learn and think like people.Behavioral and Brain Sciences, 40, 2017

    Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people.Behavioral and Brain Sciences, 40, 2017

    2017

  5. [5]

    GPT o3.https://chatgpt.com/?model=o3, 2025

    OpenAI. GPT o3.https://chatgpt.com/?model=o3, 2025

    2025

  6. [6]

    Hello GPT-4o

    OpenAI. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/ , 2024. Accessed: [Insert Access Date]

    2024

  7. [7]

    Gemini: A family of highly capable multimodal models, 2024

    Google Deepmind. Gemini: A family of highly capable multimodal models, 2024

    2024

  8. [8]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zh...

    arXiv 2025

Show all 118 references
  1. [9]

    Kimi k1.5: Scaling reinforcement learning with llms, 2025

    KimiTeam. Kimi k1.5: Scaling reinforcement learning with llms, 2025

    2025

  2. [10]

    R1-v: Reinforcing super generalization ability in vision-language models with less than $3.https://github.com/Deep-Agent/R1-V, 2025

    Chen Liang, Li Lei, Zhao Haozhe, Song Yifan, and Vinci. R1-v: Reinforcing super generalization ability in vision-language models with less than $3.https://github.com/Deep-Agent/R1-V, 2025

    2025

  3. [11]

    Easyr1: An efficient, scalable, multi-modality rl training framework

    Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. Easyr1: An efficient, scalable, multi-modality rl training framework. https://github.com/hiyouga/EasyR1, 2025

    2025

  4. [12]

    Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning.arXiv preprint arXiv:2403.16999, 2024

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning.arXiv preprint arXiv:2403.16999, 2024

    2024 arXiv

  5. [13]

    Can we generate images with cot? let’s verify and reinforce image generation step by step, 2025

    Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng-Ann Heng. Can we generate images with cot? let’s verify and reinforce image generation step by step, 2025

    2025

  6. [14]

    Imagine while reasoning in space: Multimodal visualization-of-thought, 2025

    Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vuli´c, and Furu Wei. Imagine while reasoning in space: Multimodal visualization-of-thought, 2025

    2025

  7. [15]

    Mm-spatial: Exploring 3d spatial understanding in multimodal llms, 2025

    Erik Daxberger, Nina Wenzel, David Griffiths, Haiming Gang, Justin Lazarow, Gefen Kohavi, Kai Kang, Marcin Eichner, Yinfei Yang, Afshin Dehghan, and Peter Grasch. Mm-spatial: Exploring 3d spatial understanding in multimodal llms, 2025

    2025

  8. [16]

    Mllm-for3d: Adapting multimodal large language model for 3d reasoning segmentation, 2025

    Jiaxin Huang, Runnan Chen, Ziwen Li, Zhengqing Gao, Xiao He, Yandong Guo, Mingming Gong, and Tongliang Liu. Mllm-for3d: Adapting multimodal large language model for 3d reasoning segmentation, 2025

    2025

  9. [17]

    Mllm-sul: Multimodal large language model for semantic scene understanding and localization in traffic scenarios, 2024

    Jiaqi Fan, Jianhua Wu, Jincheng Gao, Jianhao Yu, Yafei Wang, Hongqing Chu, and Bingzhao Gao. Mllm-sul: Multimodal large language model for semantic scene understanding and localization in traffic scenarios, 2024

    2024

  10. [18]

    The second half

    Shunyu Yao. The second half. https://ysymyth.github.io/The-Second-Half/ , 2025. Accessed: 2025-05-13

    2025

  11. [19]

    How far is video generation from world model: A physical law perspective.arXiv preprint arXiv:2411.02385, 2024

    Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How far is video generation from world model: A physical law perspective.arXiv preprint arXiv:2411.02385, 2024. 10

    2024 arXiv

  12. [20]

    Contphy: Continuum physical concept learning and reasoning from videos

    Zhicheng Zheng, Xin Yan, Zhenfang Chen, Jingzhou Wang, Qin Zhi Eddie Lim, Joshua B Tenenbaum, and Chuang Gan. Contphy: Continuum physical concept learning and reasoning from videos. InInternational Conference on Machine Learning. PMLR, 2024

    2024

  13. [21]

    Mdk12-bench: A multi-discipline benchmark for evaluating reasoning in multimodal large language models, 2025

    Pengfei Zhou, Fanrui Zhang, Xiaopeng Peng, Zhaopan Xu, Jiaxin Ai, Yansheng Qiu, Chuanhao Li, Zhen Li, Ming Li, Yukang Feng, Jianwen Sun, Haoquan Zhang, Zizhen Li, Xiaofeng Mao, Wangbo Zhao, Kai Wang, Xiaojun Chang, Wenqi Shao, Yang You, and Kaipeng Zhang. Mdk12-bench: A multi-...

    2025

  14. [22]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  15. [23]

    Vlind-bench: Measuring language priors in large vision-language models, 2025

    Kang il Lee, Minbeom Kim, Seunghyun Yoon, Minsung Kim, Dongryeol Lee, Hyukhun Koh, and Kyomin Jung. Vlind-bench: Measuring language priors in large vision-language models, 2025

    2025

  16. [24]

    Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency.arXiv preprint arXiv:2502.09621, 2025

    Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, et al. Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency.arXiv preprint arXiv:2502.09621, 2025

    2025 arXiv

  17. [25]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?arXiv preprint arXiv:2403.14624, 2024

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?arXiv preprint arXiv:2403.14624, 2024

    2024 arXiv

  18. [26]

    Moka: Open-world robotic manipulation through mark-based visual prompting.arXiv preprint arXiv:2403.03174, 2024

    Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-world robotic manipulation through mark-based visual prompting.arXiv preprint arXiv:2403.03174, 2024

    2024 arXiv

  19. [27]

    Phygrasp: Generalizing robotic grasping with physics-informed large multimodal models.arXiv preprint arXiv:2402.16836, 2024

    Dingkun Guo, Yuqi Xiang, Shuqi Zhao, Xinghao Zhu, Masayoshi Tomizuka, Mingyu Ding, and Wei Zhan. Phygrasp: Generalizing robotic grasping with physics-informed large multimodal models.arXiv preprint arXiv:2402.16836, 2024

    2024 arXiv

  20. [28]

    Tenenbaum, and Alexei A

    Tyler Bonnen, Stephanie Fu, Yutong Bai, Thomas O’Connell, Yoni Friedman, Nancy Kanwisher, Joshua B. Tenenbaum, and Alexei A. Efros. Evaluating multiview object consistency in humans and image models. arXiv preprint arXiv:2409.05862, 2024

    2024 arXiv

  21. [29]

    Open eyes, then reason: Fine-grained visual mathematical understanding in mllms

    Shan Zhang, Aotian Chen, Yanpeng Sun, Jindong Gu, Yi-Yu Zheng, Piotr Koniusz, Kai Zou, Anton van den Hengel, and Yuan Xue. Open eyes, then reason: Fine-grained visual mathematical understanding in mllms. arXiv preprint arXiv:2501.06430, 2025

    2025 arXiv

  22. [30]

    V-mage: A game evaluation framework for assessing visual-centric capabilities in multimodal large language models.arXiv preprint arXiv:2504.06148, 2025

    Xiangxi Zheng, Linjie Li, Zhengyuan Yang, Ping Yu, Alex Jinpeng Wang, Rui Yan, Yuan Yao, and Lijuan Wang. V-mage: A game evaluation framework for assessing visual-centric capabilities in multimodal large language models.arXiv preprint arXiv:2504.06148, 2025

    2025 arXiv

  23. [31]

    Verify: A benchmark of visual explanation and reasoning for investigating multimodal reasoning fidelity.arXiv preprint arXiv:2503.11557, 2025

    Jing Bi, Junjia Guo, Susan Liang, Guangyu Sun, Luchuan Song, Yunlong Tang, Jinxi He, Jiarui Wu, Ali V osoughi, Chen Chen, and Chenliang Xu. Verify: A benchmark of visual explanation and reasoning for investigating multimodal reasoning fidelity.arXiv preprint arXiv:2503.11557, 2025

    2025 arXiv

  24. [32]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scienti...

    2024 arXiv

  25. [33]

    Tenen- baum

    Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenen- baum. Clevrer: Collision events for video representation and reasoning.arXiv preprint arXiv:1910.01442, 2020

    1910 arXiv

  26. [34]

    Bear, Elias Wang, Damian Mrowca, Felix J

    Daniel M. Bear, Elias Wang, Damian Mrowca, Felix J. Binder, Hsiao-Yu Fish Tung, R. T. Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Fan-Yun Sun, Li Fei-Fei, Nancy Kanwisher, Joshua B. Tenenbaum, Daniel L. K. Yamins, and Judith E. Fan. Physion: Evaluating physical predictio...

    2022 arXiv

  27. [35]

    Newton: Are large language models capable of physical reasoning?arXiv preprint arXiv:2310.07018, 2023

    Yi Ru Wang, Jiafei Duan, Dieter Fox, and Siddhartha Srinivasa. Newton: Are large language models capable of physical reasoning?arXiv preprint arXiv:2310.07018, 2023

    2023 arXiv

  28. [36]

    Krishnan

    Vahid Balazadeh, Mohammadmehdi Ataei, Hyunmin Cheong, Amir Hosein Khasahmadi, and Rahul G. Krishnan. Physics context builders: A modular framework for physical reasoning in vision-language models.arXiv preprint arXiv:2412.08619, 2025. 11

    2025

  29. [37]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. arXiv preprint arXiv:2401.12168, 2024

    2024 arXiv

  30. [38]

    Benchmarking sequential visual input reasoning and prediction in multimodal large language models.arXiv preprint arXiv:2310.13473, 2023

    Mingwei Zhu, Leigang Sha, Yu Shu, Kangjia Zhao, Tiancheng Zhao, and Jianwei Yin. Benchmarking sequential visual input reasoning and prediction in multimodal large language models.arXiv preprint arXiv:2310.13473, 2023

    2023 arXiv

  31. [39]

    Tenenbaum, Daniel LK Yamins, Judith E Fan, and Kevin A

    Hsiao-Yu Tung, Mingyu Ding, Zhenfang Chen, Daniel Bear, Chuang Gan, Joshua B. Tenenbaum, Daniel LK Yamins, Judith E Fan, and Kevin A. Smith. Physion++: Evaluating physical scene understanding that requires online inference of different physical properties.arXiv preprint arXiv:...

    2023 arXiv

  32. [40]

    Intphys: A framework and benchmark for visual intuitive physics reasoning.arXiv preprint arXiv:1803.07616, 2020

    Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. Intphys: A framework and benchmark for visual intuitive physics reasoning.arXiv preprint arXiv:1803.07616, 2020

    2020 arXiv

  33. [41]

    Visscience: An extensive benchmark for evaluating k12 educational multi-modal scientific reasoning.arXiv preprint arXiv:2409.13730, 2024

    Zhihuan Jiang, Zhen Yang, Jinhao Chen, Zhengxiao Du, Weihan Wang, Bin Xu, and Jie Tang. Visscience: An extensive benchmark for evaluating k12 educational multi-modal scientific reasoning.arXiv preprint arXiv:2409.13730, 2024

    2024 arXiv

  34. [42]

    Learn to explain: Multimodal reasoning via thought chains for science question answering.arXiv preprint arXiv:2209.09513, 2022

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering.arXiv preprint arXiv:2209.09513, 2022

    2022 arXiv

  35. [43]

    Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark.arXiv preprint arXiv:2501.05444, 2025

    Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark.arXiv preprint arXiv:2501.05444, 2025

    2025 arXiv

  36. [44]

    Physreason: A comprehensive benchmark towards physics-based reasoning.arXiv preprint arXiv:2502.12054, 2025

    Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang, and Jun Liu. Physreason: A comprehensive benchmark towards physics-based reasoning.arXiv preprint arXiv:2502.12054, 2025

    2025 arXiv

  37. [45]

    Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai.arXiv preprint arXiv:2312.16170, 2023

    Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, Xihui Liu, Cewu Lu, Dahua Lin, and Jiangmiao Pang. Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai.arXiv preprint arXiv:2312.16170, 2023

    2023 arXiv

  38. [46]

    Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie

    Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces.arXiv preprint arXiv:2412.14171, 2024

    2024 arXiv

  39. [47]

    An empirical analysis on spatial reasoning capabilities of large multimodal models.arXiv preprint arXiv:2411.06048, 2024

    Fatemeh Shiri, Xiao-Yu Guo, Mona Golestan Far, Xin Yu, Gholamreza Haffari, and Yuan-Fang Li. An empirical analysis on spatial reasoning capabilities of large multimodal models.arXiv preprint arXiv:2411.06048, 2024

    2024 arXiv

  40. [48]

    Proximity qa: Unleashing the power of multi-modal large language models for spatial proximity analysis.arXiv preprint arXiv:2401.17862, 2024

    Jianing Li, Xi Nan, Ming Lu, Li Du, and Shanghang Zhang. Proximity qa: Unleashing the power of multi-modal large language models for spatial proximity analysis.arXiv preprint arXiv:2401.17862, 2024

    2024 arXiv

  41. [49]

    Physbench: Benchmarking and enhancing vision-language models for physical world understanding.arXiv preprint arXiv:2501.16411, 2025

    Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang. Physbench: Benchmarking and enhancing vision-language models for physical world understanding.arXiv preprint arXiv:2501.16411, 2025

    2025 arXiv

  42. [50]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2024

    2024 arXiv

  43. [51]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  44. [52]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv preprint:2304.02643, 2023

    2023 arXiv

  45. [53]

    Personalize segment anything model with one shot.arXiv preprint arXiv:2305.03048, 2023

    Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Xianzheng Ma, Hao Dong, Peng Gao, and Hongsheng Li. Personalize segment anything model with one shot.arXiv preprint arXiv:2305.03048, 2023. 12

    2023 arXiv

  46. [54]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Seb...

    2022

  47. [55]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, DONGXU LI, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. InAdvances in Neural Information Processing Systems, volume 36, pages ...

    2023

  48. [56]

    3d-llm: Injecting the 3d world into large language models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. InAdvances in Neural Information Processing Systems, volume 36, pages 20482–20494, 2023

    2023

  49. [57]

    Pandagpt: One model to instruction- follow them all, 2023

    Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction- follow them all, 2023

    2023

  50. [58]

    Vita: Towards open-source interactive omni multimodal llm.arXiv preprint arXiv:2408.05211, 2024

    Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Shaoqi Dong, Xiong Wang, Di Yin, Long Ma, et al. Vita: Towards open-source interactive omni multimodal llm.arXiv preprint arXiv:2408.05211, 2024

    2024 arXiv

  51. [59]

    Vita-1.5: Towards gpt-4o level real-time vision and speech interaction.arXiv preprint arXiv:2501.01957, 2025

    Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long, Heting Gao, Ke Li, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction.arXiv preprint arXiv:2501.01957, 2025

    2025 arXiv

  52. [60]

    Vita-audio: Fast interleaved cross-modal token generation for efficient large speech-language model, 2025

    Zuwei Long, Yunhang Shen, Chaoyou Fu, Heting Gao, Lijiang Li, Peixian Chen, Mengdan Zhang, Hang Shao, Jian Li, Jinlong Peng, Haoyu Cao, Ke Li, Rongrong Ji, and Xing Sun. Vita-audio: Fast interleaved cross-modal token generation for efficient large speech-language model, 2025

    2025

  53. [61]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding

  54. [62]

    Spacevllm: Endowing multimodal large language model with spatio-temporal video grounding capability, 2025

    Jiankang Wang, Zhihan Zhang, Zhihang Liu, Yang Li, Jiannan Ge, Hongtao Xie, and Yongdong Zhang. Spacevllm: Endowing multimodal large language model with spatio-temporal video grounding capability, 2025

    2025

  55. [63]

    LLaMA-adapter: Efficient fine-tuning of large language models with zero-initialized attention

    Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. LLaMA-adapter: Efficient fine-tuning of large language models with zero-initialized attention. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  56. [64]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023

    2023

  57. [65]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023

    2023

  58. [66]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  59. [67]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  60. [68]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning.arXiv preprint arXiv:2310.09478, 2023

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechu Liu, Pengchuan Zhang, Raghuraman Krishnamoor- thi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning.arXiv preprint arXiv:2310...

    arXiv

  61. [69]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021

    2021

  62. [70]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models, 2024

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models, 2024

    2024

  63. [71]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Han...

    2025 arXiv

  64. [72]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding, 2024

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang,...

    2024

  65. [73]

    Kimi-vl technical report, 2025

    KimiTeam. Kimi-vl technical report, 2025

    2025

  66. [74]

    Seed1.5-vl technical report, 2025

    ByteDanceSeedTeam. Seed1.5-vl technical report, 2025

    2025

  67. [75]

    Skywork r1v2: Multimodal hybrid reinforcement learning for reasoning, 2025

    Chris, Yichen Wei, Yi Peng, Xiaokun Wang, Weijie Qiu, Wei Shen, Tianyidan Xie, Jiangbo Pei, Jianhao Zhang, Yunzhuo Hao, Xuchen Song, Yang Liu, and Yahui Zhou. Skywork r1v2: Multimodal hybrid reinforcement learning for reasoning, 2025

    2025

  68. [76]

    Physgame: Uncovering physical commonsense violations in gameplay videos

    Meng Cao, Haoran Tang, Haoze Zhao, Hangyu Guo, Jiaheng Liu, Ge Zhang, Ruyang Liu, Qiang Sun, Ian Reid, and Xiaodan Liang. Physgame: Uncovering physical commonsense violations in gameplay videos. arXiv preprint arXiv:2412.01800, 2024

    2024 arXiv

  69. [77]

    Embspatial-bench: Benchmarking spatial understanding for embodied tasks with large vision-language models, 2024

    Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. Embspatial-bench: Benchmarking spatial understanding for embodied tasks with large vision-language models, 2024

    2024

  70. [78]

    LLaV A-onevision: Easy visual task transfer.Transactions on Machine Learning Research, 2025

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaV A-onevision: Easy visual task transfer.Transactions on Machine Learning Research, 2025

    2025

  71. [79]

    Llava-cot: Let vision language models reason step-by-step.arXiv preprint arXiv:2411.10440, 2024

    Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step.arXiv preprint arXiv:2411.10440, 2024

    2024 arXiv

  72. [80]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi...

    2024 arXiv

  73. [81]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization.arXiv preprint arXiv:2411.10442, 2024

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization.arXiv preprint arXiv:2411.10442, 2024

    2024 arXiv

  74. [82]

    QVQ: To See the World with Wisdom, December 2024

    Qwen Team. QVQ: To See the World with Wisdom, December 2024

    2024

  75. [83]

    Claude 3.7 sonnet.https://claude.ai/new, 2025

    Anthropic. Claude 3.7 sonnet.https://claude.ai/new, 2025

    2025

  76. [84]

    Grok 3.https://grok.com, 2025

    xAI. Grok 3.https://grok.com, 2025

    2025

  77. [85]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s pe...

    2024 arXiv

  78. [86]

    Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning.arXiv preprint arXiv:2503.07365, 2025

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning.arXiv preprint arXiv:2503.07365, 2025

    2025 arXiv

  79. [87]

    R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization.arXiv preprint arXiv:2503.12937, 2025

    Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization.arXiv preprint arXiv:2503.12937, 2025

    2025 arXiv

  80. [88]

    Lighthouse Laboratory

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran As...

    2022

  81. [89]

    How to accelerate the separation of B peels?

    <image_1> Single-image Input : <image_1> Question: From A's perspective, which direction is B <image_5> <image_2> <image_4> <image_3> Figure 16Examples of Single-Image SAS–KSC Evaluation on Spatial Relations Data with GPT-4o. 28 Response : GPT-4o: Step Partition & Judgment: To...

  82. [92]

    Match": Aligns with ground truth -

    Background Information Steps - External knowledge or question context - No inference process involved ## Step Requirements - Each step must be atomic (one conclusion per step) - No content duplication across steps - Initial analysis counts as background information - Final ans...

  83. [93]

    The output format MUST be in valid JSON format without ANY other content

  84. [94]

    For highly repetitive patterns, output it as a single step

  85. [95]

    step_type

    Output maximum 35 steps. Always include the final step that contains the answer. Here is the json output format: ## Output Format [ {{ "step_type": "image description|logical inference|background information", "premise": "Evidence (only for logical inference)", "conclusion": "...

  86. [96]

    Problem: The original question/task

  87. [97]

    A Solution of a model

  88. [98]

    step_index

    Ground Truth: Essential steps required for a correct answer MATCHING PROCESS: You need to match each ground truth middle step with the solution: Match Criteria: - The middle step should exactly match in the content or is directly entailed by a certain content in the solution -...

  89. [99]

    Only output the json array with no additional information

  90. [100]

    Judge each ground truth middle step in order without omitting any step. Here is the problem，answer, solution, and the ground truth middle steps: [Problem] {question} [Answer] {answer} [Solution] {solution} [Ground Truth Information] {gt_annotation} G.2 CoT Diversity Evaluation...

  91. [101]

    Question: The original question/task

  92. [102]

    Final Answer: Answer to the original question

  93. [103]

    key_step_1

    Key Reasoning Steps: A list of essential reasoning steps, each with an ID and explanation. Output format JSON array of judgments: [ ["key_step_1", "key_step_2", "key_step_3"], ["key_step_1", "key_step_3", "key_step_2"] ] ADDITIONAL RULES:

  94. [104]

    Only output the json array with no additional information. Here is the question, answer, and the Key Reasoning Steps: [Question] {question} [Final Answer] {answer} [Solution] {solution} Diversity Evaluation Prompt Diversity Evaluation Prompt 38 G.3 CoT Efficiency Evaluation Pr...

  95. [105]

    Logical Inference Steps - Contains exactly one logical deduction - Must produce a new derived conclusion - Cannot be just a summary or observation

  96. [106]

    Image Description Steps - Pure visual observations - Only includes directly visible elements - No inferences or assumptions

  97. [107]

    step_type

    Background Information Steps - External knowledge or question context - No inference process involved ## Step Requirements - Each step must be atomic (one conclusion per step) - No content duplication across steps - Initial analysis counts as background information - Final ans...

  98. [108]

    Reach accurate conclusions aligned with ground truth

  99. [109]

    Invalid reflections include:

    Use new insights to find the mistake of the previous conclusion or verify its correctness. Invalid reflections include:

  100. [110]

    Repetition - Restating previous content or method without new insights

  101. [111]

    Wrong Conclusion - Reaching incorrect conclusions vs ground truth

  102. [112]

    Incompleteness - Proposing but not executing new analysis methods

  103. [113]

    Other - Additional error types # Input Format ``` [Problem] {question} [Solution] {solution} [Ground Truth] {gt_annotation} ``` # Output Requirements

  104. [114]

    The output format must be in valid JSON format without any other content

  105. [115]

    conclusion

    Output maximum 30 reflection steps. Here is the json output format: ## Output Format ```json [ {{ "conclusion": "One-sentence summary of reflection outcome", "judgment": "Correct|Wrong", "error_type": "N/A|Repetition|Wrong Conclusion|Incompleteness|Other" }} ] ``` # Rules

  106. [116]

    Preserve original content and order

  107. [117]

    No new interpretations

  108. [118]

    Include ALL reflection steps

  109. [119]

    Empty list if no reflections found

  110. [120]

    Direct JSON output without any other output 41 H Setup H.1 Experiment Setup Evaluation Models.To comprehensively assess performance on MVPBench, we selected a di- verse array of multimodal large language models (MLLMs), encompassing both open-source and closed-source framework...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.