Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Current multimodal AI models score near-random on a new spatial-planning benchmark, with all 12 failing the harder 3D cube-assembly task.

desk verdict MARBLE is a genuinely new benchmark worth referee time, but the 0% on M-Cube headline is statistically empty; the real difficulty signal is in CUBE-easy and the perception task. read the letter →

arxiv 2506.22992 v1 pith:7OHBPFXN submitted 2025-06-28 cs.AI cs.CLcs.CV

Yulun Jiang , Yekun Chai , Maria Brbić , Michael Moor This is my paper
classification cs.AIcs.CLcs.CV
keywords multimodalreasoningspatialplanningMLLMevaluationPortal2HappyCubeperceptionbottleneckchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that state-of-the-art multimodal language models cannot yet perform complex, multi-step spatial reasoning and planning when the input is genuinely multimodal. It introduces MARBLE, a benchmark with two tasks: M-Portal, which asks models to judge or complete long plans for Portal 2 puzzles, and M-Cube, which asks models to assemble six 3D jigsaw pieces into a cube. On M-Portal all 12 tested models score near the random baseline on the harder binary task, and on M-Cube all score 0%; only simplified subtasks separate the strongest models, with GPT-o3 reaching 17.6% and 72%. The paper also shows perception is a bottleneck, since models fail even at transcribing a rendered piece into a grid. If the results hold, the benchmark gives the community a way to measure progress in multimodal reasoning separately from visual perception and final-answer retrieval.

What carries the argument

The machinery is the benchmark itself: M-Portal and M-Cube. M-Portal takes 16 community-made Portal 2 maps, each with human-annotated ground-truth chain-of-thought solutions and five deliberately mistaken steps; these are combined into 32 candidate plans for the plan-correctness binary task and into masked fill-the-blanks questions with 2n options. M-Cube disassembles a 5x5x5 cube into six synthetic jigsaw pieces rendered from random viewpoints, and a solution validator decides whether a proposed face assignment and orientation actually closes the cube; the full task has 6! times $8^{6}$ candidate assemblies, while CUBE-easy provides the pieces as 2D arrays, forbids flips, and pre-places four pieces, leaving 32 candidates. These controls let the benchmark separate perception (reading the image) from reasoning (searching the candidate space).

What would settle it

Take a MARBLE CUBE instance and hand-assemble the six pieces physically; if a human can assemble a configuration that the validator rejects, the reported 0% result is an artifact of the evaluation. Alternatively, run any current MLLM with the image replaced by the exact 2D arrays and ask for the assembly; if it solves a full CUBE instance, the claim that current models score 0% would fall.

Watch

Extended reading notes

Core claim

The paper claims that MARBLE, a new benchmark of two tasks, shows current MLLMs cannot do complex multimodal spatial reasoning and planning. On M-Portal, all 12 models score near the random baseline on plan correctness (minority-class F1 around 6%) and at best 17.6% on fill-the-blanks (GPT-o3). On M-Cube, every model scores 0% on the full CUBE task; on the simplified CUBE-easy, only GPT-o3 does well, at 72%, while most other models remain near random. The paper argues this failure is not just reasoning: even converting a rendered jigsaw piece into a 5x5 bump/gap array yields only about 70% per-cell and 0% whole-piece accuracy, so perception is a separate bottleneck.

Load-bearing premise

The load-bearing premise is that the benchmark's ground truth is correct: the human-annotated Portal solutions and mistaken steps and the M-Cube solution validator must accurately reflect which plans and assemblies actually work.

Editorial extensions

If this is right

  • If the MARBLE results are right, near-random performance on M-Portal and 0% on CUBE mean existing MLLMs cannot yet produce or verify long, physically grounded plans from visual input.
  • The perception bottleneck suggests that improving MLLMs' ability to extract structured information from images, such as converting a 3D piece into a 5x5 array, is a prerequisite for progress on these tasks.
  • The fill-the-blanks result, around 70% with one blank but below 1% with four or more, implies that current models can do local reasoning but fail to integrate long chains of interdependent steps.
  • The validator-assisted setup shows that diagnostic feedback helps on the simplified task (GPT-o4-mini rises from 10% to 28% over five rounds) but does not yet transfer to the full task.
  • A benchmark that rewards the reasoning process rather than the final answer gives future model development a concrete target for multi-step, multimodal planning evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer from the one-blank versus many-blank gap that M-Portal is less a test of physics knowledge than of compositional planning over many coupled decisions; a model that plans hierarchically might do well without new visual abilities.
  • A testable extension the paper leaves implicit is to use the M-Cube solution validator at training time as a reward signal for reinforcement learning, since the validator already gives dense edge-conflict feedback.
  • The perception failure on a seemingly trivial task (76% per-cell, 0% per-piece accuracy) suggests that 3D rendering-to-array transcription, not reasoning, may be the binding constraint; replacing images with exact arrays should isolate reasoning and could raise CUBE scores.
  • Future work could combine the validator with interleaved thinking and tool use, which the paper notes many current reasoning models do not retain across rounds, making the interactive setup a natural testbed for agentic reasoning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper presents MARBLE, a benchmark aimed at evaluating multimodal spatial reasoning and planning in large multimodal language models. It consists of two tasks: M-Portal, derived from Portal 2 puzzles, with plan-correctness and fill-the-blanks subtasks, and M-Cube, a 3D jigsaw assembly task with a simplified CUBE-easy subtask. The authors evaluate 12 models, including open- and closed-weight MLLMs and text-only LLMs. They report near-random F1 on plan correctness, accuracy between 0% and 17.6% on fill-the-blanks, 0% accuracy on the full CUBE task, and between 0.3% and 72% on CUBE-easy. Additional experiments probe perception accuracy and a validator-assisted iterative refinement setup. The paper's central claim is that state-of-the-art models still lack the multi-step multimodal reasoning required by these tasks.

Significance. MARBLE addresses a genuine gap in multimodal reasoning evaluation by focusing on multi-step planning under spatial and physical constraints rather than single-step question answering. The benchmark design is thoughtful: the M-Cube-easy results provide a clear performance gradient (GPT-o3 at 72% versus the best other model at 16%), and the perception experiment isolates a measurable bottleneck. The data generation pipeline and the human-annotated Portal solutions are potentially reusable resources. However, the paper's headline statistical claims—'near-random performance on M-Portal' and '0% accuracy on M-Cube'—are not adequately supported as stated: the fill-the-blanks results are far above random for several models, and the 0% CUBE result lacks the statistical power to distinguish any non-random ability from chance. With the revisions described below, the benchmark would be a useful contribution.

major comments (4)
  1. [Abstract, Section 3.4] The statement that 'all the 12 advanced models obtain near-random performance on M-Portal' is not supported by the reported fill-the-blanks accuracy. In Table 2, the random baseline is 3e-3, while GPT-o3 reaches 17.6%, Gemini-2.5-pro 16.1%, DeepSeek-R1-0528 8.4%, and Claude-3.7-Sonnet 6.8%—several orders of magnitude above random. The near-random claim is only true for the plan-correctness F1, whose interpretation is compromised by class imbalance (see next comment). Please revise the abstract and the corresponding discussion to state the results separately for the two subtasks, and do not describe the fill-the-blanks performance as near-random.
  2. [Section 3.1] The plan-correctness task has an extreme class imbalance (1 correct candidate versus 31 incorrect per map). Under such imbalance, the F1 score for the positive class is highly insensitive: a model with 50% recall and 50% specificity on the 32 candidates would yield F1 approximately 6%, statistically indistinguishable from the random baseline. Thus the reported near-random F1 does not rule out partial reasoning ability. The authors should either report a metric that is informative under class imbalance (e.g., AUROC, or partial credit for detecting the presence or absence of mistakes), or explicitly state the diagnostic limits of this task and base the 'near-random M-Portal' claim primarily on fill-the-blanks.
  3. [Section 3.5, Table 3, Appendix B] The 0% accuracy on CUBE is used as headline evidence of complete failure, but this result has insufficient statistical power to support that interpretation. For closed-source models evaluated on 200 examples (Appendix B), the 95% Clopper-Pearson interval for 0/200 is [0, 1.8%], which is orders of magnitude above the random baseline (approximately 1e-5). For open-source models evaluated on 1,000 examples, the interval is [0, 0.37%]. In both cases, observing zero successes is exactly what would be expected under random guessing, and the upper confidence bound does not distinguish zero ability from a success rate well above random. The authors should report binomial confidence intervals, avoid phrasing such as 'completely fail,' and rely on the CUBE-easy results (which show a clear contrast between GPT-o3 and other models) as the primary evidence of difficulty. The 0% result can be reported as 'no successes in N trials,' but not as evidence that models are at or below random.
  4. [Section 3.2] The correctness of the M-Cube evaluation rests entirely on the solution validator, but the paper gives no description of its algorithm, its handling of rotational symmetries and edge complementarity, or any validation that it accepts all valid solutions and rejects all invalid ones. If the validator has false negatives, the reported 0% on CUBE could be an artifact. Please provide a detailed specification of the validator and report a sanity check on a set of known-valid and known-invalid configurations (e.g., generated from the ground truth) to demonstrate that it has no false positives or false negatives.
minor comments (6)
  1. [Figure 5] The 0% whole-piece accuracy is a deterministic consequence of the per-cell accuracy (0.7^25 is approximately 0.0008) and is not an independent finding; please report per-cell accuracy as the primary metric and describe the 0% as a derived quantity.
  2. [Appendix B] The selection procedure for the '200 representative examples' used for closed-source models on M-Cube is not described. Please specify how these were sampled (e.g., stratified by difficulty) and confirm that the subset is balanced across puzzle configurations.
  3. [Section 3.5] The sentence 'despite more than 10,000 tokens spent on thinking' is contradicted by Table 3, where several models (Llama-4-Scout, Qwen2.5-VL, GPT-4o, InternVL3-78B, GPT-o4-mini, GPT-o3) use fewer than 2,000 tokens on CUBE. Please correct or qualify this statement.
  4. [Table 3] The units for the random baseline '1e-5' are ambiguous (fraction vs. percentage). Please clarify the caption and ensure consistency with the text.
  5. [Table 4 and Section 3.3] The model name appears inconsistently as 'GPT-4o-mini' (Appendix B) and 'GPT-o4-mini' (main text and Table 3). Please standardize.
  6. [References] The reference list contains a duplicate entry for MathVista ([15] and [16] are the same paper).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MARBLE is an empirical benchmark evaluation with no derivation that reduces to its own inputs.

full rationale

The paper constructs two new benchmark tasks (M-Portal and M-Cube) with human-annotated or synthetically generated ground truth and independently defined random baselines, then measures MLLM performance against them. No parameter is fitted to model outputs and then re-reported as a prediction, and no load-bearing claim is imported from a self-citation chain. The central negative result (models near random on M-Portal, 0% on CUBE) is a contingent empirical finding rather than a consequence of the benchmark's definition. The M-Cube solution validator checks geometric validity of proposed assemblies rather than comparing against a fitted target, and the random baseline (1e-5 for CUBE, 3.1% for CUBE-easy) is computed from the search space independently of model behavior. The only substantive concern is statistical, not circular: closed-source models are evaluated on 200 representative CUBE examples (Appendix B), so 0 successes in 200 trials has a 95% Clopper-Pearson upper bound of about 1.8%, and the headline 0% figure cannot by itself distinguish model failure from the 1e-5 uniform-random rate. However, low statistical power is a measurement-decisiveness issue, not a reduction of the claim to its inputs.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims are empirical. The main assumptions are that the benchmark's ground truth is correct: the Portal solutions and mistakes are accurately annotated on the basis of one expert's judgment, the cube solution validator correctly checks assembly constraints, and the random baselines are computed correctly. No free parameters or invented entities are introduced.

assumptions (3)
  • domain assumption Human-annotated solutions for M-Portal maps are correct, and the five mistaken steps per map are truly incorrect.
    The dataset generation relies on a single annotator's judgment of Portal 2 puzzle solutions; no inter-annotator agreement or independent verification is reported.
  • domain assumption The M-Cube solution validator correctly determines whether a proposed assembly forms a perfect 5x5x5 cube.
    The validator is a program that checks edge complementarity; its correctness is not formally verified or tested against a ground-truth set.
  • domain assumption The random baseline for CUBE-easy (3.1%) is computed correctly from the search space of 32 valid solutions.
    The paper states CUBE-easy has 32 possible solutions; this count underpins the random baseline but the derivation is not shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning." pith.science (2026). https://pith.science/paper/7OHBPFXN

@misc{pith2026250622992,
  author       = {Pith},
  title        = {Pith review of: MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7OHBPFXN}},
  note         = {Machine review of arXiv:2506.22992}
}
read the original abstract

The ability to process information from multiple modalities and to reason through it step-by-step remains a critical challenge in advancing artificial intelligence. However, existing reasoning benchmarks focus on text-only reasoning, or employ multimodal questions that can be answered by directly retrieving information from a non-text modality. Thus, complex reasoning remains poorly understood in multimodal domains. Here, we present MARBLE, a challenging multimodal reasoning benchmark that is designed to scrutinize multimodal language models (MLLMs) in their ability to carefully reason step-by-step through complex multimodal problems and environments. MARBLE is composed of two highly challenging tasks, M-Portal and M-Cube, that require the crafting and understanding of multistep plans under spatial, visual, and physical constraints. We find that current MLLMs perform poorly on MARBLE -- all the 12 advanced models obtain near-random performance on M-Portal and 0% accuracy on M-Cube. Only in simplified subtasks some models outperform the random baseline, indicating that complex reasoning is still a challenge for existing MLLMs. Moreover, we show that perception remains a bottleneck, where MLLMs occasionally fail to extract information from the visual inputs. By shedding a light on the limitations of MLLMs, we hope that MARBLE will spur the development of the next generation of models with the ability to reason and plan across many, multimodal reasoning steps.

Figures

Figures reproduced from arXiv: 2506.22992 by the authors.

Figure 1
Figure 1. Overview of the M-Portal Dataset of the MARBLE Benchmark. Illustrated is a rather basic level Portal 2 problem, which only requires seven steps to solve. For comparison, the advanced problems introduced in this benchmark may involve several dozens of steps. Also, steps are not always decomposed into their most atomic form to keep enough complexity within a step to make mistaken steps harder to detect. Appendix A pro… view at source ↗
Figure 2
Figure 2. Data generation and evaluation pipeline for the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Overview of the M-Cube workflow including data generation, problem rendering, as well as solution validation. Appendix A provides more dataset examples. Evaluation subtasks. To measure the performance of MLLMs with controlled difficulty level, we create two subtasks called CUBE and CUBE-easy. Each subtask contains 1000 examples. CUBE-easy is a simplified version of CUBE along three axes: i) the input pieces are repr… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The influence of number of blanks to M-Portal. Influence of blanks. In the fill-the-blanks task on M-Portal, each question contains multiple steps in the complete solution, and part of them are masked. To sys￾tematically understand the impact of missing information, we…
Figure 5
Figure 5. Figure 5: Perception remains a bottleneck for M-Cube. Left: A perception task designed to test MLLM’s ability on retrieve structured information from visual input (full prompt in Appendix A) and example response of an MLLM. Right: Performance of 8 MLLMs on this perception task b…
Figure 6
Figure 6. Figure 6: Search space of the M-Cube dataset under different configurations. 1 2 3 4 5 6 # Missing pieces 0 10 20 30 40 50 Accuracy (%) Performance Comparison Flip needed Flip ignored [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 8
Figure 8. Figure 8: Performance of GPT-o4-mini on CUBE-easy with binary or detailed feedback from solution validator. On CUBE, the performance will remain 0%. the performance using the solution validator tool remains 0% regardless of the feedback type, highlighting the lim￾itation of curr…
Figure 9
Figure 9. Figure 9: Overview of the Portal-2 Dataset of the MARBLE-Benchmark. Illustrated is a rather [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Illustration of an example problem of the [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: presents a complete example question of M-Cube task, and the solution to the instance with the corresponding 2D and 3D visualization [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Prompt for evaluating the perception ability of MLLMs on [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Example of MLLM using solution validator as a tool to gather feedback and iteratively [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Supervised fine-tuning collapses LLM action diversity in board-game play beyond what the accuracy–diversity tradeoff requires; augmenting SFT data with all optimal actions per state partially prevents this.

Reference graph

Works this paper leans on

38 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    Szilárd András, Kinga Sipos, and Anna Soós.Which is harder?-Classification of Happy Cube puzzles. 2013

    work page 2013

  2. [2]

    Anthropic 3.7 Sonnet and Claude Code, February 2025

    Anthropic. Anthropic 3.7 Sonnet and Claude Code, February 2025. URL https://www. anthropic.com/news/claude-3-7-sonnet

    work page 2025

  3. [3]

    Qwen2.5-VL technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  4. [4]

    MEGA-Bench: Scaling multimodal evaluation to over 500 real-world tasks.arXiv preprint arXiv:2410.10563, 2024

    Jiacheng Chen, Tianhao Liang, Sherman Siu, Zhengqing Wang, Kai Wang, Yubo Wang, Yuan- sheng Ni, Wang Zhu, Ziyan Jiang, Bohan Lyu, et al. MEGA-Bench: Scaling multimodal evaluation to over 500 real-world tasks.arXiv preprint arXiv:2410.10563, 2024

    arXiv

  5. [5]

    PuzzleVQA: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns.arXiv preprint arXiv:2403.13315, 2024

    Yew Ken Chia, Vernon Toh Yan Han, Deepanway Ghosal, Lidong Bing, and Soujanya Poria. PuzzleVQA: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns.arXiv preprint arXiv:2403.13315, 2024

    arXiv 2024

  6. [6]

    Arc prize 2024: Technical report.arXiv preprint arXiv:2412.04604, 2024

    Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. Arc prize 2024: Technical report.arXiv preprint arXiv:2412.04604, 2024. 12

    arXiv 2024

  7. [7]

    Gemini 2.5: Our most intelligent ai model, March 2025

    Google DeepMind. Gemini 2.5: Our most intelligent ai model, March 2025. URLhttps://blog. google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/ #gemini-2-5-thinking

    work page 2025

  8. [8]

    Deepseek-R1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-R1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

Show all 38 references
  1. [9]

    Can MLLMs reason in multimodality? EMMA: an enhanced multimodal reasoning benchmark.arXiv preprint arXiv:2501.05444, 2025

    Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can MLLMs reason in multimodality? EMMA: an enhanced multimodal reasoning benchmark.arXiv preprint arXiv:2501.05444, 2025

    2025 arXiv

  2. [10]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  3. [11]

    Openai o1 system card.CoRR, abs/2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya K...

    2024 doi arXiv

  4. [12]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems ...

    2023

  5. [13]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  6. [14]

    Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 2022

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 2022

    2022

  7. [15]

    MathVista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023. 13

    arXiv

  8. [16]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. InInternational Conference on Learning Representati...

    2023

  9. [17]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, April

    Meta. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, April

  10. [18]

    Introducing OpenAI o3 and o4-mini, April 2025

    OpenAI. Introducing OpenAI o3 and o4-mini, April 2025. URLhttps://openai.com/index/ introducing-o3-and-o4-mini/

    2025

  11. [19]

    Balrog: Bench- marking agentic llm and vlm reasoning on games.arXiv preprint arXiv:2411.13543, 2024

    Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, et al. Balrog: Bench- marking agentic llm and vlm reasoning on games.arXiv preprint arXiv:2411.13543, 2024

    2024 arXiv

  12. [20]

    Vision language models are blind

    Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. InProceedings of the Asian Conference on Computer Vision, 2024

    2024

  13. [21]

    Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems, 36: 68539–68551, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems, 36: 68539–68551, 2023

    2023

  14. [22]

    Seed1.5-VL technical report.arXiv preprint arXiv:2505.07062, 2025

    ByteDance Seed Team. Seed1.5-VL technical report.arXiv preprint arXiv:2505.07062, 2025

    2025 arXiv

  15. [23]

    Eyes wide shut? Exploring the visual shortcomings of multimodal LLMs

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? Exploring the visual shortcomings of multimodal LLMs. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    2024

  16. [24]

    Evaluating large language models with grid-based game competitions: an extensible llm benchmark and leaderboard.arXiv preprint arXiv:2407.07796, 2024

    Oguzhan Topsakal, Colby Jacob Edell, and Jackson Bailey Harper. Evaluating large language models with grid-based game competitions: an extensible llm benchmark and leaderboard.arXiv preprint arXiv:2407.07796, 2024

    2024 arXiv

  17. [25]

    Multimodal chain-of-thought reasoning: A comprehensive survey.arXiv preprint arXiv:2503.12605, 2025

    Yaoting Wang, Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, and Hao Fei. Multimodal chain-of-thought reasoning: A comprehensive survey.arXiv preprint arXiv:2503.12605, 2025

    2025 arXiv

  18. [26]

    How do multimodal large language models handle complex multimodal reasoning? placing them in an extensible escape game.arXiv preprint arXiv:2503.10042, 2025

    Ziyue Wang, Yurui Dong, Fuwen Luo, Minyuan Ruan, Zhili Cheng, Chi Chen, Peng Li, and Yang Liu. How do multimodal large language models handle complex multimodal reasoning? placing them in an extensible escape game.arXiv preprint arXiv:2503.10042, 2025

    2025 arXiv

  19. [27]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 2022

    2022

  20. [28]

    A surprising failure? multimodal llms and the NLVR challenge.arXiv preprint arXiv:2402.17793, 2024

    Anne Wu, Kianté Brantley, and Yoav Artzi. A surprising failure? multimodal llms and the NLVR challenge.arXiv preprint arXiv:2402.17793, 2024

    2024 arXiv

  21. [29]

    Multimodal inconsistency reasoning (mmir): A new benchmark for multimodal reasoning models.arXiv preprint arXiv:2502.16033, 2025

    Qianqi Yan, Yue Fan, Hongquan Li, Shan Jiang, Yang Zhao, Xinze Guan, Ching-Chen Kuo, and Xin Eric Wang. Multimodal inconsistency reasoning (mmir): A new benchmark for multimodal reasoning models.arXiv preprint arXiv:2502.16033, 2025

    2025 arXiv

  22. [30]

    MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on Co...

    2024

  23. [31]

    Critic-V: VLM critics help catch vlm errors in multimodal reasoning

    Di Zhang, Jingdi Lei, Junxian Li, Xunzhi Wang, Yujie Liu, Zonglin Yang, Jiatong Li, Weida Wang, Suorong Yang, Jianbo Wu, et al. Critic-V: VLM critics help catch vlm errors in multimodal reasoning. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 9...

    2025

  24. [32]

    V-mage: A game evaluation framework for assessing visual-centric capabilities in multimodal large language models.arXiv preprint arXiv:2504.06148, 2025

    Xiangxi Zheng, Linjie Li, Zhengyuan Yang, Ping Yu, Alex Jinpeng Wang, Rui Yan, Yuan Yao, and Lijuan Wang. V-mage: A game evaluation framework for assessing visual-centric capabilities in multimodal large language models.arXiv preprint arXiv:2504.06148, 2025

    2025 arXiv

  25. [33]

    MiniGPT-4: En- hancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: En- hancing vision-language understanding with advanced large language models. InThe Twelfth International Conference on Learning Representations

  26. [34]

    Step 1: Go to room 2 (on the right) and press the switch to drop a cube

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 15 A Illustration o...

    2025 arXiv

  27. [36]

    Hold the finished cube in both hands

  28. [37]

    Turn the whole cube until the face you are about to name is now facing you like the front of a box

  29. [38]

    Here is the converted 2D array: [array]

    In this view, write down the edge numbers located on left and top. • Format: (Cube-Face, Left-Edge#, Top-Edge#) Note that left-edge and top-edge must be adjacent. OUTPUT Answer the question with the arrangement of 6 pieces. Your response should end with: The final solution is ...

    2025

  30. [2025]

    URLhttps://ai.meta.com/blog/llama-4-multimodal-intelligence/

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.