arxiv: 2604.06725 · v1 · submitted 2026-04-08 · 💻 cs.CV

Recognition: 2 theorem links

· Lean Theorem

Enhancing MLLM Spatial Understanding via Active 3D Scene Exploration for Multi-Perspective Reasoning

Jiahua Chen , Qihong Tang , Weinong Wang , Qi Fan

Authors on Pith no claims yet

Pith reviewed 2026-05-10 19:11 UTC · model grok-4.3

classification 💻 cs.CV

keywords spatial reasoning3D mesh reconstructionnovel view synthesisvisual chain of thoughtmultimodal large language modelstraining freeperspective takingactive exploration

0 comments

The pith

A training-free 3D reconstruction pipeline with novel view synthesis improves spatial reasoning in multimodal large language models

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to solve the problem of poor 3D spatial reasoning in multimodal large language models that stems from their reliance on 2D images. It does this by proposing a training-free approach that first builds a 3D mesh of the scene and then generates new viewpoints to allow the model to reason from multiple angles. This process is presented as a Visual Chain-of-Thought that emulates human exploration of a space. A sympathetic reader would care because it promises better performance on spatial tasks without the need for costly retraining on 3D datasets.

Core claim

The authors claim that by reconstructing a high-fidelity 3D mesh from a single image via MLLM-guided keyword extraction and multi-granularity mask generation, and then using an external knowledge base to select optimal camera parameters for synthesizing novel views, their framework introduces an effective Visual Chain-of-Thought for multi-perspective reasoning that enhances spatial comprehension.

What carries the argument

Visual Chain-of-Thought mechanism grounded in explicit 3D reconstruction, where the 3D mesh enables iterative synthesis of novel views through computed camera extrinsics.

If this is right

Spatial comprehension improves by considering multiple perspectives rather than a single view.
The approach works with existing models without any post-training.
It provides viewpoint flexibility missing in rigid tool-calling methods.
Explicit geometric understanding replaces reliance on 2D visual priors.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the method works, it indicates that explicit 3D geometry can substitute for learned 3D understanding in these models.
The technique might be extended to handle video or dynamic scenes by updating the 3D reconstruction over time.
Replacing the external knowledge base with a learned component could make the system fully self-contained.
This could connect to problems in robotics where agents need to plan movements based on spatial relations.

Load-bearing premise

The MLLM can accurately extract keywords and generate masks at multiple scales to create a reliable 3D mesh, while the knowledge base can compute camera parameters that produce informative new views for reasoning.

What would settle it

Running the framework on a benchmark and finding that accuracy on spatial questions does not increase when novel views are provided compared to the original single image input.

Figures

Figures reproduced from arXiv: 2604.06725 by Jiahua Chen, Qi Fan, Qihong Tang, Weinong Wang.

**Figure 1.** Figure 1: Effectiveness of the proposed Visual CoT method driven by 3D reconstruction. Vanilla MLLMs (left) often fail to answer complex spatial questions correctly, including height comparisons, location assessments, and facing orientations, due to perspective ambiguities and reliance on 2D visual priors derived from a single static image. In contrast, the proposed approach (right) reconstructs the underlying 3D sc… view at source ↗

**Figure 2.** Figure 2: Architecture of the proposed end-to-end framework for spatial comprehension, comprising the 3D reconstruction and perspective transformation modules. The reconstruction phase employs MLLMs to extract multi-granularity keywords, guiding mask generation and IoU deduplication to construct a threedimensional mesh. The perspective transformation phase integrates an external knowledge base for optimal viewpoin… view at source ↗

**Figure 3.** Figure 3: Distribution of sub-task types across different benchmarks (3DSR, Rel3D, and SpatialScore). The horizontal axis indicates the relative proportion of each task category, whereas the absolute number of samples (N) for each benchmark is annotated on the right. A significant bias towards the Orientation task is observable in the SpatialScore dataset. The abbreviations for the sub-tasks denote the following: He… view at source ↗

**Figure 5.** Figure 5: The reasoning pipeline of the proposed method on two representative examples from the dataset. Each box presents the output information from the MLLM alongside the final answer to the question. precise spatial localization. In the ablation study, Step 2 is modified to allow the model to directly predict the yaw, pitch, and camera-to-scene distance required to transform a side view to the target viewpoint. … view at source ↗

read the original abstract

Although Multimodal Large Language Models have achieved remarkable progress, they still struggle with complex 3D spatial reasoning due to the reliance on 2D visual priors. Existing approaches typically mitigate this limitation either through computationally expensive post-training procedures on limited 3D datasets or through rigid tool-calling mechanisms that lack explicit geometric understanding and viewpoint flexibility. To address these challenges, we propose a \textit{training-free} framework that introduces a Visual Chain-of-Thought mechanism grounded in explicit 3D reconstruction. The proposed pipeline first reconstructs a high-fidelity 3D mesh from a single image using MLLM-guided keyword extraction and mask generation at multiple granularities. Subsequently, the framework leverages an external knowledge base to iteratively compute optimal camera extrinsic parameters and synthesize novel views, thereby emulating human perspective-taking. Extensive experiments demonstrate that the proposed approach significantly enhances spatial comprehension. Specifically, the framework outperforms specialized spatial models and general-purpose MLLMs, including \textit{GPT-5.2} and \textit{Gemini-2.5-Flash}, on major benchmarks such as 3DSRBench and Rel3D.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The training-free pipeline using MLLM-guided single-image 3D reconstruction plus knowledge-base view synthesis is a reasonable direction but the central claims cannot be evaluated without geometry accuracy metrics or experimental details.

read the letter

The paper introduces a training-free framework for improving spatial reasoning in multimodal large language models by reconstructing a 3D scene from a single image and generating novel views through an external knowledge base. This setup aims to create a visual chain-of-thought that mimics human perspective-taking without any model fine-tuning. What stands out is the specific pipeline: MLLM-guided keyword extraction and multi-granularity mask generation for mesh building, followed by iterative camera parameter computation from a knowledge base to synthesize useful new perspectives. This differs from the usual post-training on 3D data or simple tool-calling approaches mentioned in the abstract. It does a decent job of targeting a known weakness in MLLMs, where they struggle with complex 3D relations because they stay stuck in 2D image features. The soft spot is the reliance on high-fidelity 3D reconstruction from one view. Single-image mesh creation is notoriously difficult, and without any reported metrics on reconstruction accuracy such as Chamfer distance or depth errors on the test scenes, it's unclear if the synthesized views are geometrically consistent with the input. The abstract says the method beats specialized spatial models and general MLLMs like GPT-5.2 and Gemini-2.5-Flash on 3DSRBench and Rel3D, but the lack of experimental details, baselines, or controls in the summary makes it difficult to attribute any gains to the 3D exploration rather than other factors. If the mesh has errors, the whole visual reasoning chain falls apart. This work would interest researchers focused on enhancing MLLMs for robotics, AR, or scene understanding tasks. A reader looking for practical, training-free alternatives to fine-tuning would get some ideas from it, though they would need to see the full results to decide on its reliability. I recommend sending it for peer review because the core problem is important and the proposed direction is worth testing, even if the current evidence needs strengthening with geometry validation and more detailed ablations.

Referee Report

3 major / 2 minor

Summary. The manuscript proposes a training-free framework to improve MLLMs' 3D spatial reasoning by introducing a Visual Chain-of-Thought grounded in explicit 3D reconstruction. From a single image, MLLM-guided keyword extraction and multi-granularity mask generation are used to produce a high-fidelity 3D mesh; an external knowledge base then iteratively computes optimal camera extrinsic parameters to synthesize novel views that emulate human perspective-taking. The authors state that extensive experiments show the approach significantly outperforms specialized spatial models and general-purpose MLLMs (including GPT-5.2 and Gemini-2.5-Flash) on benchmarks such as 3DSRBench and Rel3D.

Significance. If the reconstruction fidelity and downstream gains can be substantiated, the work would be significant as a training-free alternative to post-training or rigid tool-calling for spatial understanding in MLLMs. It explicitly links geometric 3D exploration to multi-perspective reasoning, which could advance methods that currently rely on 2D priors. The approach also demonstrates a concrete pipeline for active viewpoint selection, but its impact depends on whether the single-image mesh is accurate enough to support geometrically consistent novel views.

major comments (3)

[Abstract] Abstract: the central claim that 'extensive experiments demonstrate that the proposed approach significantly enhances spatial comprehension' and 'outperforms ... on major benchmarks such as 3DSRBench and Rel3D' is unsupported by any quantitative results, tables, baselines, error bars, or controls. Without these data it is impossible to verify the magnitude of improvement or rule out confounds such as prompt engineering.
[Proposed Pipeline] Proposed pipeline (3D reconstruction step): the framework rests on the assumption that MLLM-guided keyword extraction plus multi-granularity mask generation yields a 'high-fidelity 3D mesh' whose geometry is accurate enough for an external KB to compute camera extrinsics that produce consistent novel views. No reconstruction-quality metrics (Chamfer distance, surface IoU, depth error, or view-consistency checks against ground-truth geometry on the evaluation scenes) are reported. Single-view reconstruction is severely under-constrained; without such metrics any reported spatial-reasoning gains cannot be confidently attributed to explicit 3D exploration rather than other factors.
[Method] Method (knowledge-base view synthesis): the description of how the external knowledge base 'iteratively compute[s] optimal camera extrinsic parameters' lacks concrete criteria for optimality, the precise interface between the reconstructed mesh and the KB, and any ablation showing that the synthesized views are geometrically consistent with the input image. This step is load-bearing for the Visual Chain-of-Thought claim.

minor comments (2)

[Abstract] The abstract references GPT-5.2; if this is a hypothetical or forthcoming model, a clarifying footnote or citation would help readers assess the comparison.
[Method] Notation for the multi-granularity masks and the exact form of the camera-extrinsic optimization objective should be formalized with equations to improve reproducibility.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We are grateful to the referee for the thorough review and valuable suggestions. We have carefully considered each major comment and provide our responses below. Where revisions are needed, we indicate the changes to be made in the revised manuscript.

read point-by-point responses

Referee: [Abstract] Abstract: the central claim that 'extensive experiments demonstrate that the proposed approach significantly enhances spatial comprehension' and 'outperforms ... on major benchmarks such as 3DSRBench and Rel3D' is unsupported by any quantitative results, tables, baselines, error bars, or controls. Without these data it is impossible to verify the magnitude of improvement or rule out confounds such as prompt engineering.

Authors: We appreciate this observation. The manuscript includes detailed quantitative results in the Experiments section, featuring tables with performance metrics on 3DSRBench and Rel3D, comparisons to baselines including specialized spatial models, GPT-5.2, and Gemini-2.5-Flash, along with error bars and controls for potential confounds like prompt engineering. To address the referee's concern and make the abstract self-contained, we will revise the abstract to include key quantitative highlights, such as the percentage improvements over baselines. revision: yes
Referee: [Proposed Pipeline] Proposed pipeline (3D reconstruction step): the framework rests on the assumption that MLLM-guided keyword extraction plus multi-granularity mask generation yields a 'high-fidelity 3D mesh' whose geometry is accurate enough for an external KB to compute camera extrinsics that produce consistent novel views. No reconstruction-quality metrics (Chamfer distance, surface IoU, depth error, or view-consistency checks against ground-truth geometry on the evaluation scenes) are reported. Single-view reconstruction is severely under-constrained; without such metrics any reported spatial-reasoning gains cannot be confidently attributed to explicit 3D exploration rather than other factors.

Authors: This is a fair and important point. Although the framework builds upon MLLM-enhanced single-view reconstruction, we did not report explicit quality metrics for the generated meshes. In the revised version, we will include reconstruction quality evaluations using Chamfer distance, surface IoU, and depth error on available ground-truth scenes from the benchmarks. We will also add view-consistency analyses to better attribute the spatial reasoning improvements to the 3D reconstruction and exploration steps rather than other elements of the pipeline. revision: yes
Referee: [Method] Method (knowledge-base view synthesis): the description of how the external knowledge base 'iteratively compute[s] optimal camera extrinsic parameters' lacks concrete criteria for optimality, the precise interface between the reconstructed mesh and the KB, and any ablation showing that the synthesized views are geometrically consistent with the input image. This step is load-bearing for the Visual Chain-of-Thought claim.

Authors: We agree that additional details and validation are required for this critical component. The optimality criteria involve selecting camera poses that maximize coverage of query-relevant spatial elements via mesh-based ray tracing. The interface passes the reconstructed mesh to a geometric optimization module within the KB. In the revision, we will expand the method section with the precise optimality objective, the interface specification, pseudocode for the iterative synthesis, and an ablation study demonstrating the geometric consistency of the novel views and their impact on reasoning performance. revision: yes

Circularity Check

0 steps flagged

No circularity: engineering pipeline with external components and empirical validation

full rationale

The paper presents a training-free pipeline that reconstructs a 3D mesh via MLLM-guided masks then uses an external KB for novel-view synthesis. No equations or derivations are provided that reduce a claimed result to its own inputs by construction. Performance claims rest on benchmark comparisons (3DSRBench, Rel3D) against external models rather than self-referential fits or self-citations. The method is self-contained as an applied framework; no load-bearing self-definition, fitted prediction renamed as result, or uniqueness theorem imported from the authors' prior work appears in the abstract or described chain.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on two key domain assumptions about the reliability of single-image 3D reconstruction and external KB view selection; these are not derived from first principles or supported by independent evidence in the abstract.

axioms (2)

domain assumption MLLM-guided keyword extraction and multi-granularity mask generation from a single image produces a high-fidelity 3D mesh
Invoked as the first step of the pipeline to enable subsequent view synthesis.
domain assumption An external knowledge base can iteratively compute optimal camera extrinsic parameters for synthesizing useful novel views
Invoked to emulate human perspective-taking and support multi-perspective reasoning.

pith-pipeline@v0.9.0 · 5508 in / 1508 out tokens · 64552 ms · 2026-05-10T19:11:50.584850+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/AlexanderDuality.lean alexander_duality_circle_linking (D=3 forced by non-trivial circle linking) echoes

?

echoes
ECHOES: this paper passage has the same mathematical shape or conceptual pattern as the Recognition theorem, but is not a direct formal dependency.

The proposed pipeline first reconstructs a high-fidelity 3D mesh from a single image using MLLM-guided keyword extraction and mask generation at multiple granularities. Subsequently, the framework leverages an external knowledge base to iteratively compute optimal camera extrinsic parameters and synthesize novel views.
IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We propose a training-free framework that introduces a Visual Chain-of-Thought mechanism grounded in explicit 3D reconstruction.

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

76 extracted references · 25 canonical work pages · 12 internal anchors

[1]

LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training

An, X., Xie, Y., Yang, K., Zhang, W., Zhao, X., Cheng, Z., Wang, Y., Xu, S., Chen, C., Zhu, D., et al.: Llava-onevision-1.5: Fully open framework for democratized multimodal training. arXiv preprint arXiv:2509.23661 (2025)

work page internal anchor Pith review arXiv 2025
[2]

Qwen3-VL Technical Report

Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)

work page internal anchor Pith review Pith/arXiv arXiv 2025
[3]

Qwen2.5-VL Technical Report

Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report. arXiv:2502.13923 (2025)

work page internal anchor Pith review Pith/arXiv arXiv 2025
[4]

Spatialdreamer: Incentivizing spatial reasoning via active mental imagery.arXiv preprint arXiv:2512.07733,

Cao, M., Li, X., Liu, X., Reid, I., Liang, X.: Spatialdreamer: Incentivizing spatial reasoning via active mental imagery. arXiv preprint arXiv:2512.07733 (2025)

work page arXiv 2025
[5]

SAM 3: Segment Anything with Concepts

Carion, N., Gustafson, L., Hu, Y.T., Debnath, S., Hu, R., Suris, D., Ryali, C., Alwala, K.V., Khedr, H., Huang, A., et al.: Sam 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719 (2025)

work page internal anchor Pith review Pith/arXiv arXiv 2025
[6]

In: CVPR (2024)

Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., Xia, F.: Spatialvlm: Endowing vision- language models with spatial reasoning capabilities. In: CVPR (2024)

2024
[7]

In: ICCV (2019)

Chen, R., Han, S., Xu, J., Su, H.: Point-based multi-view stereo network. In: ICCV (2019)

2019
[8]

SAM 3D: 3Dfy Anything in Images

Chen, X., Chu, F.J., Gleize, P., Liang, K.J., Sax, A., Tang, H., Wang, W., Guo, M., Hardin, T., Li, X., et al.: Sam 3d: 3dfy anything in images. arXiv preprint arXiv:2511.16624 (2025)

work page internal anchor Pith review arXiv 2025
[9]

Geometrically-constrained agent for spatial reasoning.arXiv preprint arXiv:2511.22659, 2025

Chen, Z., Lu, X., Zheng, Z., Li, P., He, L., Zhou, Y., Shao, J., Zhuang, B., Sheng, L.: Geometrically- constrained agent for spatial reasoning. arXiv preprint arXiv:2511.22659 (2025)

work page arXiv 2025
[10]

In: CVPR (2026)

Chen, Z., Zhang, X., Xu, H., Xie, J., Tu, Z.: Cvp: Central-peripheral vision-inspired multimodal model for spatial reasoning. In: CVPR (2026)

2026
[11]

Think with 3d: Geometric imagina- tion grounded spatial reasoning from limited views.arXiv preprint arXiv:2510.18632, 2025

Chen, Z., Zhang, M., Yu, X., Luo, X., Sun, M., Pan, Z., Feng, Y., Pei, P., Cai, X., Huang, R.: Think with 3d: Geometric imagination grounded spatial reasoning from limited views. arXiv preprint arXiv:2510.18632 (2025)

work page arXiv 2025
[12]

Dong, Z., Yi, J., Zheng, Z., Han, H., Zheng, X., Wang, A.J., Liu, F., Li, L.: Seeing is not reasoning: Mvpbenchforgraph-basedevaluationofmulti-pathvisualphysicalcot.arXivpreprintarXiv:2505.24182 (2025)

work page arXiv 2025
[13]

In: CVPR (2026)

Fan, Z., Zhang, J., Li, R., Zhang, J., Chen, R., Hu, H., Wang, K., Qu, H., Wang, D., Yan, Z., Xu, H., Theiss, J., Chen, T., Li, J., Tu, Z., Wang, Z., Ranjan, R.: Vlm-3r: Vision-language models augmented with instruction-aligned 3d reconstruction. In: CVPR (2026)

2026
[14]

Google: Gemini-3.https://blog.google/products-and-platforms/products/gemini/gemini-2-5- flash-preview/(2025)

2025
[15]

In: NeurIPS (2020)

Goyal, A., Yang, K., Yang, D., Deng, J.: Rel3d: A minimally contrastive benchmark for grounding spatial relations in 3d. In: NeurIPS (2020)

2020
[16]

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al.: Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

work page internal anchor Pith review Pith/arXiv arXiv 2025
[17]

arXiv preprint arXiv:2511.11239 , year=

Guo, Z., Liu, J., Li, Y., Gao, W., Yang, Z., Li, C., Zhang, X., Jian, P.: Beyond flatlands: Unlocking spa- tial intelligence by decoupling 3d reasoning from numerical regression. arXiv preprint arXiv:2511.11239 (2025)

work page arXiv 2025
[18]

In: CVPR (2026)

Guo, Z., Hong, M., Zhang, F., Jia, K., Jin, T.: Thinking with programming vision: Towards a unified view for thinking with images. In: CVPR (2026)

2026
[19]

arXiv preprint arXiv:2512.16237 (2025)

Helu, Z., Jingjing, H., Wang, X., Yangbin, X., Wanyue, Z., Baoyang, J., Shirui, D., Liang, Z., Fangfang, L., Tiejun, Z., et al.: Scaling spatial reasoning in mllms through programmatic data synthesis. arXiv preprint arXiv:2512.16237 (2025)

work page arXiv 2025
[20]

In: CVPR (2021)

Hong, Y., Wu, Q., Qi, Y., Rodriguez-Opazo, C., Gould, S.: Vln bert: A recurrent vision-and-language bert for navigation. In: CVPR (2021)

2021
[21]

In: NeurIPS (2025) 16 J

Hu, J., Zhang, Y., Han, Q., Jiang, D., Zhang, X., Shum, H.Y.: Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. In: NeurIPS (2025) 16 J. Chen et al

2025
[22]

In: CVPR (2026)

Hu, W., Lin, J., Long, Y., Ran, Y., Jiang, L., Wang, Y., Zhu, C., Xu, R., Wang, T., Pang, J.: G2vlm: Geometry grounded vision language model with unified 3d reconstruction and spatial reasoning. In: CVPR (2026)

2026
[23]

In: NeurIPS (2024)

Hu, Y., Shi, W., Fu, X., Roth, D., Ostendorf, M., Zettlemoyer, L., Smith, N.A., Krishna, R.: Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. In: NeurIPS (2024)

2024
[24]

In: NeurIPS (2025)

Huang, C.P., Wu, Y.H., Chen, M.H., Wang, Y.C.F., Yang, F.E.: Thinkact: Vision-language-action reasoning via reinforced visual latent planning. In: NeurIPS (2025)

2025
[25]

In: ICLR (2026)

Jia, M., Qi, Z., Zhang, S., Zhang, W., Yu, X., He, J., Wang, H., Yi, L.: Omnispatial: Towards compre- hensive spatial reasoning benchmark for vision language models. In: ICLR (2026)

2026
[26]

In: EMNLP (2023)

Jiang, Z., Xu, F.F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., Neubig, G.: Active retrieval augmented generation. In: EMNLP (2023)

2023
[27]

In: ICCV (2025)

Lee, P.Y., Je, J., Park, C., Uy, M.A., Guibas, L., Sung, M.: Perspective-aware reasoning in vision- language models via mental imagery simulation. In: ICCV (2025)

2025
[28]

In: NeurIPS (2020)

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. In: NeurIPS (2020)

2020
[29]

In: EMNLP (2024)

Li, C., Zhang, C., Zhou, H., Collier, N., Korhonen, A., Vulić, I.: Topviewrs: Vision-language models as top-view spatial reasoners. In: EMNLP (2024)

2024
[30]

In: IJCAI (2024)

Li,F.,Hogg, D.C.,Cohn,A.G.: Reframingspatial reasoningevaluationin languagemodels: Areal-world simulation benchmark for qualitative reasoning. In: IJCAI (2024)

2024
[31]

LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models

Li, F., Zhang, R., Zhang, H., Zhang, Y., Li, B., Li, W., Ma, Z., Li, C.: Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895 (2024)

work page internal anchor Pith review arXiv 2024
[32]

Improved visual-spatial reasoning via r1-zero-like training.arXiv preprint arXiv:2504.00883, 2025

Liao, Z., Xie, Q., Zhang, Y., Kong, Z., Lu, H., Yang, Z., Deng, Z.: Improved visual-spatial reasoning via r1-zero-like training. arXiv preprint arXiv:2504.00883 (2025)

work page arXiv 2025
[33]

In: CVPR (2025)

Lin, K.Q., Li, L., Gao, D., Yang, Z., Wu, S., Bai, Z., Lei, S.W., Wang, L., Shou, M.Z.: Showui: One vision-language-action model for gui visual agent. In: CVPR (2025)

2025
[34]

In: CVPR (2024)

Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: CVPR (2024)

2024
[35]

Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning.arXiv preprint arXiv:2505.18719, 2025

Lu, G., Guo, W., Zhang, C., Zhou, Y., Jiang, H., Gao, Z., Tang, Y., Wang, Z.: Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning. arXiv preprint arXiv:2505.18719 (2025)

work page arXiv 2025
[36]

In: NeurIPS (2024)

Ma, C., Lu, K., Cheng, T.Y., Trigoni, N., Markham, A.: Spatialpin: Enhancing spatial reasoning capa- bilities of vision-language models through prompting and interacting 3d priors. In: NeurIPS (2024)

2024
[37]

In: ICCV (2025)

Ma, W., Chen, H., Zhang, G., Chou, Y.C., Chen, J., de Melo, C., Yuille, A.: 3dsrbench: A comprehensive 3d spatial reasoning benchmark. In: ICCV (2025)

2025
[38]

In: CVPR (2025)

Ma, W., Ye, L., de Melo, C.M., Yuille, A., Chen, J.: Spatialllm: A compound 3d-informed design towards spatially-intelligent large multimodal models. In: CVPR (2025)

2025
[39]

In: ACL (2021)

Mirzaee, R., Faghihi, H.R., Ning, Q., Kordjamshidi, P.: Spartqa: A textual question answering bench- mark for spatial reasoning. In: ACL (2021)

2021
[40]

Nature Machine Intelligence (2025)

Mon-Williams, R., Li, G., Long, R., Du, W., Lucas, C.G.: Embodied large language models enable robots to complete complex tasks in unpredictable environments. Nature Machine Intelligence (2025)

2025
[41]

OpenAI: GPT-5.2.https://openai.com/index/introducing-gpt-5-2/(2025)

2025
[42]

Vln-r1: Vision-language navigation via reinforcement fine-tuning.arXiv preprint arXiv:2506.17221,

Qi, Z., Zhang, Z., Yu, Y., Wang, J., Zhao, H.: Vln-r1: Vision-language navigation via reinforcement fine-tuning. arXiv preprint arXiv:2506.17221 (2025)

work page arXiv 2025
[43]

RemyxAI: SpaceMantis.https://huggingface.co/remyxai/SpaceMantis(2025)

2025
[44]

co / remyxai / SpaceQwen2.5-VL-3B-Instruct(2025)

RemyxAI, QwenLM Team: SpaceQwen2.5-VL-3B-Instruct.https : / / huggingface . co / remyxai / SpaceQwen2.5-VL-3B-Instruct(2025)

2025
[45]

Proximal Policy Optimization Algorithms

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algo- rithms. arXiv preprint arXiv:1707.06347 (2017)

work page internal anchor Pith review Pith/arXiv arXiv 2017
[46]

In: NeurIPS (2024) Active 3D Exploration for MLLM Spatial Understanding 17

Shao, H., Qian, S., Xiao, H., Song, G., Zong, Z., Wang, L., Liu, Y., Li, H.: Visual cot: Advancing multi- modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. In: NeurIPS (2024) Active 3D Exploration for MLLM Spatial Understanding 17

2024
[47]

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024)

work page internal anchor Pith review Pith/arXiv arXiv 2024
[48]

In: CVPR (2025)

Szot, A., Mazoure, B., Attia, O., Timofeev, A., Agrawal, H., Hjelm, D., Gan, Z., Kira, Z., Toshev, A.: From multimodal llms to generalist embodied agents: Methods and lessons. In: CVPR (2025)

2025
[49]

In: NeurIPS (2024)

Tong, P., Brown, E., Wu, P., Woo, S., Iyer, A.J.V., Akula, S.C., Yang, S., Yang, J., Middepogu, M., Wang, Z., et al.: Cambrian-1: A fully open, vision-centric exploration of multimodal llms. In: NeurIPS (2024)

2024
[50]

InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency

Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., et al.: Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265 (2025)

work page internal anchor Pith review Pith/arXiv arXiv 2025
[51]

In: CVPR (2025)

Wang, X., Ma, W., Zhang, T., de Melo, C.M., Chen, J., Yuille, A.: Spatial457: A diagnostic benchmark for 6d spatial reasoning of large mutimodal models. In: CVPR (2025)

2025
[52]

N3d-vlm: Native 3d grounding enables accurate spatial reasoning in vision-language models.arXiv preprint arXiv:2512.16561, 2025

Wang, Y., Ke, L., Zhang, B., Qu, T., Yu, H., Huang, Z., Yu, M., Xu, D., Yu, D.: N3d-vlm: Native 3d grounding enables accurate spatial reasoning in vision-language models. arXiv preprint arXiv:2512.16561 (2025)

work page arXiv 2025
[53]

In: NeurIPS (2022)

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of- thought prompting elicits reasoning in large language models. In: NeurIPS (2022)

2022
[54]

In: NeurIPS (2025)

Wu, D., Liu, F., Hung, Y.H., Duan, Y.: Spatial-mllm: Boosting mllm capabilities in visual-based spatial intelligence. In: NeurIPS (2025)

2025
[55]

SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence

Wu, H., Huang, X., Chen, Y., Zhang, Y., Wang, Y., Xie, W.: Spatialscore: Towards unified evaluation for multimodal spatial understanding. arXiv preprint arXiv:2505.17012 (2025)

work page internal anchor Pith review Pith/arXiv arXiv 2025
[56]

Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

Yang, J., Zhang, H., Li, F., Zou, X., Li, C., Gao, J.: Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441 (2023)

work page internal anchor Pith review arXiv 2023
[57]

In: CVPR (2022)

Yang, Z., Ren, Z., Shan, Q., Huang, Q.: Mvs2d: Efficient multi-view stereo via attention-driven 2d convolutions. In: CVPR (2022)

2022
[58]

In: ICLR (2022)

Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K.R., Cao, Y.: React: Synergizing reasoning and acting in language models. In: ICLR (2022)

2022
[59]

arXiv preprint arXiv:2506.21458 (2025)

Yin, B., Wang, Q., Zhang, P., Zhang, J., Wang, K., Wang, Z., Zhang, J., Chandrasegaran, K., Liu, H., Krishna, R., Xie, S., Li, M., Wu, J., Fei-Fei, L.: Spatial mental modeling from limited views. arXiv preprint arXiv:2506.21458 (2025)

work page arXiv 2025
[60]

In: CVPR (2020)

Yu, Z., Gao, S.: Fast-mvsnet: Sparse-to-dense multi-view stereo with learned propagation and gauss- newton refinement. In: CVPR (2020)

2020
[61]

In: CVPR (2025)

Zhao, Q., Lu, Y., Kim, M.J., Fu, Z., Zhang, Z., Wu, Y., Li, Z., Ma, Q., Han, S., Finn, C., et al.: Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. In: CVPR (2025)

2025
[62]

NeurIPS (2023)

Zheng, G., Yang, B., Tang, J., Zhou, H.Y., Yang, S.: Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models. NeurIPS (2023)

2023
[63]

aha moment

Zhou,H.,Li,X.,Wang,R.,Cheng,M.,Zhou,T.,Hsieh,C.J.:R1-zero’s"ahamoment"invisualreasoning on a 2b non-sft model. arXiv preprint arXiv:2503.05132 (2025)

work page arXiv 2025
[64]

Image-of-thought prompt- ing for visual reasoning refinement in multimodal large language models.arXiv preprint arXiv:2405.13872, 2024

Zhou, Q., Zhou, R., Hu, Z., Lu, P., Gao, S., Zhang, Y.: Image-of-thought prompting for visual reasoning refinement in multimodal large language models. arXiv preprint arXiv:2405.13872 (2024) 18 J. Chen et al. A Detailed Subtask Composition and Question Types of the Benchmarks As discussed in Sec. 4.1, the evaluation benchmarks utilized in this study consi...

work page arXiv 2024
[65]

The"Think"tag should contain the reasoning for selecting the viewpoint position. If similar problems and the corresponding viewpoint selections can be found in the ex- perience library, please refer to them; otherwise, based on independent understanding and imagination, analyze the scene expected from different viewpoints and determine whether this scene ...
[66]

Perspective

The"Perspective"tag should contain a brief description of the selected viewpoint position. Step 2: Coordinate Selection.Building upon the target perspective described in the previous step, the MLLM is tasked with predicting the corresponding spatial coordinates and the viewing angle to compute the camera extrinsic. To visually ground this numerical predic...
[67]

The selected XY plane coordinates should not be located inside the object (e.g., [0, 0]), but rather outside the object at a distance of approximately one empty grid interval
[68]

Think": ...,

The value range of the pitch angle is [-90, 90]. Output json format requirements: { "Think": ..., "Coords": [x, y, pitch] } Where:
[69]

Select the coordinates in the plane map and the side view that match the description of the perspective position and the sequential requirements based on the observation results

The"Think"tag must include the sequential observation results of the plane coordi- nate map and the side view (i.e., state the coordinate position of each object). Select the coordinates in the plane map and the side view that match the description of the perspective position and the sequential requirements based on the observation results
[70]

Step 3: Final Reasoning and Answer.In this final step, the pipeline renders a novel view using the camera extrinsic predicted in Step 2

The"Coords"tag must contain the selected coordinate position. Step 3: Final Reasoning and Answer.In this final step, the pipeline renders a novel view using the camera extrinsic predicted in Step 2. The MLLM is then provided with this synthesized image, acting as both a view evaluator and a spatial reasoner. Initially, the model must verify the quality of...
[71]

All objects related to the question are present in the frame
[72]

The displayed perspective is consistent with the perspective description obtained in Step 1 (i.e., {})
[73]

Observation

There is no issue of incomplete display or overly small display regarding the objects. (Partial occlusion between objects is permitted.) Output json format requirements: { "Observation": ..., "Reasoning": ..., "Answer": ... } Where:
[74]

Observation

The"Observation"tag includes the observations of the original image, the plane coor- dinate map, the side view, and the perspective transformation image, providing a visual description related to the question for each image respectively
[75]

Reasoning

The"Reasoning"tag includes the solution process for the question based on the obser- vations, analyzes whether the perspective images meet all the aforementioned require- ments and conditions, and finally infers the answer to the question
[76]

Answer"tag contains the final answer to the question; output only

The"Answer"tag contains the final answer to the question; output only "None" if any of the aforementioned conditions are not met upon analysis. B.3 Evaluation Prompt Design This subsection describes the design of the prompt used for the evaluations presented in Tab. 1. To further evaluate the instruction-following capability of both thegeneral-purpose mod...