Pith. sign in

REVIEW 3 major objections 5 minor 109 references

SpatialCLI: Learning to Reason With Spatial Tools, Then Without Them

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A three-stage pipeline gives vision-language models specialist spatial perception on demand, then transfers that perception into the model's own weights so the tools are no longer required.

desk verdict A thorough, well-ablated pipeline for teaching VLMs to use and then internalize spatial tools; the internalization evidence is suggestive rather than airtight, but the external benchmark gains deserve a serious referee. read the letter →

arxiv 2607.27703 v2 pith:EZV5JO5V submitted 2026-07-30 cs.AI

Yang Zhou , Zixuan Huang , Sunzhu Li , Zhuo Yang , Chen Zhang , Shunian Chen , Caijun Yan , Jianyao Xu
show 5 more authors
Shunyu Liu Weijie Fu Peiliang Li Xiaozhi Chen Yuxiang Cai
This is my paper
classification cs.AI
keywords vision-languagemodelsspatialreasoningtool-augmentedagentsreinforcementlearningcapabilityinternalizationperceptionembodiedAIchain-of-thoughtdistillation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models reason well about tasks but see poorly at the level of exact boxes, boundaries, depths, and poses; specialist vision models measure those quantities precisely but cannot decide when to call them or what a task needs. This paper proposes SpatialCLI, a three-stage framework that lets a VLM borrow specialist perception at inference time as four callable spatial tools, trains it to use them through supervised cold-start data and reinforcement learning, and then converts its own successful tool-use trajectories into evidence-grounded reasoning text that trains it to answer without the tools. The central empirical claim is that both abilities improve together: an 8-billion-parameter VLM climbs from 29.3% to 84.6% on MindCube with tools and retains 73.8% without them, and the pattern holds across model scales and benchmarks, including a new 516-question benchmark, SpatialCLI-Bench, built to require composing localization, segmentation, depth, and pose in a single answer. If the claim holds, external tool use and internalized direct reasoning can coexist in one model, which is a step toward embodied foundation models that do not permanently depend on specialist vision backends.

What carries the argument

The load-bearing object is the interaction trajectory—the interleaved reasoning, tool calls, and returned results a VLM produces in a ReAct-style loop. Four tools expose specialist models: Locate (Locate Anything fused with Grounding DINO), Segment (SAM 3), Depth (Depth Anything 3), and Pose (Orient Anything V2, plus VGGT for camera motion). Learning uses cold-start SFT on filtered teacher trajectories, then agentic RL with the GRPO objective and outcome rewards. Internalization applies Progressive Evidence-Grounded Trajectory Verbalization—turn-wise evidence consolidation followed by global verbalization by a large teacher LLM—and Dual-View Capability Internalization, jointly optimizing an

What would settle it

Unpoisoned audit: on a random sample of the trajectories used for internalization, diff every number and relational claim in the verbalized reasoning chain against the raw tool result it was built from; a non-negligible mismatch rate would show the supervision is corrupted at its source. Poisoned test: run the full pipeline with one tool deliberately biased (e.g., all depths shifted +20%) on a held-out set, then check whether the tool-free model reproduces that same bias—if it does, the model has memorized tool outputs rather than learned geometry.

Watch

Extended reading notes

Core claim

The central claim is that perceptual evidence a VLM collects from external spatial tools—bounding boxes, masks, depths, pose directions—can be converted into training text that transfers those capabilities into the model's own weights. SpatialCLI verbalizes each successful tool trajectory twice: turn by turn, consolidating each tool result into an evidence-reasoning unit, then globally, merging units into a tool-free reasoning chain that reaches the correct answer. Dual-view training supervises the model on both the tool-free chain and the original tool-call trace, so internalization does not overwrite tool use. Across benchmarks and model scales, trained models beat their base VLMs with too

Load-bearing premise

The entire transfer step trusts that the teacher model's written summaries of tool outputs are exact—no dropped, rounded, or invented coordinate, count, depth, or relation—and the paper does not measure how often the verbalizer preserves every value.

Editorial extensions

If this is right

  • A single checkpoint can serve both tool-augmented and tool-free inference: SpatialCLI-8B scores 91.3% with tools and 72.7% without on SpatialCLI-Bench, versus 35.3% for the untrained base model without tools.
  • Structured textual returns—coordinates, polygons, metric depths—carry the gain; rendering the same evidence as annotated images adds no consistent benefit, and structured text is what makes verbalization into language training possible.
  • Tool-use training and direct-reasoning training are complementary: cold-start SFT plus agentic RL keeps tool calls near 2.6 per task and stabilizes learning, whereas RL without tools degrades tool competence and internalizes far less (52.7% vs 72.7% on SpatialCLI-Bench).
  • Internalization scales with both data and model size: task accuracy and the Capability Internalization Index rise together as more trajectories are verbalized, and larger models absorb more capability even after tool-use performance saturates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same verbalize-and-train loop likely transfers to other tool classes—search, code execution, simulation—wherever a tool's structured output can be written as text; the paper itself notes its scope is limited to structured perceptual outputs.
  • The framework implies a renewable-curriculum view: swap in a stronger specialist tool, regenerate successful trajectories, re-run internalization, and the model's native capability should upgrade—a test the paper does not run.
  • Because the verbalizer is itself a language model, internalized knowledge is a second-hand copy of tool outputs; any systematic verbalizer drift (dropped, rounded, or invented values) would be baked into the trained model, so auditing verbalizer fidelity is the natural first check on the method.
  • The benchmark's design—forcing composition of multiple spatial capabilities per question—suggests isolated-capability benchmarks understate how far frontier models lag tool-augmented reasoning; on SpatialCLI-Bench a frontier model scores 48.8% without tools and 72.9% with them, a gap worth probing on other compositional tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SpatialCLI, a three-stage framework (Call–Learn–Internalize) for improving vision-language models' spatial reasoning. In the Call stage, specialist vision models are exposed as spatial tools for localization, segmentation, depth, and pose. In the Learn stage, cold-start SFT followed by GRPO-based agentic RL trains the VLM to use these tools. In the Internalize stage, successful tool-use trajectories are verbalized into evidence-grounded reasoning chains via turn-wise consolidation and a global verbalizer, and the model is trained jointly on a capability-internalization view and a tool-use view. The authors also introduce SpatialCLI-Bench, a 516-example six-choice benchmark for compositional spatial reasoning. Experiments on Qwen3-VL-8B, Qwen3.6-35B-A3B, and Qwen3.6-27B, plus several frontier models, report large w/ Tools and w/o Tools gains across MindCube, SpatialCLI-Bench, MMSI, DA-2K, and BOPASK.

Significance. If the internalization stage is faithful, the paper makes a valuable contribution: it demonstrates that tool-grounded perceptual reasoning can be converted into direct, tool-free VLM capability without sacrificing tool use, and it provides a new benchmark for compositional spatial perception. The work is unusually thorough in its ablations, including tool-return format comparisons, verbalization-variant comparisons, λ sensitivity, and training-dynamics analysis. The authors also provide code, model, and dataset links, and the external benchmarks (MindCube, MMSI, DA-2K, BOPASK) are not fitted by design. The principal weakness is that the key verbalization step is unverified and answer-conditioned, leaving open the possibility that the w/o Tools gains partly reflect rationalization rather than internalized tool evidence. With an explicit fidelity audit and variance reporting, the contribution would be solid.

major comments (3)
  1. [§3.3, Box G.3] The global verbalizer Φ_verb receives the correct answer y* as input. The turn-wise consolidator deliberately withholds y* to prevent answer-conditioned evidence reconstruction, but the global pass reintroduces it and is instructed to construct a reasoning chain 'explaining why the evidence supports that answer.' The prompt rules prohibit introducing unsupported entities/relations, but no verification or quantitative audit is reported. If the teacher silently fills gaps in Eτ to fit y*, then the trained w/o Tools model may be learning answer-shaped reasoning rather than internalized spatial capabilities. This is load-bearing because both the w/o Tools performance and the CII metric are trained on these chains. Please provide a chain-fidelity audit (e.g., automated or human verification that every factual statement in c is entailed by the corresponding tool result or explicit observation)
  2. [Appendix E, §4.3] The Capability Internalization Index is defined as similarity between the model's tool-free outputs and the specialist-tool outputs. But those same tool outputs were used as the supervision targets for internalization training, so high CII is expected by construction and does not independently establish 'genuine transfer of specialist perceptual capabilities rather than final-answer memorization.' The external w/o Tools gains on MindCube, BOPASK, MMSI, and DA-2K are more convincing evidence, but the CII-based claim in §4.3 should be reframed as a measurement of reproduction fidelity, with the external benchmarks carrying the burden of showing transfer.
  3. [§4.1, Appendix B.2] All task-performance numbers are reported as means over three evaluation runs, without error bars, confidence intervals, or per-run values. Several improvements in Table 1 are small (e.g., MMSI Pos-Cam-Cam 49.5 vs 53.8, or BOPASK ObjRrr 48.5 vs 52.3) and could be within run-to-run noise. The claim of 'consistent' improvement across all subsets would be much stronger with a measure of variance or a statistical test. Please report standard deviations or the three individual-run results for at least the main tables.
minor comments (5)
  1. [Abstract] The word 'Learn' is typeset as 'L earn' in the abstract. Please fix the formatting glitch.
  2. [Figure 4] Panel (a) is labeled 'Performance vs. Steps' but the y-axis label is only 'Score (%)'. Please specify which benchmark the curve refers to, or add per-panel captions.
  3. [Table 1] The column header 'Pos-Cam-Cam Traj. ObjRrr' is unclear. Please use consistent subset names (e.g., 'Pos-Cam-Cam', 'Trajectory', 'Object-Rearrangement') and match them to the text in §4.1.
  4. [Appendix C.5] Human review is described as 'two graduate students' who must agree with the generated answer. Reporting inter-annotator agreement or at least the per-annotator accuracy rates would strengthen the claim of independent verification.
  5. [Appendix B.3] The hardware is described as 'PPU-ZW810E accelerators.' Please provide the vendor/model name or a reference, since this is not a standard publicly identifiable device.

Circularity Check

1 steps flagged · score 2.0 of 10

Central w/o Tools gains are independently grounded on external benchmarks; the CII diagnostic is self-referential but not load-bearing, and the verbalizer-fidelity concern is a validity risk rather than circularity.

  1. self definitional [Section 4.3 and Appendix E.4; supervision source in Section 3.3]
    "To quantify internalization, we use the Capability Internalization Index (CII), which measures how closely a model can reproduce the corresponding spatial-tool outputs without invoking the tools; higher values indicate stronger internalization... SpatialCLI therefore uses successful SpatialCLI-RL trajectories as the source of supervision for internalizing the specialist perceptual capabilities supplied by spatial tools."

    CII's reference standard is the spatial-tool output, and the internalization supervision is a verbalization of trajectories produced by those same spatial tools. Thus CII measures agreement with the source distribution that generated the training target; an increase after training is expected from fitting that distribution. The paper uses the synchronized rise of CII and task scores to claim 'genuine transfer ... rather than final-answer memorization,' but CII cannot independently distinguish internalization from distributional mimicry because the metric and the training signal share the same oracle. The held-out CII suite mitigates direct memorization, and the main external-benchmark gains remain independent, so this is a self-referential diagnostic rather than a derivation of the central

full rationale

The central claim—SpatialCLI improves both w/ Tools and w/o Tools reasoning—is supported by Table 1 on external benchmarks (MindCube, MMSI, DA-2K, BOPASK) that are not constructed by the paper and are claimed to be non-overlapping with training data. Those w/o Tools gains (e.g., MindCube 29.3→73.8 for Qwen3-VL-8B-Instruct) are not fitted to the evaluation sets and give the main result independent content. The self-referential components are auxiliary: CII is defined as reproducing the same specialist-tool outputs that generated the internalization supervision, and SpatialCLI-Bench questions are conditioned on specialist evidence, but neither is the sole evidence for transfer. A separate validity risk is that the global verbalizer receives the correct answer y* before composing the reasoning chain (Section 3.3, Box G.3); the prompt rules forbid unsupported additions but no fidelity audit is reported. This could make the w/o Tools chains answer-conditioned rationalizations, but it is a correctness/robustness threat, not a circularity, because the model does not see y* at inference and the external benchmark evaluations are independent. Overall, no load-bearing step reduces by construction to its inputs; the paper's derivation is self-contained enough for a low circularity score.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The framework does not introduce new physical entities, forces, or dimensions. The main unstated costs are the hand-set tool thresholds and the trust placed in the teacher verbalizer and specialist models.

free parameters (5)
  • Tool-Use View loss weight λ = 0.5
    Hand-chosen after a sensitivity sweep over {0.2, 0.5, 1.0, 1.5} balancing capability internalization (CII, w/o Tools score) against preservation of the tool-use policy (w/ Tools score). Reported in Appendix F.4.
  • Locate fusion thresholds = confidence 0.30, NMS IoU 0.50, cross-confirm IoU 0.50, dedup IoU 0.75, near-contained ratio 0.90
    Hand-set thresholds controlling how Grounding DINO and Locate Anything detections are fused and deduplicated. They determine the boxes the VLM sees and therefore the evidence for localization and downstream depth queries. Appendix D.2.
  • Segment simplification parameters = confidence 0.30, mask suppression IoU 0.90, polygon target IoU 0.97, vertex budget 16, min component area 16 px²
    Hand-set parameters for converting SAM 3 masks into polygon_2d outputs. The target IoU of 0.97 is described as a target, not unconditional; if not met the bounded candidate is returned. Appendix D.2.
  • Depth backend settings = processing resolution 504 px, deterministic metric-scale alignment
    Choice of Depth Anything 3 backend and resolution affects the depth_m values that ground all depth comparisons and benchmark construction. Appendix D.2.
  • Pose discretization thresholds = 45° sectors, translation norm 0.002, rotation suppression 2°
    Hand-set thresholds defining how VGGT/Orient Anything V2 outputs are serialized into direction names and axes. These determine the pose evidence used in training and benchmark construction. Appendix D.2.
assumptions (4)
  • domain assumption Specialist vision models (SAM 3, Grounding DINO, Locate Anything, Depth Anything 3, VGGT, Orient Anything V2) produce sufficiently reliable spatial outputs to serve as ground-truth evidence.
    The tools' outputs are used both as the evidence for benchmark questions and as supervision/reward for the VLM. If they are systematically wrong, the whole pipeline inherits their errors. Invoked in Sections 3.1, 3.4, and throughout Appendix D.
  • domain assumption The teacher LLM (Qwen3.5-397B-A17B) generates correct tool-use trajectories and faithfully verbalizes them without hallucinating or omitting evidence.
    Cold-Start SFT data and the internalization targets come from this model. The prompts in Boxes G.2 and G.3 instruct faithfulness, but there is no verification that the verbalized chains contain no unsupported entities or values. Sections 3.2, 3.3.
  • domain assumption GRPO/DAPO reinforcement learning with sparse outcome rewards shapes the tool-use policy as expected.
    The method relies on RL to improve tool selection and termination from a binary/multiple-choice reward. This is a standard assumption for such training, not proven in the paper. Section 3.2 and Appendix B.1.
  • domain assumption Gemini 3.1 Pro and human expert review produce valid, unbiased ground truth for SpatialCLI-Bench.
    Benchmark questions, answers, and distractors are generated by Gemini 3.1 Pro conditioned on specialist outputs and filtered by human agreement (71.67% retention). The benchmark's validity depends on both. Section 3.4 and Appendix C.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SpatialCLI: Learning to Reason With Spatial Tools, Then Without Them." pith.science (2026). https://pith.science/paper/EZV5JO5V

@misc{pith2026260727703,
  author       = {Pith},
  title        = {Pith review of: SpatialCLI: Learning to Reason With Spatial Tools, Then Without Them},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EZV5JO5V}},
  note         = {Machine review of arXiv:2607.27703}
}
read the original abstract

Vision-language models (VLMs) are increasingly used in embodied agents to interpret visual inputs, reason about spatial relationships, and make task-level decisions based on that reasoning. However, a fundamental capability mismatch remains: general VLMs can reason about the overall task but often miss the visual details that determine success, while specialist vision models can capture those details but cannot translate them into task-level decisions. In this work, we propose SpatialCLI, a framework that teaches VLMs to reason with spatial tools and progressively internalize the specialist perceptual capabilities they provide. SpatialCLI proceeds in three stages: (1) Call exposes specialist vision models as spatial tools to augment the VLM's perception; (2) Learn uses Cold-Start SFT and agentic RL to improve tool use; and (3) Internalize verbalizes successful tool-use trajectories to internalize specialist perceptual capabilities. We further introduce SpatialCLI-Bench, a 516-example benchmark for compositional perception across localization, segmentation, depth, and pose. On MindCube, SpatialCLI raises Qwen3-VL-8B-Instruct from 29.3% to 84.6% with tools, surpassing GPT-5.6 Sol with tools (72.1%), while retaining 73.8% without tools after internalization.

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

109 extracted references · 52 canonical work pages

  1. [1]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    arXiv 2025

  2. [2]

    Bop-ask: Object-interaction reasoning for vision-language models

    Vineet Bhat, Sungsu Kim, Valts Blukis, Greg Heinrich, Prashanth Krishnamurthy, Ramesh Karri, Stan Birchfield, Farshad Khorrami, and Jonathan Tremblay. Bop-ask: Object-interaction reasoning for vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16746–16757, 2026

    2026

  3. [3]

    VLM3: Vision Language Models Are Native 3D Learners

    Zhipeng Cai, Zhuang Liu, Yunyang Xiong, Zechun Liu, Vikas Chandra, and Yangyang Shi. Vlm3: Vision language models are native 3d learners.arXiv preprint arXiv:2605.30561, 2026

    work page Pith review arXiv 2026

  4. [4]

    Sam 3: Segment anything with concepts

    Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. Sam 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719, 2025

    arXiv 2025

  5. [5]

    Spatialvlm: Endowingvision-languagemodelswithspatialreasoningcapabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowingvision-languagemodelswithspatialreasoningcapabilities. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455–14465, 2024

    2024

  6. [6]

    VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation

    Siyi Chen, Hugo Hadfield, Alex Zook, Mikaela Angelina Uy, Chan Hee Song, Erwin Coumans, Xuning Yang, Faisal Ladhak, Qing Qu, Stan Birchfield, et al. Volo: A physical orchestrator for open-vocabulary long-horizon manipulation.arXiv preprint arXiv:2606.07723, 2026

    work page Pith review arXiv 2026

  7. [7]

    Spacetools: Tool-augmented spatial reasoning via double interactive rl

    Siyi Chen, Mikaela Angelina Uy, Chan Hee Song, Faisal Ladhak, Adithyavairavan Murali, Qing Qu, Stan Birch- field, Valts Blukis, and Jonathan Tremblay. Spacetools: Tool-augmented spatial reasoning via double interactive rl. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 37109–37120, 2026

    2026

  8. [8]

    Geometrically-constrained agent for spatial reasoning

    Zeren Chen, Xiaoya Lu, Zhijie Zheng, Pengrui Li, Lehan He, Yijin Zhou, Jing Shao, Bohan Zhuang, and Lu Sheng. Geometrically-constrained agent for spatial reasoning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 38689–38699, 2026

    2026

Show all 109 references
  1. [9]

    Spatialrgpt: Grounded spatial reasoning in vision-language models

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. InAdvances in Neural Information Processing Systems, volume 37, pages 135062–135093, 2024

    2024

  2. [10]

    S-agent: Spatial tool-use elicits reasoning for spatial intelligence.arXiv preprint arXiv:2606.20515, 2026

    Yalun Dai, Hao Li, Shulin Tian, Runmao Yao, Yuhao Dong, Fangzhou Hong, Zhaoxi Chen, Fangfu Liu, Baoliang Tian, Dingwen Zhang, et al. S-agent: Spatial tool-use elicits reasoning for spatial intelligence.arXiv preprint arXiv:2606.20515, 2026

    2026 arXiv

  3. [11]

    Gemini 3.1 Pro model card, February 2026.https://deepmind.google/models/model-cards/ gemini-3-1-pro/

    Google DeepMind. Gemini 3.1 Pro model card, February 2026.https://deepmind.google/models/model-cards/ gemini-3-1-pro/

    2026

  4. [12]

    Handal: A dataset of real-world manipulable object categories with pose annotations, affordances, and reconstructions

    Andrew Guo, Bowen Wen, Jianhe Yuan, Jonathan Tremblay, Stephen Tyree, Jeffrey Smith, and Stan Birch- field. Handal: A dataset of real-world manipulable object categories with pose annotations, affordances, and reconstructions. In2023 IEEE/RSJ International Conference on Intell...

    2023

  5. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  6. [14]

    Visual programming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14953–14962, 2023

    2023

  7. [15]

    Tiger: Tool-integrated geometric reasoning in vision-language models for robotics.arXiv preprint arXiv:2510.07181, 2025

    Yi Han, Enshen Zhou, Shanyu Rong, Jingkun An, Pengwei Wang, Zhongyuan Wang, Cheng Chi, Lu Sheng, and Shanghang Zhang. Tiger: Tool-integrated geometric reasoning in vision-language models for robotics.arXiv preprint arXiv:2510.07181, 2025

    2025

  8. [16]

    Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes

    Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab. Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. InAsian conference on computer vision, pages 548–562...

    2012

  9. [17]

    What matters in orchestrating robot policies: A systematic study of hierarchical vla agents.arXiv preprint arXiv:2606.10267, 2026

    Jiaheng Hu, Mohit Shridhar, Caden Lu, Dhruv Shah, Hao-Tien Lewis Chiang, Jie Tan, and Annie Xie. What matters in orchestrating robot policies: A systematic study of hierarchical vla agents.arXiv preprint arXiv:2606.10267, 2026

    2026 arXiv

  10. [18]

    Visual program distillation: Distilling tools and programmatic reasoning into vision-language models

    Yushi Hu, Otilia Stretcu, Chun-Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision-language models. InProceedings of the IEEE/CVF Conference on Computer...

    2024

  11. [19]

    Does your reasoning model implicitly know when to stop thinking?arXiv preprint arXiv:2602.08354, 2026

    Zixuan Huang, Xin Xia, Yuxi Ren, Jianbin Zheng, Xuanda Wang, Zhixia Zhang, Hongyan Xie, Songshi Liang, Zehao Chen, Xuefeng Xiao, et al. Does your reasoning model implicitly know when to stop thinking?arXiv preprint arXiv:2602.08354, 2026

    2026 arXiv

  12. [20]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    2025 arXiv

  13. [21]

    Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    2024 arXiv

  14. [22]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.arXiv preprint arXiv:1811.00982, 2018

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Ste- fan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection...

    2018 arXiv

  15. [23]

    Cubify anything: Scaling indoor 3d object detection.arXiv preprint arXiv:2412.04458, 2024

    Justin Lazarow, David Griffiths, Gefen Kohavi, Francisco Crespo, and Afshin Dehghan. Cubify anything: Scaling indoor 3d object detection.arXiv preprint arXiv:2412.04458, 2024

    2024 arXiv

  16. [24]

    Towards long-horizon embodied agents with tool-aligned vision-language-action models.arXiv preprint arXiv:2605.13119, 2026

    Zixing Lei, Changxing Liu, Yichen Xiong, Minhao Xiong, Yuanzhuo Ding, Zhipeng Zhang, Weixin Li, and Siheng Chen. Towards long-horizon embodied agents with tool-aligned vision-language-action models.arXiv preprint arXiv:2605.13119, 2026

    2026 arXiv

  17. [25]

    Depth anything 3: Recovering the visual space from any views.arXiv preprint arXiv:2511.10647, 2025

    Haotong Lin, Sili Chen, Junhao Liew, Donny Y Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang. Depth anything 3: Recovering the visual space from any views.arXiv preprint arXiv:2511.10647, 2025

    2025 arXiv

  18. [26]

    Deepseek-v3.2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556, 2025

    Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseek-v3.2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556, 2025

    2025 arXiv

  19. [27]

    Guava: An effective and universal harness for embodied manipulation.arXiv preprint arXiv:2606.18363, 2026

    Haowen Liu, Xirui Li, Shaoxiong Yao, Peng Shi, Tianyi Zhou, Jia-Bin Huang, Furong Huang, and Jiayuan Mao. Guava: An effective and universal harness for embodied manipulation.arXiv preprint arXiv:2606.18363, 2026

    2026 arXiv

  20. [28]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuropean conference on computer vision, pages 38–55. Springer, 2024

    2024

  21. [29]

    Veriweb: Verifiable long-chain web benchmark for agentic information-seeking.arXiv preprint arXiv:2508.04026, 2025

    Shunyu Liu, Minghao Liu, Huichi Zhou, Zhenyu Cui, Yang Zhou, Yuhao Zhou, Jialiang Gao, Heng Zhou, Yunhao Yang, Wendong Fan, et al. Veriweb: Verifiable long-chain web benchmark for agentic information-seeking.arXiv preprint arXiv:2508.04026, 2025

    2025

  22. [30]

    Budget-aware tool-use enables effective agent scaling.arXiv preprint arXiv:2511.17006, 2025

    Tengxiao Liu, Zifeng Wang, Jin Miao, I Hsu, Jun Yan, Jiefeng Chen, Rujun Han, Fangyuan Xu, Yanfei Chen, Ke Jiang, et al. Budget-aware tool-use enables effective agent scaling.arXiv preprint arXiv:2511.17006, 2025

    2025

  23. [31]

    Spatialpin: Enhancing spatial reasoning capabilities of vision-language models through prompting and interacting 3d priors

    Chenyang Ma, Kai Lu, Ta-Ying Cheng, Niki Trigoni, and Andrew Markham. Spatialpin: Enhancing spatial reasoning capabilities of vision-language models through prompting and interacting 3d priors. InAdvances in neural information processing systems, volume 37, pages 68803–68832, 2024

    2024

  24. [32]

    Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shan- tanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021

    2021 arXiv

  25. [33]

    Gpt-5.6: Frontier intelligence that scales with your ambition, July 2026.https://openai.com/index/ gpt-5-6/

    OpenAI. Gpt-5.6: Frontier intelligence that scales with your ambition, July 2026.https://openai.com/index/ gpt-5-6/

    2026

  26. [34]

    Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025. 12

    2025 arXiv

  27. [35]

    Qwen3.5: Towards native multimodal agents, February 2026.https://qwen.ai/blog?id=qwen3.5

    Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026.https://qwen.ai/blog?id=qwen3.5

    2026

  28. [36]

    Qwen3.7-Plus: Multimodal agent intelligence, June 2026.https://qwen.ai/blog?id=qwen3.7-plus

    Qwen Team. Qwen3.7-Plus: Multimodal agent intelligence, June 2026.https://qwen.ai/blog?id=qwen3.7-plus

    2026

  29. [37]

    Al- lospatial: Agenticharnessframeworkforspatialreasoninginfoundationmodels.arXiv preprint arXiv:2606.08952, 2026

    ShouweiRuan, BinWang, ZhenyuWu, QihuiZhu, YuxiangZhang, JingzhiLi, YubinWang, andXingxingWei. Al- lospatial: Agenticharnessframeworkforspatialreasoninginfoundationmodels.arXiv preprint arXiv:2606.08952, 2026

    2026 arXiv

  30. [38]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Advances in neural information processing systems, volume 36, page...

    2023

  31. [39]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  32. [40]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. InAdvances in Neural Information Processing Systems, volume 36, pages 38154–38180, 2023

    2023

  33. [41]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv:2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv:2409.19256, 2024

    2024 arXiv

  34. [42]

    Vipergpt: Visual inference via python execution for reasoning

    Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. InProceedings of the IEEE/CVF international conference on computer vision, pages 11888–11898, 2023

    2023

  35. [43]

    Hy-embodied-0.5: Embodied foundation models for real-world agents.arXiv preprint arXiv:2604.07430, 2026

    HY Team, Xumin Yu, Zuyan Liu, Ziyi Wang, He Zhang, Yongming Rao, Fangfu Liu, Yani Zhang, Ruowen Zhao, Oran Wang, et al. Hy-embodied-0.5: Embodied foundation models for real-world agents.arXiv preprint arXiv:2604.07430, 2026

    2026 arXiv

  36. [44]

    6-dof pose estimation of household objects for robotic manipulation: An accessible dataset and benchmark

    Stephen Tyree, Jonathan Tremblay, Thang To, Jia Cheng, Terry Mosier, Jeffrey Smith, and Stan Birchfield. 6-dof pose estimation of household objects for robotic manipulation: An accessible dataset and benchmark. In2022 IEEE/RSJ International Conference on Intelligent Robots and...

    2022

  37. [45]

    Vggt: Visual geometry grounded transformer

    Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 5294–5306, 2025

    2025

  38. [46]

    Locateanything: Fast and high-quality vision-language grounding with parallel box decoding.arXiv preprint arXiv:2605.27365, 2026

    Shihao Wang, Shilong Liu, Yuanguo Kuang, Xinyu Wei, Yangzhou Liu, Zhiqi Li, Yunze Man, Guo Chen, Andrew Tao, Guilin Liu, et al. Locateanything: Fast and high-quality vision-language grounding with parallel box decoding.arXiv preprint arXiv:2605.27365, 2026

    2026 arXiv

  39. [47]

    Orient anything v2: Unifying orientation and rotation understanding.arXiv preprint arXiv:2601.05573, 2026

    Zehan Wang, Ziang Zhang, Jiayang Xu, Jialei Wang, Tianyu Pang, Chao Du, Hengshuang Zhao, and Zhou Zhao. Orient anything v2: Unifying orientation and rotation understanding.arXiv preprint arXiv:2601.05573, 2026

    2026

  40. [48]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in neural information pro- cessing systems, volume 35, pages 24824–24837, 2022

    2022

  41. [49]

    Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes.arXiv preprint arXiv:1711.00199, 2017

    Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes.arXiv preprint arXiv:1711.00199, 2017

    2017 arXiv

  42. [50]

    Vlaser: Vision-language-action model with synergistic embodied reasoning.arXiv preprint arXiv:2510.11027, 2025

    Ganlin Yang, Tianyi Zhang, Haoran Hao, Weiyun Wang, Yibin Liu, Dehui Wang, Guanzhou Chen, Zijian Cai, Junting Chen, Weijie Su, et al. Vlaser: Vision-language-action model with synergistic embodied reasoning.arXiv preprint arXiv:2510.11027, 2025

    2025

  43. [51]

    Thinking in space: How multimodal large language models see, remember, and recall spaces

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 10632–10643, 2025

    2025

  44. [52]

    Swe-agent: Agent-computer interfaces enable automated software engineering

    John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. InAdvances in Neural Information Processing Systems, volume 37, pages 50528–50652, 2024

    2024

  45. [53]

    Depth anything v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. InAdvances in Neural Information Processing Systems, volume 37, pages 21875–21911, 2024. 13

    2024

  46. [54]

    Mmsi-bench: A benchmark for multi-image spatial intelligence.arXiv preprint arXiv:2505.23764, 2025

    Sihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang, Mo Li, Jingli Lin, Chenming Zhu, Xiaochen Chen, Haodong Duan, Xiangyu Yue, et al. Mmsi-bench: A benchmark for multi-image spatial intelligence.arXiv preprint arXiv:2505.23764, 2025

    2025 arXiv

  47. [55]

    Agentic robot: A brain-inspired framework for vision-language-action models in embodied agents.arXiv preprint arXiv:2505.23450, 2025

    Zhejian Yang, Yongchao Chen, Xueyang Zhou, Jiangyue Yan, Dingjie Song, Yinuo Liu, Yuting Li, Yu Zhang, Pan Zhou, Hechang Chen, et al. Agentic robot: A brain-inspired framework for vision-language-action models in embodied agents.arXiv preprint arXiv:2505.23450, 2025

    2025 arXiv

  48. [56]

    React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  49. [57]

    Spatial mental modeling from limited views

    Baiqiao Yin, Qineng Wang, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, et al. Spatial mental modeling from limited views. InStructural Priors for Vision Workshop at ICCV’25, 2025

    2025

  50. [58]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. InAdvances in Neural Information Processing Systems, volume 38, pages 113222–113244, 2025

    2025

  51. [59]

    Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, 2026

    Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, 2026

    2026 arXiv

  52. [60]

    Harness vla: Steering frozen vlas into reliable manipulation primitives via memory-guided agents.arXiv preprint arXiv:2607.08448, 2026

    Yixian Zhang, Huanming Zhang, Feng Gao, Xiao Li, Zhihao Liu, Chunyang Zhu, Jiaxing Qiu, Yuchen Yan, Jiyuan Liu, Wenhao Tang, et al. Harness vla: Steering frozen vlas into reliable manipulation primitives via memory-guided agents.arXiv preprint arXiv:2607.08448, 2026

    2026 arXiv

  53. [61]

    Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614, 2024

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614, 2024

    2024 arXiv

  54. [62]

    Roborefer: Towards spatial referring with reasoning in vision-language models for robotics

    Enshen Zhou, Jingkun An, Cheng Chi, Yi Han, Shanyu Rong, Chi Zhang, Pengwei Wang, Zhongyuan Wang, Tiejun Huang, Lu Sheng, et al. Roborefer: Towards spatial referring with reasoning in vision-language models for robotics. InAdvances in Neural Information Processing Systems, vol...

    2025

  55. [63]

    name": "query_locate

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023. 14 Ap...

    2023

  56. [64]

    Do not turn a plan, a proposed tool action, answer-option analysis, or an uncertain guess into an observed fact

    Every perceptual claim must be traceable to the current tool result, an explicit visual observation written in the current original reasoning, or a previous unit. Do not turn a plan, a proposed tool action, answer-option analysis, or an uncertain guess into an observed fact

  57. [72]

    Exhaustively verbalize every item and every field exposed by the current tool result. Preserve the returned count, every instance, image index, label, bounding box, point, polygon component and vertex, depth value, pose field, motion field, direction, axis, error, and other re...

  58. [75]

    Use only entities, attributes, values, and relations present in the evidence units

  59. [76]

    Merge evidence in dependency order and remove repeated or superseded statements, but do not discard an explicit visual or spatial fact merely because it is unnecessary for choosing the answer

  60. [77]

    Preserve the concrete visual-spatial descriptions carried by the evidence units and include every essential inference needed to support all clauses of the answer

  61. [78]

    Never mention tools, calls, results, evidence units, confidence scores, or this instruction

    Write as direct visual reasoning. Never mention tools, calls, results, evidence units, confidence scores, or this instruction

  62. [79]

    Do not invent entities, attributes, measurements, or relations. 34

  63. [80]

    answer":

    Put reasoning and answer in separate XML sections. Copy the provided final answer exactly. 12 13Return the structured result in this form: 14<internalization_target> 15<reasoning_chain> 16Complete, detailed, tool-free perceptual reasoning. 17</reasoning_chain> 18<final_answer>...

  64. [81]

    Locate the beige trash can and the blue trash can in the first image

  65. [82]

    Get depth for both to confirm which is closer

  66. [83]

    This includes left/right, above/below, front/behind, near/far, overlap, containment, adjacency, occlusion, orientation, size, appearance, and count descriptions

    Preserve every explicit visual observation and absolute or relative visual-spatial relation from the current original reasoning, even when it is unrelated to the current task. This includes left/right, above/below, front/behind, near/far, overlap, containment, adjacency, occlu...

  67. [84]

    Confirm left/right position based on location

  68. [85]

    camera motion

    Use ‘query_pose‘ with "camera motion" to determine the rotation direction from image 1 to image 2. **Execution:**

  69. [86]

    beige trash can

    ‘query_locate‘ for "beige trash can" and "blue trash can" in image 1

  70. [87]

    ‘query_depth‘ on their centers

  71. [88]

    camera motion

    ‘query_pose‘ for "camera motion" between image 1 and 2. Let’s refine the queries. - Query 1: "beige trash can with black bag" in image 1. - Query 2: "blue trash can with clear bag" in image 1. - Query 3: "camera motion" from image 1 to 2. Wait, looking at image 2, the blue bin...

  72. [89]

    Since 0.776 < 0.928, the **blue receptacle is closer**

    **Distances**: The depth query shows the beige receptacle (point [310, 524]) is at **0.928m** and the blue receptacle (point [773, 643]) is at **0.776m**. Since 0.776 < 0.928, the **blue receptacle is closer**

  73. [90]

    Therefore, the closer receptacle (blue) is located on the **right**

    **Location**: In the first image, the beige can is at x=310 (left side) and the blue can is at x=773 (right side). Therefore, the closer receptacle (blue) is located on the **right**

  74. [91]

    answer":

    **Camera Motion**: The pose query for camera motion from image 1 to 2 indicates a ‘view_rotation‘ direction of **right**. Matching these facts to the options: Closer: Blue Location: Right Rotation: Right This corresponds to Option D. {"answer": "D"} H.2 Case 2: Two-Image Camer...

  75. [92]

    white sandals with pearl straps

    **Locate the sandals in Image 1:** I need to find the two white sandals. Looking at Image 1, there are two distinct white heeled sandals sitting on a pink cloth. They have pearl-like embellishments on the toe strap. One is slightly behind/left of the other. Wait, let’s look cl...

  76. [93]

    Specifically, look at ‘x_max‘ (the third value in ‘[x_min, y_min, x_max, y_max]‘)

    **Determine the right-most sandal:** Compare the ‘bbox_2d‘ values. Specifically, look at ‘x_max‘ (the third value in ‘[x_min, y_min, x_max, y_max]‘). - Sandal A (foreground): likely has a larger x_max because it’s more to the right. - Sandal B (background): likely has a smaller x_max

  77. [94]

    is estimated at

    Use calibrated perceptual wording such as "is estimated at", "appears", "approximately", or "the result indicates" where appropriate, because tool outputs are perception estimates. Keep every returned numeric value exactly as given: do not round, alter, normalize again, or inv...

  78. [95]

    **Find max y-coordinate:** Once the right-most sandal is identified, take its ‘y_max‘ (the fourth value in the bbox)

  79. [96]

    camera motion

    **Determine camera motion:** Use ‘query_pose‘ with query "camera motion" comparing Image 1 to Image 2. - Image 1 shows a close-up of the shoes on a bed/surface with a headboard behind. - Image 2 shows the same shoes but from a different angle. The background has changed signif...

  80. [97]

    That’s the front/right sandal [143, 508, 765, 856]

    Target sandal: The one with larger max x. That’s the front/right sandal [143, 508, 765, 856]. Max x is 765. The other is [159, 492, 715, 747], max x 715

  81. [98]

    Max y of target sandal: 856

  82. [99]

    forward-left

    Camera translation: Backward (zoom out/move away) and Left (sneaker moves from right edge to center). Therefore, Option C: Maximum y-coordinate: 856; Camera translation: backward-left. Let’s double check the bounding box values provided in the prompt’s hidden ground truth or j...

  83. [100]

    Yellow ball)

    **Which object is further to the right in the first image?** (Clear plastic bottle vs. Yellow ball)

  84. [101]

    Yellow ball)

    **Which object is closer to the camera in the first image?** (Clear plastic bottle vs. Yellow ball)

  85. [102]

    clear plastic bottle

    **In which direction did the camera translate to capture the second image?** Let’s break this down step-by-step. **Step 1: Analyze the first image for horizontal position (left/right).** - I need to locate the "clear plastic bottle" and the "yellow ball". - Looking at Image 1,...

  86. [103]

    clear plastic bottle

    Locate "clear plastic bottle" and "yellow ball" in Image 1 to confirm positions and get points for depth

  87. [104]

    Query depth for the centers of these two objects in Image 1 to confirm which is closer

  88. [105]

    Clearly resolve a prior visual estimate when stronger tool evidence supersedes it

    Explain how the new evidence confirms, revises, or extends prior evidence. Clearly resolve a prior visual estimate when stronger tool evidence supersedes it

  89. [106]

    camera motion

    Query "camera motion" between Image 1 and Image 2. **Execution:**

  90. [107]

    clear plastic bottle

    ‘query_locate‘ for "clear plastic bottle" and "yellow ball" in image 1

  91. [108]

    ‘query_depth‘ using the points obtained

  92. [109]

    camera motion

    ‘query_pose‘ for "camera motion". Let’s refine the queries. - Query 1: ‘query_locate‘ with query="clear plastic bottle", image_indices=[1] - Query 2: ‘query_locate‘ with query="yellow ball", image_indices=[1] - Query 3: ‘query_depth‘ using points from above. - Query 4: ‘query_...

  93. [110]

    **Which object is further to the right?** - Clear plastic bottle center: [329, 417] - Yellow ball center: [644, 348] - The x-coordinate of the yellow ball (644) is greater than the x-coordinate of the bottle (329), so the yellow ball is further to the right

  94. [111]

    **Which object is closer to the camera?** - Bottle depth: 0.351 m - Ball depth: 0.637 m - Smaller depth means closer, so the clear plastic bottle (0.351 m) is closer to the camera than the yellow ball (0.637 m)

  95. [112]

    forward-left

    **Camera translation direction:** - The query_pose result shows position.direction = "forward-left" when going from image 1 to image 2. So the correct option should state: - The yellow ball is further to the right✓ - The clear plastic bottle is closer to the camera✓ - Camera t...

  96. [116]

    Do not repeat unchanged evidence already preserved in a previous unit, but never discard newly returned or newly stated visual information merely because it is task-irrelevant

  97. [127]

    13 14Return the structured result in this form: 15<evidence_unit> 16<new_evidence> 17Complete readable natural-language statements verbalizing all returned information

    Empty evidence is valid only when the interaction failed or truly returned no information; preserve the failure or empty-result details themselves. 13 14Return the structured result in this form: 15<evidence_unit> 16<new_evidence> 17Complete readable natural-language statement...

  98. [291]

    31- Output: count and zero or more matched instances

    query_locate — open-vocabulary 2D localization 30- Input: a short visible object/category description and optional 1-based image_indices. 31- Output: count and zero or more matched instances. Each instance may contain bbox_2d=[x1,y1,x2,y2] and point_2d=[x,y]. 32- Coordinates u...

  99. [362]

    38- The exposed segmentation result contains bbox_2d, point_2d, and polygon_2d

    query_segment — open-vocabulary instance segmentation 37- Input: a short visible object/category description and optional 1-based image_indices. 38- The exposed segmentation result contains bbox_2d, point_2d, and polygon_2d. point_2d is derived from bbox_2d rather than indepen...

  100. [443]

    46- Output: one result per point with point_2d and depth_m

    query_depth — metric camera-axis depth at image points 45- Input: one or more normalized 0–999 points and optional 1-based image_indices. 46- Output: one result per point with point_2d and depth_m. 47- depth_m is estimated camera-axis distance in meters. Smaller depth_m means ...

  101. [514]

    Output may include image_index, bbox_2d, visible_side, and facing_direction_camera

    query_pose — object orientation or cross-view camera motion 52- Object mode input: a short object description, optionally with image_indices. Output may include image_index, bbox_2d, visible_side, and facing_direction_camera. 33 53- visible_side names the object’s side seen by...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.