Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Visual Agentic AI for Spatial Reasoning with a Dynamic API

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that VADAR, which generates its own Python API from a small batch of queries, beats static-API program synthesis by more than 20% on 3D spatial reasoning and that the remaining bottleneck is the vision specialists.

desk verdict Solid dynamic-API idea, but the API construction borrows test queries without a disclosed split, so the >20% gains over static APIs are not yet fully convincing. read the letter →

arxiv 2502.06787 v2 pith:WBVDYCFF submitted 2025-02-10 cs.CV

Damiano Marsili , Rohun Agrawal , Yisong Yue , Georgia Gkioxari This is my paper · ORCID
classification cs.CV
keywords visualreasoningspatialprogramsynthesisagenticAIdynamicAPI3Dunderstandingvision-languagemodelsbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the limiting factor in 3D spatial reasoning is not the reasoning agent but the vision specialists it calls, and that an agent that writes its own API can outperform fixed-API program synthesis. The authors build VADAR, in which LLM agents propose and implement reusable Python functions from a small batch of un-answered queries, then use those functions to answer new queries. VADAR beats ViperGPT and VisProg by more than 20% on CLEVR and on the new OMNI3D-BENCH, and with perfect vision specialists it reaches 83.0% and 94.4% accuracy respectively. The paper introduces OMNI3D-BENCH, 500 free-form queries built from Omni3D scenes, as a harder test of 3D understanding than existing VQA benchmarks. If correct, the work suggests that improving specialized detectors, depth estimators, and attribute predictors is a more direct route to 3D visual reasoning than scaling monolithic VLMs.

What carries the argument

The central object is the dynamically generated API: a growing set of reusable Python methods such as find closest object 3D, is behind, count objects by attributes and position, and is left of. It is generated by two LLM agents (Signature and Implementation) with a deterministic Python interpreter as critic, then consumed by a Program Agent. The mechanism works by decomposing complex 3D queries into subproblems that get packaged as named, tested functions, so that query programs become concise calls to accurate methods rather than long error-prone code. The paper contrasts this with the static, human-defined DSLs in ViperGPT and VisProg, which force the LLM to either write everything from scratch or fall back to holistic VQA.

What would settle it

Count the overlap between the 15 API-generation queries and the evaluation tuples on CLEVR and OMNI3D-BENCH; if any overlap exists, or if re-running API generation on a disjoint held-out batch of 15 queries drops accuracy to the level of ViperGPT or VisProg, the central claim of a general dynamic API is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms: VADAR is a zero-shot visual program synthesis method that dynamically generates a Pythonic API before answering queries. In the API Generation stage, a Signature Agent scans a batch of 15 questions, without answers, and proposes method signatures; an Implementation Agent writes those methods, using other API methods, with a deterministic Test Agent catching Python errors and resolving dependencies depth-first. In the Program Synthesis stage, a Program Agent plans and emits code that calls the generated API, and an Execution Agent runs it with retries. The central empirical discovery is that this dynamic API lifts accuracy on complex 3D spatial queries from roughly 26-31% for the static-API baselines to 53.6% on CLEVR and 40.4% on OMNI3D-BENCH, and that with oracle vision specialists the same programs reach 83.0% and 94.4%, far above the best VLMs. The paper concludes that the bottleneck is the vision specialist stack, not the reasoning layer.

Load-bearing premise

The API-generation stage inspects 15 queries from the evaluation benchmark without answers; if those queries are not disjoint from the test set, the reported gains over static-API baselines could partly come from test-set adaptation rather than the dynamic-API mechanism.

Editorial extensions

If this is right

  • VADAR outperforms ViperGPT and VisProg by more than 20% on both CLEVR and OMNI3D-BENCH, with the gap widening on queries that require multiple grounding and inference steps.
  • With oracle vision specialists, VADAR attains 83.0% on CLEVR and 94.4% on OMNI3D-BENCH, implying that the remaining gap to competitive VLMs is due to the object detector, depth estimator, and attribute predictor rather than the program synthesis.
  • The training-free nature of VADAR lets it transfer to new domains, such as real-world Omni3D scenes, where 3D supervision for training neuro-symbolic modules like LEFT is impractical.
  • On GQA, which mostly tests appearance and single-step inference, VADAR is roughly on par with VisProg and below GPT4o, confirming that the dynamic API's advantage is specific to complex spatial reasoning rather than generic VQA.
  • The API is built from only 15 questions, so the signature and implementation agents run once, making per-query runtime comparable to prior program synthesis methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural follow-up is to test cross-benchmark transfer: generate the API on CLEVR questions and evaluate on OMNI3D-BENCH, and vice versa; if the API is genuinely general-purpose, performance should remain high.
  • The oracle experiments imply an engineering program: invest in specialized perception models (detection, depth, attribute grounding) and compose them with an agent-generated API, rather than scaling monolithic VQA models.
  • One could extend VADAR to let the Program Agent inspect the image during planning, which the paper lists as a limitation; this might crack queries requiring five or more inference steps.
  • The dynamic API could be reused across queries within a scene or across scenes, amortizing its cost further and enabling lifelong skill accumulation beyond the paper's single-benchmark setting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces VADAR, a two-stage agentic program-synthesis method for 3D spatial visual reasoning. In the API Generation stage, LLM agents propose and implement reusable Python functions from a batch of N=15 queries without answers; in the Program Synthesis stage, a Program Agent composes these functions into executable programs that call vision specialists (an object localizer, a depth estimator, and a VQA module). The method is evaluated on CLEVR, on a newly introduced OMNI3D-BENCH benchmark of 500 free-form 3D questions, and on GQA. The central claims are that VADAR outperforms static-API program-synthesis baselines (ViperGPT and VisProg) by more than 20% on CLEVR and OMNI3D-BENCH, that it is competitive with strong monolithic VLMs, and that oracle experiments show the residual gap is due to vision-specialist errors rather than program-synthesis errors.

Significance. The dynamic-API idea is a useful and timely departure from static, human-defined DSLs in visual program synthesis, and the paper provides a clean decomposition of program correctness versus vision-execution accuracy through oracle evaluations. The new OMNI3D-BENCH benchmark addresses a real gap: most existing VQA benchmarks emphasize appearance and 2D relations, whereas this benchmark requires multi-step grounding and 3D inference. The comparison against LEFT is valuable because it demonstrates that a training-free method can match or exceed a trained neuro-symbolic method on CLEVR. The ablations (No-API vs API, weak ICL, pseudo ICL) give evidence for the value of the proposed components. However, the evaluation protocol has a load-bearing fairness issue: the API is generated from queries drawn from the same benchmark used for evaluation, and the paper does not disclose whether those queries are disjoint from the test queries. This weakens the comparison to static-API baselines and the zero-shot framing, and it needs to be addressed before the central claims can be accepted.

major comments (3)
  1. [Section 3.1, Algorithm 1; Section 4.2, Table 1] The API Generation stage takes a set Q of queries, batches them with N=15, and proposes reusable functions; the Program Synthesis stage (Algorithm 2) is then evaluated on image-query pairs from the same benchmark. The paper never states that Q is disjoint from the evaluation set D, nor that Q is a held-out split. Because CLEVR questions are templated and OMNI3D-BENCH contains only 500 free-form questions, 15 queries can expose a substantial fraction of the query distribution or template structures. ViperGPT and VisProg use fully static, human-defined APIs, so any adaptation to the evaluation query distribution gives VADAR an asymmetric advantage. This directly affects the paper's central claim of outperforming the static baselines by more than 20%. The authors should specify the exact split, provide the 15 queries used for API generation, and ideally evaluate VADAR with an API generated from a disjoint or cross-domain query set, or otherwise quantify the sensitivity of the gains to the choice of N and to the query sample.
  2. [Section 4.2, Table 2] The oracle evaluation on OMNI3D-BENCH is performed on a 'smaller subset of 50 queries' with manual verification, but the selection procedure is not described. The resulting 94.4% accuracy is used to support the claim that VADAR's bottleneck is vision-specialist performance and that the dynamic API supports a wider range of queries. If the 50 queries were chosen because they were particularly amenable to the generated API, the oracle number may not be representative of the full benchmark. The authors should report how the subset was sampled, ideally with a random or predefined split, list or release the 50 queries, and report confidence intervals. Without this, the oracle comparison to GPT4o and Claude3.5-Sonnet on the same subset is difficult to interpret.
  3. [Section 4.2, paragraph on ViperGPT vs VisProg vs VADAR] The main comparison uses GPT-3.5 for ViperGPT and GPT-4o for VADAR; the paper states this was because GPT-3.5 performed better for ViperGPT, but no supporting ablation is provided. Since the central claim is that the dynamic API, not the underlying LLM, drives the large improvement, the comparison should include ViperGPT and VisProg with the same LLM as VADAR, or at least provide an ablation showing that the LLM choice alone does not account for the gap. This is particularly relevant because the >20% margin is the headline quantitative result.
minor comments (4)
  1. [Appendix C] The benchmark construction discards questions that are 'simplistic' and keeps only queries with complex inference steps; the exact filtering criteria and the annotator instructions should be released so that the benchmark's difficulty and selection bias can be assessed by others.
  2. [Figure 14, Implementation Agent Prompt for OMNI3D-BENCH] The prompt example contains a typo ('epislon' instead of 'epsilon') and an undefined variable 'object_coords' in the get_object_list example; since these examples are part of the prompt, they should be corrected for reproducibility.
  3. [Figure 16, Program Agent Prompt for OMNI3D-BENCH] The text 'questions rae all of the objects' contains a typo ('rae' instead of 'are'); this should be fixed in the released prompt templates.
  4. [Section 4.3, Table 4] The ablation 'CLEVR 100' is described as a randomly selected subset of 100 questions, but the seed or selection procedure is not given; specifying this would improve reproducibility of the ablation results.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the dynamic API is generated from queries without answers, and the reported benchmark gains are empirical comparisons rather than consequences of the API definition.

full rationale

VADAR's derivation chain is not circular. The API Generation stage consumes 15 queries without answers and produces reusable function signatures via LLM agents; the Program Synthesis stage then composes those functions on individual image-query pairs. The central claim, that VADAR outperforms ViperGPT and VisProg by more than 20%, is an experimental comparison, not a mathematical derivation from the API inputs. The generated API is not defined in terms of the benchmark answers, and no fitted parameter is renamed as a prediction. The only legitimate concern is that Algorithm 1's 'Questions Q' is not stated to be disjoint from Algorithm 2's evaluation pairs D; if Q overlaps D, the comparison would be test-set-adaptive rather than zero-shot. That is an evaluation-protocol and correctness risk, not circularity, because benchmark accuracy is not forced by construction from the 15 queries. The paper explicitly discloses the 15-query input: 'our API agent uses a small sample (=15) of questions only, without answers, to construct the API.' The oracle experiments (83.0% on CLEVR and 94.4% on a 50-query OMNI3D subset) and the No-API ablation provide independent content. No load-bearing self-citation chain is present; Omni3D is a data source, not an authority invoked to justify the method's conclusions. Accordingly, no circular step is identifiable from the paper's own statements.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the accuracy of off-the-shelf vision models, the correctness of the manually constructed benchmark, and the assumption that using a sample of benchmark questions to build the API does not leak test-set information. No new physical entities are postulated.

free parameters (5)
  • LLM temperature = 0.7
    Used for all agents; chosen by hand with no sensitivity analysis (Section 4.2).
  • API generation batch size N = 15
    Number of benchmark questions used to propose API signatures (Algorithm 1, Section 3.1).
  • Max implementation attempts = 5
    Number of retries before skipping or deleting methods (Algorithm 1, Section 3.1).
  • same_object IoU threshold = 0.92
    Hand-set threshold for deciding if two points refer to the same object (appendix, predefined module implementations).
  • Max program retries = 5
    Number of regeneration attempts for the Program Agent (Algorithm 2, Section 3.2).
assumptions (4)
  • domain assumption Vision specialists (Molmo, GroundingDINO, SAM, UniDepth, GPT4o VQA) provide sufficiently accurate localization, depth, and attribute predictions for the synthesized programs to yield correct answers.
    The entire execution accuracy depends on these off-the-shelf models; oracle results show the ceiling is much higher, so this assumption is load-bearing and acknowledged as the main bottleneck.
  • domain assumption UniDepth monocular depth is a reliable metric depth signal for scaling 2D sizes to 3D.
    The OMNI3D-BENCH prompts instruct agents that 3D size equals 2D size times depth; if depth is unreliable, the numerical answers fail.
  • domain assumption The 15 sample queries used for API generation do not compromise the fairness of evaluation.
    The paper does not state whether these queries are disjoint from the evaluation set; the evaluation may be contaminated.
  • domain assumption Ground-truth answers for OMNI3D-BENCH, manually derived from Omni3D annotations, are correct and unambiguous.
    The benchmark answers are collected manually; ambiguous questions could undermine the evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Visual Agentic AI for Spatial Reasoning with a Dynamic API." pith.science (2026). https://pith.science/paper/WBVDYCFF

@misc{pith2026250206787,
  author       = {Pith},
  title        = {Pith review of: Visual Agentic AI for Spatial Reasoning with a Dynamic API},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WBVDYCFF}},
  note         = {Machine review of arXiv:2502.06787}
}
read the original abstract

Visual reasoning -- the ability to interpret the visual world -- is crucial for embodied agents that operate within three-dimensional scenes. Progress in AI has led to vision and language models capable of answering questions from images. However, their performance declines when tasked with 3D spatial reasoning. To tackle the complexity of such reasoning problems, we introduce an agentic program synthesis approach where LLM agents collaboratively generate a Pythonic API with new functions to solve common subproblems. Our method overcomes limitations of prior approaches that rely on a static, human-defined API, allowing it to handle a wider range of queries. To assess AI capabilities for 3D understanding, we introduce a new benchmark of queries involving multiple steps of grounding and inference. We show that our method outperforms prior zero-shot models for visual reasoning in 3D and empirically validate the effectiveness of our agentic framework for 3D spatial reasoning tasks. Project website: https://glab-caltech.github.io/vadar/

Figures

Figures reproduced from arXiv: 2502.06787 by the authors.

Figure 1
Figure 1. Spatial reasoning in 3D is challenging as it requires multiple steps of grounding and inference. We introduce a benchmark for [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview. VADAR consists of an API generation stage and a program synthesis stage. The Signature & Implementation Agents generate an API that is used by the Program Agent to produce a program to answer the question, executed by the Execution Agent. aid in solving the queries. First, the Signature Agent re￾ceives a batch of N queries (N = 15), without answers, and is instructed to produce general method signatures fo… view at source ↗
Figure 3
Figure 3. LEFT [14] vs VADAR on CLEVR. LEFT requires su￾pervision. We vary the amount of training data (x-axis) and report accuracy (y-axis). VADAR requires no supervision but takes in 15 queries without answers to guide the creation of the API. VADAR outperforms LEFT trained with ≤ 10, 000 supervised examples. From Tab. 1, we observe that on CLEVR, GPT4o, Claude-Sonnet, and Gemini perform best on average while VADAR slightly… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Program outputs for VisProg, ViperGPT and VADAR. For each example, we show the query, the input image, and the method’s program generations. Queries are from our benchmark and pertain to 3D understanding of scenes. Zoom-in to read the programs. observe that ViperGPT an…
Figure 5
Figure 5. Figure 5: (a) The No-API agent produces longer programs and is prone to errors, often mistakenly using depth for left/right com￾parisons. (b) In contrast, our agentic VADAR creates shorter pro￾grams by leveraging methods from the API. From Tab. 4 we observe that the No-API Agent…
Figure 6
Figure 6. Figure 6: VADAR program outputs on CLEVR [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: VADAR program outputs on OMNI3D-BENCH [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: VADAR program outputs on GQA [16]. 3 [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Pre-defined Modules for CLEVR. These modules are used to initialize the dynamic API. As CLEVR defines size to be either large or small, we omit the get 2D object size method. 4 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Pre-defined Modules for OMNI3D-BENCH. These modules are used to initialize the dynamic API. 5 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Python Implementation of Predefined Modules. VADAR uses Molmo [8] for object detection, SAM2 [22] for segmentation, GPT4o [1] for VQA, and UniDepth [31] for depth estimation. 6 [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Signature Agent Prompt used for both CLEVR and OMNI3D-BENCH. 7 [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Implementation Agent Prompt for CLEVR. This prompt differs from the prompt used for OMNI3D-BENCH as we omit examples illustrating usage of the get 2D object size method. The prompt features Weak ICL examples illustrating correct usage of the pre-defined modules, as we…
Figure 14
Figure 14. Figure 14: Implementation Agent Prompt for OMNI3D-BENCH. The prompt features Weak ICL examples illustrating correct usage of the pre-defined modules, as well as Pseudo ICL in the form of natural language instructions and definitions. 9 [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Program Agent Prompt for CLEVR. In the prompt, we provide a list of all available attributes in CLEVR, a Pseudo ICL example in natural language, and some helpful tips. 10 [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Program Agent Prompt for OMNI3D-BENCH. The prompt features Pseudo ICL in the form of two natural language examples and helpful tips for handling 2D and 3D dimensions. 11 [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL

    cs.CV 2025-12 conditional novelty 6.0 of 10

    A two-phase interactive RL framework (DIRL) lets a 3B VLM learn to coordinate multiple vision and robot tools, reaching top benchmark scores and 86% real-robot pick-and-place success.

Reference graph

Works this paper leans on

86 extracted references · 69 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Claude, 2024

    Anthropic. Claude, 2024. 1, 2, 5, 6

    2024

  3. [3]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015. 2, 3

    2015

  4. [4]

    Arc prize, 2024

    ARC-AGI. Arc prize, 2024. 3

    2024

  5. [5]

    Omni3D: A large benchmark and model for 3D object detection in the wild

    Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3D: A large benchmark and model for 3D object detection in the wild. In CVPR, 2023. 2, 3, 5, 1

    2023

  6. [6]

    Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties. In CVPR, 2024. 1, 2, 5, 6

    2024

  7. [7]

    Spatial- rgpt: Grounded spatial reasoning in vision-language models

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Rui- han Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatial- rgpt: Grounded spatial reasoning in vision-language models. In NeurIPS, 2024. 2

    2024

  8. [8]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tri- pathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146, 2024. 1, 3, 5, 6

    arXiv 2024

Show all 86 references
  1. [9]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  2. [10]

    Dreamcoder: growing generalizable, interpretable knowledge with wake– sleep bayesian program learning

    Kevin Ellis, Lionel Wong, Maxwell Nye, Mathias Sable- Meyer, Luc Cary, Lore Anaya Pozo, Luke Hewitt, Armando Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: growing generalizable, interpretable knowledge with wake– sleep bayesian program learning. Philosophical Transac- tion...

    2023

  3. [11]

    Symbolic regression with a learned concept library

    Arya Grayeli, Atharva Sehgal, Omar Costilla-Reyes, Miles Cranmer, and Swarat Chaudhuri. Symbolic regression with a learned concept library. In NeurIPS, 2024. 3

    2024

  4. [12]

    Visual program- ming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual program- ming: Compositional visual reasoning without training. In CVPR, 2023. 2, 5, 6, 8, 1

    2023

  5. [13]

    Ns3d: Neuro- symbolic grounding of 3d objects and relations

    Joy Hsu, Jiayuan Mao, and Jiajun Wu. Ns3d: Neuro- symbolic grounding of 3d objects and relations. In CVPR,

  6. [14]

    What’s left? concept grounding with logic-enhanced foun- dation models

    Joy Hsu, Jiayuan Mao, Josh Tenenbaum, and Jiajun Wu. What’s left? concept grounding with logic-enhanced foun- dation models. In NeurIPS, 2024. 3, 5, 6, 7

    2024

  7. [15]

    Scenecraft: An llm agent for synthesizing 3d scenes as blender code

    Ziniu Hu, Ahmet Iscen, Aashi Jain, Thomas Kipf, Yisong Yue, David A Ross, Cordelia Schmid, and Alireza Fathi. Scenecraft: An llm agent for synthesizing 3d scenes as blender code. In ICML, 2024. 3

    2024

  8. [16]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 2, 3, 5, 7

    2019

  9. [17]

    Mantis: Interleaved multi-image instruction tuning, 2024

    Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning, 2024. 5, 6, 1

    2024

  10. [18]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. 2, 5

    2017

  11. [19]

    What’s” up” with vision-language models? investigating their strug- gle with spatial reasoning

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s” up” with vision-language models? investigating their strug- gle with spatial reasoning. arXiv preprint arXiv:2310.19785,

    arXiv

  12. [20]

    Referitgame: Referring to objects in pho- tographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in pho- tographs of natural scenes. In EMNLP, 2014. 3

    2014

  13. [21]

    Vilt: Vision- and-language transformer without convolution or region su- pervision

    Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision- and-language transformer without convolution or region su- pervision. In ICML, 2021. 2

    2021

  14. [22]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In ICCV, 2023. 3, 6

    2023

  15. [23]

    Human-level concept learning through proba- bilistic program induction

    Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through proba- bilistic program induction. Science, 350(6266):1332–1338,

  16. [24]

    Lab-bench: Measuring capabilities of language models for biology research

    Jon M Laurent, Joseph D Janizek, Michael Ruzo, Michaela M Hinks, Michael J Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D White, and Samuel G Rodriques. Lab-bench: Measuring capabilities of language models for biology research. arXiv preprint arXiv:2407.10362, 2024. 3

    2024 arXiv

  17. [25]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In CVPR, 2022. 2

    2022

  18. [26]

    Reasoning paths with reference objects elicit quanti- tative spatial reasoning in large vision-language models

    Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, and David Acuna. Reasoning paths with reference objects elicit quanti- tative spatial reasoning in large vision-language models. In EMNLP, 2024. 3

    2024

  19. [27]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 3

    2023 arXiv

  20. [28]

    The neuro-symbolic concept 9 learner: Interpreting scenes, words, and sentences from nat- ural supervision

    Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro-symbolic concept 9 learner: Interpreting scenes, words, and sentences from nat- ural supervision. ICLR, 2019. 3

    2019

  21. [29]

    Simple open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection. In ECCV, 2022. 2

    2022

  22. [30]

    Linc: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers

    Theo X Olausson, Alex Gu, Benjamin Lipkin, Cedegao E Zhang, Armando Solar-Lezama, Joshua B Tenenbaum, and Roger Levy. Linc: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers. In EMNLP, 2023. 3

    2023

  23. [31]

    Unidepth: Universal monocular metric depth estimation

    Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In CVPR,

  24. [32]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 2

    2021

  25. [33]

    Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer

    Ren ´e Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE TPAMI, 2020. 2

    2020

  26. [34]

    Neurosymbolic programming for science

    Jennifer J Sun, Megan Tjandrasuwita, Atharva Sehgal, Ar- mando Solar-Lezama, Swarat Chaudhuri, Yisong Yue, and Omar Costilla Reyes. Neurosymbolic programming for science. In NeurIPS 2022 Workshop on AI for Science: Progress and Promises, 2022. 3

    2022

  27. [35]

    Vipergpt: Vi- sual inference via python execution for reasoning

    D ´ıdac Sur´ıs, Sachit Menon, and Carl V ondrick. Vipergpt: Vi- sual inference via python execution for reasoning. In ICCV,

  28. [36]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1, 2, 5, 6

    2023 arXiv

  29. [37]

    An in-context learning agent for formal theorem-proving

    Amitayush Thakur, George Tsoukalas, Yeming Wen, Jimmy Xin, and Swarat Chaudhuri. An in-context learning agent for formal theorem-proving. In CoLM, 2024. 3

    2024

  30. [38]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. NeurIPS, 2024. 1, 2, 3

    2024

  31. [39]

    Houdini: Lifelong learning as program synthesis

    Lazar Valkov, Dipak Chaudhari, Akash Srivastava, Charles Sutton, and Swarat Chaudhuri. Houdini: Lifelong learning as program synthesis. Advances in neural information pro- cessing systems, 31, 2018. 3

    2018

  32. [40]

    V oyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandku- mar. V oyager: An open-ended embodied agent with large language models. TMLR, 2024. 3

    2024

  33. [41]

    Chain-of-thought prompting elicits reasoning in large lan- guage models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. NeurIPS, 2022. 5, 8

    2022

  34. [42]

    Promptchainer: Chaining large language model prompts through visual programming, 2022

    Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. Promptchainer: Chaining large language model prompts through visual programming, 2022. 8

    2022

  35. [43]

    Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts

    Tongshuang Wu, Michael Terry, and Carrie Jun Cai. Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, New York, NY , USA, 2022. Association for C...

    2022

  36. [44]

    Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie

    Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How Multi- modal Large Language Models See, Remember and Recall Spaces. arXiv preprint arXiv:2412.14171, 2024. 3, 5, 1, 2

    2024 arXiv

  37. [45]

    Is there a sofa in the image?

    Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vi- sion language pre-training: Aligning texts with visual con- cepts. arXiv preprint arXiv:2111.08276, 2021. 2 10 Visual Agentic AI for Spatial Reasoning with a Dynamic API Supplementary Material Method CLEVR OMNI3D-BENCH VL...

    2021 arXiv

  38. [48]

    \"\" Returns a list of objects in the images Args: image (IMAGE): Image to search for objects in Returns: list: List of strings corresponding to all of the objects in the image. \

    When searching for objects to compare to a reference object, make sure to remove the reference object from the retrieved objects. You can check if two objects are the same with the same_object method. Do not define new methods here, simply solve the problem using the existing ...

  39. [49]

    On" is defined as the closest object ABOVE another object. Only use this definition for

    2D distance/size refers to distance/size in pixel space. 2) 3D distance/size refers to distance/size in the real world. 3D size is equal to 2D size times the depth of the object. 3) "On" is defined as the closest object ABOVE another object. Only use this definition for "on". ...

  40. [50]

    6) "Depth" measures distance from the camera in 3D

    Width is the same as length. 6) "Depth" measures distance from the camera in 3D. Here are some helpful tips:

  41. [51]

    What is the shape of the matte object in front of the red cylinder?

    When you need to search over objects satisfying a condition, remember to check all the objects that satisfy the condition and don’t just return the first one. 2) You already have an initialized variable named "image" - no need to initialize it yourself! 3) When searching for o...

  42. [52]

    Find all the cylinders (loc(image, ’cylinders’))

  43. [53]

    If cylinders are found, loop through each of the cylinders found

  44. [54]

    Store the red cylinder if you find it and break from the loop

    For each cylinder found, check if the color of this cylinder is red. Store the red cylinder if you find it and break from the loop

  45. [56]

    For each object, check if the object is rubber (matte is not in the available attributes, so we replace it with rubber)

  46. [57]

    For each rubber object O you found, check if the depth of O is less than the depth of the red cylinder

  47. [58]

    final_result

    If that is true, return the shape of that object Now here is an API of methods, you will want to solve the problem in a logical and sequential manner as I showed you ------------------ API ------------------ {pre_defined_signatures} {api} ------------------ API ---------------...

  48. [60]

    You already have an initialized variable named "image" - no need to initialize it yourself! 3) Do not define new methods here, simply solve the problem using the existing methods

  49. [61]

    final_result

    When searching for objects to compare to a reference object, make sure to remove the reference object from the retrieved objects. You can check if two objects are the same with the same_object method. Again, available sizes are {{small, large}}, available shapes are {{square, ...

  50. [62]

    Find all the pillows (loc(image, ’pillow’))

  51. [63]

    If pillows are found, loop through each of the pillows found

  52. [64]

    Store the blue pillow if you find it and break from the loop

    For each pillow found, check if the color of this pillow is blue. Store the blue pillow if you find it and break from the loop

  53. [66]

    For each object, check if the object is red

  54. [67]

    For each red object O you found, check if the depth of O is less than the depth of the blue pillow

  55. [68]

    How many objects have the same color as the metal bowl?

    If that is true, return the shape of that object. Example 2: Question: "How many objects have the same color as the metal bowl?" Solution:

  56. [69]

    Find all the bowls (loc(image, ’bowls’))

  57. [70]

    If bowls are found, loop through each of the bowls found

  58. [71]

    Store the metal bowl if you find it and break from the loop

    For each bowl found, check if the material of this bowl is metal. Store the metal bowl if you find it and break from the loop

  59. [72]

    Find and store the color of the metal bowl

  60. [73]

    Find all the objects

  61. [74]

    If it is, skip it

    For each object O, check if O is the same object as the small bowl (same_object(image, metal_bowl_x, metal_bowl_y, object_x, object_y)). If it is, skip it

  62. [75]

    For each O you don’t skip, check if the color of O is the same as the color of the metal bowl

  63. [76]

    If it is, increment the counter

  64. [77]

    final_result

    When you are done looping, return the counter. Now here is an API of methods, you will want to solve the problem in a logical and sequential manner as I showed you ------------------ API ------------------ {predef_signatures} {api} ------------------ API ------------------ Ple...

  65. [78]

    2D distance/size refers to distance/size in pixel space

  66. [79]

    3D size is equal to 2D size times the depth of the object

    3D distance/size refers to distance/size in the real world. 3D size is equal to 2D size times the depth of the object

  67. [80]

    On" is defined as the closest object ABOVE another object. Only use this definition for

    "On" is defined as the closest object ABOVE another object. Only use this definition for "on"

  68. [81]

    "Next to" is defined as the closest object

  69. [82]

    Width is the same as length

  70. [83]

    Here are some helpful tips:

    "Depth" measures distance from the camera in 3D. Here are some helpful tips:

  71. [84]

    When you need to search over objects satisfying a condition, remember to check all the objects that satisfy the condition and don’t just return the first one

  72. [85]

    You already have an initialized variable named "image" - no need to initialize it yourself!

  73. [86]

    You can check if two objects are the same with the same_object method

    When searching for objects to compare to a reference object, make sure to remove the reference object from the retrieved objects. You can check if two objects are the same with the same_object method

  74. [87]

    Do not assume that the objects you see in these questions rae all of the objects you will see, keep the methods general

  75. [88]

    If two objects have the same 2D width, then the object with the largest depth has the largest 3D width

  76. [89]

    If two objects have the same 2D height, then the object with the largest depth has the largest 3D height

  77. [90]

    To convert to height and width in 3D space, it needs to be multiplied by the depth!

    2D sizes convey the height and width in IMAGE SPACE. To convert to height and width in 3D space, it needs to be multiplied by the depth!

  78. [91]

    final_result

    If you are given a reference size, scale your output predicted size accordingly! Again, answer the question by using the provided API to write a program in the tags <program></program> and ensure the program stores the answer in a variable called "final_result". It is critical...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.