Pith. sign in

REVIEW 4 major objections 6 minor 18 references

Warehouse Spatial Question Answering with LLM Agent

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An LLM agent with small perception tools beats MLLM finetuning on warehouse spatial QA, hitting 95.86 percent.

desk verdict This first-place challenge paper is a credible systems report whose headline number is externally verified, but the missing per-module ablations keep the result from being decomposable into perception versus reasoning quality. read the letter →

arxiv 2507.10778 v2 pith:DXKNQNXH submitted 2025-07-14 cs.CV cs.AI

Hsiang-Wei Huang , Jen-Hao Cheng , Kuang-Ming Chen , Cheng-Yen Yang , Bahaa Alattar , Yi-Ru Lin , Pyongkun Kim , Sangwon Kim
show 3 more authors
Kwangju Kim Chung-I Huang Jenq-Neng Hwang
This is my paper
classification cs.CVcs.AI
keywords LLMagentspatialreasoningwarehousequestionansweringfunctioncallingdistanceestimationinclusionclassificationAICityChallengemultimodallargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that complex spatial question answering in warehouses does not require large-scale finetuning of a multimodal large language model (an image-and-text model). It proposes SpatialAgent, a system in which a general-purpose LLM with function-calling abilities answers questions by invoking small, specialized perception tools: a distance regression model, an inclusion classifier, and simple geometric rules for left/right relations. On the 2025 AI City Challenge warehouse benchmark, the system reaches 95.86 percent accuracy on the hidden test set and ranks first, ahead of teams using the finetuning-heavy approach. The practical point is that spatial reasoning can be decomposed into cheap perception modules plus language-level reasoning, which avoids the massive training data and compute that prior spatial MLLMs needed.

What carries the argument

The load-bearing object is the SpatialAgent system itself: a function-calling LLM that maintains a structured message history and executes tool calls enclosed in execution tags before producing a final answer. Around it sit two lightweight perception modules, a cascaded ResNet-50 distance regressor trained with L2 loss (with a second model specializing in distances under three meters) and a ResNet-50 inclusion classifier trained with focal loss, plus rule-based geometric tools that use mask centroids for left/right and region queries. What carries the argument is the division of labor: the LLM handles question decomposition and spatial reasoning, while the small modules handle the measurements that reasoning depends on.

What would settle it

Take the public validation split, isolate questions whose correct answer depends on the distance regressor or the inclusion classifier, and measure those categories' accuracy separately from questions answerable by the left/right centroid rule; if the perception-dependent categories are near chance while the overall score stays high, the 95.86 percent result would not demonstrate general spatial reasoning.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that an LLM agent with tool interaction is enough to solve the warehouse spatial QA task at the top level. The system takes an RGB image, object masks, and a natural-language question; a rule-based parser rewrites masks into object identifiers; the LLM then plans multi-step reasoning and calls APIs such as distance, left/right, and inclusion checks. The distance models are ResNet-50 regressors, one general and one specializing in distances under three meters, and inclusion is a ResNet-50 binary classifier trained with focal loss. With this composition the agent obtains 95.8638 percent on the hidden test set, the top score in the challenge, which the paper presents as evidence that a data-efficient LLM agent can outperform or match large-scale MLLM finetuning on this benchmark.

Load-bearing premise

The load-bearing premise is that the two small distance and inclusion models, trained only on the benchmark's training split, stay accurate enough on the hidden test scenes that their errors do not dominate the 95.86 percent score.

Editorial extensions

If this is right

  • The same recipe can be applied to other spatial QA benchmarks where masks or detections are available; one keeps the LLM general and retrains only the small perception APIs.
  • Accuracy on distance and counting questions becomes a direct function of the small regressors and classifiers, so improving those modules is the most direct route to pushing beyond 95.86 percent.
  • Adapting to a new warehouse layout or camera setup becomes cheap, because only the lightweight perception models need retraining rather than the entire multimodal model.
  • The multi-turn execution leaves an audit trail: every final answer is backed by the sequence of tool calls and intermediate results, which can be inspected when an answer is wrong.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves implicit is to swap the reasoning LLM for a cheaper or open-weight model; if accuracy barely moves, the method's strength lies mostly in the perception tools, not the particular LLM.
  • The paper reports no standalone accuracy for the distance and inclusion models, so the natural next measurement is to evaluate those modules alone and separate perception error from reasoning error in the 95.86 percent.
  • Because the benchmark is synthetic and supplies clean masks, transferring this system to real warehouses would require adding an object detector to produce masks; that is a follow-up the paper does not claim to solve.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents SpatialAgent, an LLM-agent system for warehouse spatial question answering. The system takes an RGB image, binary object masks, and a spatial question; a rule-based parser converts mask tags into object identifiers; a Gemini 2.5 Flash agent then performs multi-turn reasoning and calls API tools for distance estimation, inclusion classification, relative left/right positioning, and region queries. Distance and inclusion are computed by trained ResNet-50 models (with a second small-distance cascade model), while simpler spatial relations use mask-centroid heuristics. The authors report 95.86% weighted accuracy on the hidden test set of the 2025 AI City Challenge Physical AI Spatial Intelligence Warehouse benchmark, ranking 1st on the official leaderboard, and they argue this demonstrates a data-efficient alternative to large-scale MLLM finetuning. The code is publicly released.

Significance. If the reported numbers are accurate, the result is significant: it achieves state-of-the-art accuracy on an externally scored challenge leaderboard using a modular agent with lightweight perception models rather than finetuning a large multimodal model. The official leaderboard provides a credible, independently verifiable evaluation, and the open-source code and clear system description support reproducibility. The paper is also honest in its empirical framing: the perception models are trained on the benchmark training split and evaluated on the hidden test set, with no indication of circularity or test-set leakage. However, the central interpretation—that the LLM agent's spatial reasoning, rather than the trained perception modules, drives the high accuracy—is not yet supported by the presented evidence, and the absence of per-question-type and per-module breakdowns limits how much can be concluded from the aggregate score.

major comments (4)
  1. [Section 4.4, Table 1] The paper reports only a single weighted-average accuracy over the four benchmark question types (spatial relations, multi-choice, distance estimation, counting), as described in Sections 4.1 and 4.2. Without a per-question-type breakdown, the 95.86% aggregate could be dominated by easier query types, so the claim that the agent demonstrates 'complex spatial reasoning' is not substantiated. I request a table reporting accuracy separately for each question type, and ideally a comparison with the runner-up on the same breakdown.
  2. [Sections 3.2 and 3.3] No isolated validation accuracy is reported for the two trained perception models: the distance regression networks F and F_small, and the inclusion classifier G. The end-to-end 95.86% accuracy is a composition of mask parsing, LLM function calling, and these perception predictions, so the aggregate number cannot reveal whether the agent's reasoning or the perception modules contribute most to performance. To support the paper's interpretive claims, please report the validation MAE/accuracy of F, F_small, and G, and include an ablation that replaces the perception APIs with oracle/ground-truth values to isolate the LLM agent's reasoning contribution.
  3. [Section 3.2] The cascade threshold is described as 'whenever F predicts a value smaller than 3m, we use F_small to predict again', with the motivation that F is inaccurate for distances under 3m. However, no validation analysis of this threshold is given, and it is a free parameter of the system. A threshold sweep or per-distance-bin accuracy comparison between F and F_small would be needed to support the claimed improvement from the cascade.
  4. [Section 4.3] The stochastic nature of the LLM agent is not addressed: the temperature is set to 0.2 and a retry mechanism adds 128 tokens, but the paper reports a single leaderboard result without variance. Since the same question can yield different outputs across runs, a repeated evaluation on the validation set with mean and standard deviation (or at least a statement about run-to-run stability) would strengthen the claim that 95.86% is a reproducible system-level accuracy.
minor comments (6)
  1. [Equation (1)] The typesetting of the L2 distance loss is corrupted in the manuscript; the expression should be written cleanly as ||D_hat - D_gt||_2^2.
  2. [Section 3.3] The focal loss hyperparameters alpha_t and gamma are named but their values are not reported; please provide the values used for training the inclusion classifier.
  3. [Table 1] The table caption should state the evaluation metric (weighted average success rate with Acc@10 for distance/counting questions) and the date on which the leaderboard was accessed, since leaderboards can change.
  4. [Abstract and Section 2.2] The phrase 'data-efficient' is used to contrast with MLLM finetuning, but no quantitative comparison of training data size, parameters, or compute is given; adding such numbers would make the claim more precise.
  5. [Section 3.1] The geometric heuristics for 'closest', 'most left', and 'middle' are described only in words; a precise definition (e.g., whether 'closest' uses predicted distances or image-plane centroid distances) would improve reproducibility.
  6. [References] Reference titles have inconsistent capitalization (e.g., [2] 'Spatialvlm', [13] 'Vlm-grounder'); please normalize them.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the system is an empirical pipeline trained on benchmark training data and evaluated on a held-out leaderboard.

full rationale

The paper's central claim is an empirical leaderboard result, not a quantity derived from its own inputs by construction. The distance estimation model (Sec. 3.2) is trained with an L2 regression loss against ground-truth distances from the benchmark's training split, and the inclusion classification model (Sec. 3.3) is trained with focal loss against counting-question free-form answers from the training split. Both are then applied to the hidden test set through the LLM agent's function calls, and the reported 95.86% accuracy is the official external evaluation. This is standard supervised learning and held-out evaluation; there is no equation that makes the predicted output equal to the fitted target by construction, and the test answers are not used in training. The left/right relationship is computed directly from mask centroids, an independent geometric rule. The only self-citations are two prior works by the same group cited in the introduction for general context; they are not load-bearing for the claimed result and no uniqueness theorem or unsupported premise is imported from them. Concerns raised by a skeptical reader about missing per-module validation accuracy or ablations are evidence-completeness issues, not circularity. The derivation chain is therefore self-contained with respect to the benchmark's training/test separation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new theoretical entities or physical postulates. Its load-bearing components are learned model weights, a hand-set cascade threshold, and LLM hyperparameters. These are legitimate engineering choices, but they are fitted to the benchmark's training distribution and their isolated accuracy is not reported, which limits how much of the final score can be attributed to the LLM agent rather than the perception modules.

free parameters (5)
  • ResNet-50 distance estimation model weights (F and Fsmall) = Trained on 245K training samples from distance questions
    The distance API is a learned regressor; the paper reports no validation accuracy, so its contribution to the 95.86% system accuracy is unknown (Section 3.2, 4.3).
  • ResNet-50 inclusion classification model weights (G) = Trained on 158K training samples from counting questions
    The inclusion API is a learned binary classifier; again, no validation accuracy is reported (Section 3.3, 4.3).
  • Distance cascade threshold = 3 meters
    Hand-chosen in Section 3.2; when the general model F predicts a distance under 3m, the small-distance model Fsmall is used instead, based on an experiment finding F inaccurate at small distances.
  • LLM temperature = 0.2
    Chosen in Section 4.3 for Gemini 2.5-Flash; affects sampling randomness and therefore reproducibility of results.
  • LLM thinking budget retry increment = 128 tokens
    Added when function execution fails, as described in Section 4.3; an ad hoc hyperparameter that could affect success rate on edge cases.
assumptions (5)
  • domain assumption The 2025 AI City Challenge test-set labels are independent of the training split and the official leaderboard protocol (Acc@10 and exact match) is correctly applied.
    The central accuracy claim relies entirely on the external benchmark's evaluation (Sections 4.1, 4.2, 4.4); the manuscript itself does not verify this.
  • domain assumption Gemini 2.5-Flash behaves as a reliable function-calling and reasoning agent at temperature 0.2, with no benchmark contamination or API version drift.
    The system depends on a closed commercial API (Section 4.3); the paper provides no control for contamination or reproducibility across API versions.
  • domain assumption The training data for the distance and inclusion models come from the benchmark's training split and do not include test or validation annotations.
    Stated in Section 4.3; if the split were misused, the 95.86% result would be invalid.
  • domain assumption The rule-based mask-to-object parser correctly maps <mask> tokens to object IDs for the vast majority of questions, with LLM rephrasing covering corner cases.
    Described in Section 4.3 Question Pre-processing; parsing errors would propagate into every downstream API call and final answer.
  • standard math L2 loss and focal loss training on ResNet-50 backbones produce adequately calibrated regressors and classifiers for the perception APIs.
    The system assumes standard supervised learning behaves as expected (Sections 3.2, 3.3); no verification of model calibration is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Warehouse Spatial Question Answering with LLM Agent." pith.science (2026). https://pith.science/paper/DXKNQNXH

@misc{pith2026250710778,
  author       = {Pith},
  title        = {Pith review of: Warehouse Spatial Question Answering with LLM Agent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DXKNQNXH}},
  note         = {Machine review of arXiv:2507.10778}
}
read the original abstract

Spatial understanding has been a challenging task for existing Multi-modal Large Language Models~(MLLMs). Previous methods leverage large-scale MLLM finetuning to enhance MLLM's spatial understanding ability. In this paper, we present a data-efficient approach. We propose a LLM agent system with strong and advanced spatial reasoning ability, which can be used to solve the challenging spatial question answering task in complex indoor warehouse scenarios. Our system integrates multiple tools that allow the LLM agent to conduct spatial reasoning and API tools interaction to answer the given complicated spatial question. Extensive evaluations on the 2025 AI City Challenge Physical AI Spatial Intelligence Warehouse dataset demonstrate that our system achieves high accuracy and efficiency in tasks such as object retrieval, counting, and distance estimation. The code is available at: https://github.com/hsiangwei0903/SpatialAgent

Figures

Figures reproduced from arXiv: 2507.10778 by the authors.

Figure 1
Figure 1. An illustration of (a) MLLM-finetuned paradigm like [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An illustration of our SpatialAgent framework. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 12 canonical work pages

  1. [1]

    Spatialbot: Precise spatial understanding with vision language models

    Wenxiao Cai, Iaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. arXiv preprint arXiv:2406.13642, 2024. 2

    arXiv 2024

  2. [2]

    Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 14455–14465,

  3. [3]

    Spatial- rgpt: Grounded spatial reasoning in vision-language mod- els

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Rui- han Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatial- rgpt: Grounded spatial reasoning in vision-language mod- els. Advances in Neural Information Processing Systems , 37:135062–135093, 2025. 1, 2

    work page 2025

  4. [4]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blis- tein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. 2, 4

    arXiv 2025

  5. [5]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 3

    2016

  6. [6]

    ToSA: Token Merging with Spatial Awareness

    Hsiang-Wei Huang, Wenhao Chai, Kuang-Ming Chen, Cheng-Yen Yang, and Jenq-Neng Hwang. Tosa: To- ken merging with spatial awareness. arXiv preprint arXiv:2506.20066, 2025. 1

    work page Pith review arXiv 2025

  7. [7]

    Zero-shot 3d question answering via voxel-based dynamic token compres- sion

    Hsiang-Wei Huang, Fu-Chen Chen, Wenhao Chai, Che- Chun Su, Lu Xia, Sanghun Jung, Cheng-Yen Yang, Jenq- Neng Hwang, Min Sun, and Cheng-Hao Kuo. Zero-shot 3d question answering via voxel-based dynamic token compres- sion. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19424–19434, 2025. 1

    work page 2025

  8. [8]

    Embodied agent inter- face: Benchmarking llms for embodied decision making

    Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Er- ran Li Li, Ruohan Zhang, et al. Embodied agent inter- face: Benchmarking llms for embodied decision making. Advances in Neural Information Processing Systems , 37: 100428–100534, 2024. 2

    work page 2024

Show all 18 references
  1. [9]

    Seeground: See and ground for zero-shot open- vocabulary 3d visual grounding

    Rong Li, Shijie Li, Lingdong Kong, Xulei Yang, and Jun- wei Liang. Seeground: See and ground for zero-shot open- vocabulary 3d visual grounding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3707–3717, 2025. 2

    2025

  2. [10]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ar. Focal loss for dense object detection. In Pro- ceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. 3

    2017

  3. [11]

    Zheng Tang, Shuo Wang, David C. Anastasiu, Ming- Ching Chang, Anuj Sharma, Quan Kong, Norimasa Ko- bori, Munkhjargal Gochoo, Ganzorig Batnasan, Munkh- Erdene Otgonbold, Fady Alnajjar, Jun-Wei Hsieh, Tomasz Kornuta, Xiaolong Li, Yilin Zhao, Han Zhang, Subhashree Radhakrishnan, ...

    2025

  4. [12]

    Videoagent: Long-form video understanding with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung- Levy. Videoagent: Long-form video understanding with large language model as agent. In European Conference on Computer Vision, pages 58–76. Springer, 2024. 2

    2024

  5. [13]

    Vlm-grounder: A vlm agent for zero-shot 3d visual grounding

    Runsen Xu, Zhiwei Huang, Tai Wang, Yilun Chen, Jiang- miao Pang, and Dahua Lin. Vlm-grounder: A vlm agent for zero-shot 3d visual grounding. In CoRL, 2024. 2

    2024

  6. [14]

    Fouhey, and Joyce Chai

    Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F. Fouhey, and Joyce Chai. Llm- grounder: Open-vocabulary 3d visual grounding with large language model as an agent. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages...

    2024

  7. [15]

    Visual programming for zero-shot open-vocabulary 3d visual grounding

    Zhihao Yuan, Jinke Ren, Chun-Mei Feng, Hengshuang Zhao, Shuguang Cui, and Zhen Li. Visual programming for zero-shot open-vocabulary 3d visual grounding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20623–20633, 2024. 1, 2

    2024

  8. [16]

    Agent3d-zero: An agent for zero-shot 3d understanding

    Sha Zhang, Di Huang, Jiajun Deng, Shixiang Tang, Wanli Ouyang, Tong He, and Yanyong Zhang. Agent3d-zero: An agent for zero-shot 3d understanding. In European Confer- ence on Computer Vision, pages 186–202. Springer, 2024. 1, 2

    2024

  9. [17]

    See and think: Embodied agent in virtual environment

    Zhonghan Zhao, Wenhao Chai, Xuan Wang, Boyi Li, Shengyu Hao, Shidong Cao, Tian Ye, and Gaoang Wang. See and think: Embodied agent in virtual environment. In European Conference on Computer Vision, pages 187–204. Springer, 2024. 2

    2024

  10. [18]

    Video-3d llm: Learning position-aware video representation for 3d scene understanding

    Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d llm: Learning position-aware video representation for 3d scene understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 8995–9006, 2025. 1

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.