Pith. sign in

REVIEW 3 major objections 5 minor 57 references

SpatialPrompting: Keyframe-driven Zero-Shot Spatial Reasoning with Off-the-Shelf Multimodal Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SpatialPrompting shows that keyframes plus camera poses can replace 3D fine-tuning for spatial question answering.

desk verdict The paper's zero-shot SOTA claim is driven by a training-answer prior, not by spatial prompting, though the keyframe/camera-pose method itself has merit. read the letter →

arxiv 2505.04911 v1 pith:RHJXDHON submitted 2025-05-08 cs.CV cs.AIcs.CL

Shun Taguchi , Hideki Deguchi , Takumi Hamazaki , Hiroyuki Sakai This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords spatialquestionansweringzero-shotlearningmultimodallargelanguagemodelskeyframeselectioncameraposeprompting3DvisualScanQASQA3D
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SpatialPrompting claims that off-the-shelf multimodal large language models can answer questions about 3D scenes without any 3D-specific training, provided they receive a small set of well-chosen keyframes plus the camera pose of each frame. On the ScanQA benchmark this prompt-only approach reaches the state of the art on exact match, ROUGE-L, and SPICE, and it stays competitive on the situated-question benchmark SQA3D, against models that were fine-tuned on point clouds or voxel features. The proposal matters because it replaces an expensive training pipeline with a prompt-engineering recipe built from standard components: vision-language features, a spatial-distance metric, and a pose-annotated image template. The paper also shows that most of the gain depends on a few-shot annotation listing the most frequent training answers, which is the part a stricter reading of 'zero-shot' would question.

What carries the argument

The machinery is the keyframe-selection algorithm together with the pose-aware prompt template. For every pair of frames, the combined distance $d'(i,j)=d(i,j)+\alpha(1-S(i,j))$ mixes a Mahalanobis distance $d$ between the mean and covariance of the frames' back-projected point clouds with a CLIP cosine-similarity term $S$; the algorithm repeatedly removes the frame with the lower quality score $q_t=\det|\Sigma_t|+\beta\,\mathrm{Var}(\nabla^2 I_t)$, which prefers wider point-cloud spread and sharper images. The prompt then attaches to each surviving image a camera pose given as position $[x,y,z]$ in meters and rotation as Euler angles, a representation the paper found easier for LLMs to interpret than quaternions or matrices. This combination is what lets the LLM reconstruct spatial layout without ever seeing a point cloud.

What would settle it

Take the ScanQA validation set and replace the annotation's top-20 training answers with a permuted or unrelated list (for example, the same words in random order); if EM@1 falls toward the 19.83 no-annotation ablation, the reported gain is a training-answer prior rather than a spatial-reasoning effect, whereas if it stays near 27.34 the model is using the list only as an output-format cue.

Watch

Extended reading notes

Core claim

The central claim is that spatial reasoning in 3D environments is largely an emergent capability of modern multimodal LLMs, and that the role of the vision stack is to expose the right views with the right positional context. SpatialPrompting selects keyframes by combining CLIP-based semantic similarity with the Mahalanobis distance between per-frame point-cloud statistics, then scores candidates by a quality term favoring wide field of view and sharpness. The surviving keyframes are placed in a prompt template that states each camera's position and Euler-angle rotation, followed by a task annotation and the user question. With GPT-4o this pipeline obtains 27.34 EM@1, 43.39 ROUGE-L, and 20.49 SPICE on ScanQA validation, ahead of all compared 3D-tuned systems on those metrics, and an average of 52.74 on SQA3D, competitive with but slightly below the specialized state of the art. The authors conclude that expensive 3D-specific fine-tuning is not necessary for strong spatial QA, and that the remaining weaknesses are directional questions that depend on the user's orientation and precise metric estimation.

Load-bearing premise

The method counts as zero-shot only if feeding the model the most common answers from each benchmark's training set is a permissible prompt aid rather than a benchmark-specific prior.

Editorial extensions

If this is right

  • 3D visual question answering no longer requires a dedicated 3D encoder or fine-tuning; any multimodal LLM with strong spatial priors can be pointed at a scene through this prompt recipe.
  • The method's ScanQA exact-match score (27.34) and ROUGE-L (43.39) set a new prompt-only reference, so future work can benchmark vision-language prompting against 3D-tuned models without training.
  • The ablation shows that camera pose contributes about 1.6 EM@1 points on ScanQA and keyframe selection adds roughly one more point over uniform sampling, so both components are load-bearing for the reported gains.
  • On SQA3D, removing the camera pose slightly improves the average score, indicating that in situated questions the camera coordinate frame can conflict with the user's orientation; the paper treats this as a limitation, not a universal advantage.
  • The approach inherits the counting and orientation weaknesses of the underlying LLM, as documented in the paper's failure analysis, so applications needing exact counts or user-centric directions will require supplemental modules.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Under a stricter definition of zero-shot that forbids using benchmark training statistics, the method is better described as few-shot with a benchmark-specific answer prior: the annotation lists the top 20 training answers per question type, and removing it drops EM@1 from 27.34 to 19.83.
  • The same keyframe-and-pose recipe could be adapted to embodied tasks such as navigation or manipulation, where poses come from SLAM in real time; the SQA3D failure analysis suggests that transforming poses into the user's coordinate frame, not the camera's, would be a necessary adaptation for situated questions.
  • A testable extension is to replace the static top-20 answer list with a per-scene prior generated from object labels detected in the keyframes, which could preserve the output-formatting benefit without leaking benchmark answer frequencies.
  • Since the method delegates reasoning to the off-the-shelf LLM, its ceiling is set by that model's spatial priors; running the same prompt on smaller open-weight models would reveal how much of the result is prompting versus model scale.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SpatialPrompting, a training-free framework for 3D question answering. It selects a diverse set of keyframes from RGB-D video using CLIP-based vision-language similarity, Mahalanobis distance on point-cloud distributions, field-of-view spread, and image sharpness, then feeds these keyframes together with camera poses and a structured prompt to off-the-shelf multimodal LLMs (GPT-4o and Gemini-2.0). The prompt includes a preamble, keyframe/camera-pose blocks, an 'annotation' component, and the user query. The authors report state-of-the-art zero-shot performance on ScanQA (EM@1 27.34, ROUGE-L 43.39, SPICE 20.49) and competitive results on SQA3D without any 3D-specific fine-tuning. The contribution is framed as replacing expensive 3D-specific training with a scalable, prompt-based alternative.

Significance. If the central claim held, the paper would make a useful practical contribution: demonstrating that off-the-shelf MLLMs can perform 3D spatial QA from keyframes and camera poses would open a low-cost alternative to specialized 3D-VQA models. The keyframe selection algorithm is clearly described, the ablations are informative, and the qualitative failure analysis (e.g., miscounting, direction confusion) is honest and useful. However, the headline 'zero-shot SOTA' claim is compromised by the fact that the prompt's annotation component is built from the top-20 most frequent answers in each benchmark's training split, and the ablation in Table 2 shows that this component is responsible for most of the reported gain. The proposed keyframe/camera-pose mechanism alone, without the answer prior, does not outperform the raw GPT-4o baseline on ScanQA. The paper would need to either reframe its claims as few-shot assisted performance or demonstrate an independent benefit of the prompt mechanism without benchmark-derived answer lists.

major comments (3)
  1. [Sec. 4.3, 'Prompt generation', and Table 2] The paper's 'zero-shot' claim is contradicted by the few-shot annotation. Section 4.3 explicitly states that the annotation for benchmark datasets is 'the top 20 common answers for each question type' obtained from the training dataset. Removing this annotation drops ScanQA EM@1 from 27.34 to 19.83 and ROUGE-L from 43.39 to 31.89, which is below the GPT-4o baseline (21.43 and 32.18). Thus the reported SOTA is largely driven by a benchmark-specific answer prior, not by the proposed keyframe and camera-pose prompting. The title, abstract, and conclusion call the method 'zero-shot,' while the text self-describes the annotation as 'few-shot prompting.' This is a load-bearing inconsistency. Please either (a) revise the claim to 'few-shot assisted' and provide an evaluation where the few-shot demonstrations are not drawn from the benchmark's training answers, or (b) remove the answer-list component and report the resulting performance, tempering the SOTA statement accordingly.
  2. [Table 1 and Sec. 4.3] The comparison with previous methods is not apples-to-apples. Table 1 compares SpatialPrompting exclusively against methods that are fine-tuned on 3D data; there is no zero-shot baseline in the table (the GPT-4o baseline appears only in the ablation Table 2). Moreover, the reported margins over the strongest fine-tuned baselines are extremely small (EM@1 27.34 vs 27.2; SPICE 20.49 vs 20.44). Because the evaluation uses a stochastic API model (GPT-4o), the paper should report repeated runs, confidence intervals, or a statistical significance test before claiming 'state-of-the-art zero-shot performance.'
  3. [Table 2 and Sec. 4.3.2] The ablation shows that the core mechanism is not independently effective. The 'w/o Annotation' row gives ScanQA EM@1 of 19.83 and ROUGE-L of 31.89, both below the GPT-4o baseline of 21.43 and 32.18. This means that keyframe extraction plus camera poses, without the benchmark answer prior, provides no benefit over raw GPT-4o and may even hurt. The paper should explicitly discuss this fact and report what fraction of the total gain is attributable to the prompt structure versus the answer prior. Without such a decomposition, the claim that 'keyframe-driven prompt generation' enables zero-shot spatial reasoning is not supported.
minor comments (5)
  1. [Sec. 4.2] Section 4.1 appears as a heading with no content; the datasets and metrics are only described in Sec. 4.2. Please renumber the sections so the 'Quantitative Results' heading is not empty.
  2. [Table 5] In the row for '5' images, the value '39,07' uses a comma as a decimal separator; it should be '39.07' for consistency with the rest of the table.
  3. [Supp. C] The sentence 'The annotation is consists of the top 20 frequent answers' should be 'The annotation consists of the top 20 frequent answers.'
  4. [Sec. 3.3] The camera-pose template has an inconsistent space in 'Camera rotation : [x= {roll}°, y= {pitch}°, z={yaw}°]'. Please make the formatting uniform and check that the rendered prompt matches the intended syntax.
  5. [Sec. 4.3] The term 'annotation' is used for two different things: the instruction not to refer to images, and the benchmark-specific few-shot answer list. Please disambiguate these two uses to avoid confusion.

Circularity Check

1 steps flagged · score 6.0 of 10

Benchmark answer prior, not keyframe prompting, carries the reported zero-shot SOTA.

  1. fitted input called prediction [Sec. 4.3 (Implementation Details); Table 2; Supp. C.1]
    "common answer corresponds to the most frequent answers observed in the training dataset for each question category. We supply the top 20 common answers for each question type as the few-shot prompt."

    The paper's headline 'state-of-the-art zero-shot performance' is obtained by injecting a benchmark-specific answer prior: the top 20 most frequent ground-truth answers from the training split, organized by question type. This is a fitted input drawn from the same benchmarks on which the method is evaluated. The method's own ablation shows the prior is the dominant term: removing it drops ScanQA EM@1 from 27.34 to 19.83, below the GPT-4o baseline (21.43), and SQA3D average from 52.74 to 47.77, below the baseline (48.51). Thus the reported prediction gain is substantially forced by the training-derived answer list, not by the keyframe-driven spatial prompting.

full rationale

The central circularity is the 'zero-shot' benchmark claim. The paper fits a per-benchmark answer prior (top-20 training answers by question type) into the prompt, labels it 'annotation', and reports SOTA numbers on the same benchmarks. The method's own ablation shows this prior is the dominant term: removing it drops ScanQA EM@1 from 27.34 to 19.83 (below the 21.43 GPT-4o baseline) and SQA3D average from 52.74 to 47.77 (below the 48.51 baseline). Thus, under a strict zero-shot definition, the proposed keyframe and camera-pose prompting alone provides no gain over the raw model; the headline result is substantially an artifact of a training-set answer distribution. The keyframe extraction and camera-pose modules are not themselves circular, and the self-citation to Deguchi et al. [13] is only a limitation remark, not load-bearing. Because the central 'SOTA zero-shot' claim is carried by a fitted answer prior, the paper is partially circular and the honest zero-shot variant is the poor 'w/o Annotation' row.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on tuned scalars (alpha, beta), a chosen keyframe count, and the training-set answer priors. The MLLM's spatial reasoning ability and the validity of the pose encoding are domain assumptions, not proven facts.

free parameters (4)
  • alpha = 5.0
    Balances Mahalanobis distance and vision-language similarity in the keyframe distance metric; selected via iterative experimentation on the benchmarks (Sec. B.3).
  • beta = 1.0
    Weights the Laplacian variance (sharpness) in the quality score; tuned on the benchmarks (Sec. B.3).
  • Nmax = 30
    Maximum number of keyframes; chosen based on sensitivity analysis (Table 5) as the best balance between coverage and performance.
  • top-20 common answers per question type = Dataset-dependent
    Few-shot prompt examples drawn from the training set of each benchmark; this injects the answer distribution into the prompt and is fitted to the evaluation data.
assumptions (4)
  • domain assumption Off-the-shelf MLLMs such as GPT-4o and Gemini-2.0 possess enough zero-shot spatial reasoning ability to answer 3D-VQA questions from images and camera poses.
    The entire method relies on this emergent capability; invoked in the Introduction and Section 3.
  • domain assumption Camera positions and Euler angles are a sufficient spatial summary for the MLLM to reason about 3D relationships.
    Section 3.3 states that Euler angles are easier for LLMs to interpret than quaternions or matrices, but does not prove that this representation preserves all task-relevant spatial information.
  • domain assumption Mahalanobis distance between point cloud means with pooled covariance is a valid proxy for field-of-view overlap.
    Section 3.2 introduces this as an indicator of FOV overlap without validation against actual FOV overlap.
  • domain assumption The depth maps and camera poses (from SLAM or dataset) are accurate enough for the point cloud statistics to be meaningful.
    Section 3.2 assumes depth maps and poses are given; errors in these inputs would propagate into keyframe selection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SpatialPrompting: Keyframe-driven Zero-Shot Spatial Reasoning with Off-the-Shelf Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/RHJXDHON

@misc{pith2026250504911,
  author       = {Pith},
  title        = {Pith review of: SpatialPrompting: Keyframe-driven Zero-Shot Spatial Reasoning with Off-the-Shelf Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RHJXDHON}},
  note         = {Machine review of arXiv:2505.04911}
}
read the original abstract

This study introduces SpatialPrompting, a novel framework that harnesses the emergent reasoning capabilities of off-the-shelf multimodal large language models to achieve zero-shot spatial reasoning in three-dimensional (3D) environments. Unlike existing methods that rely on expensive 3D-specific fine-tuning with specialized 3D inputs such as point clouds or voxel-based features, SpatialPrompting employs a keyframe-driven prompt generation strategy. This framework uses metrics such as vision-language similarity, Mahalanobis distance, field of view, and image sharpness to select a diverse and informative set of keyframes from image sequences and then integrates them with corresponding camera pose data to effectively abstract spatial relationships and infer complex 3D structures. The proposed framework not only establishes a new paradigm for flexible spatial reasoning that utilizes intuitive visual and positional cues but also achieves state-of-the-art zero-shot performance on benchmark datasets, such as ScanQA and SQA3D, across several metrics. The proposed method effectively eliminates the need for specialized 3D inputs and fine-tuning, offering a simpler and more scalable alternative to conventional approaches.

Figures

Figures reproduced from arXiv: 2505.04911 by the authors.

Figure 1
Figure 1. SpatialPrompting is a framework that employs keyframe-driven prompt generation for zero-shot spatial reasoning with multimodal LLMs. The proposed approach enables accurate spatial reasoning without additional 3D-specific training. Several attempts have already been made to achieve SpatialQA. Previous methods [2, 27, 33] focused on task￾specific models that utilized 3D features such as point cloud representations. Re… view at source ↗
Figure 2
Figure 2. Overview of SpatialPrompting. In keyframe extraction, both spatial and semantic features are used to select keyframes. In prompt generation, these keyframes and camera poses are combined with a preamble, annotation, and user query to form prompts for multimodal LLMs, enabling SpatialQA. Algorithm 1 Keyframe extraction algorithm 1: Input: Set of frames {I1, . . . , IN }, maximum number of keyframes Nmax 2: Output: Se… view at source ↗
Figure 3
Figure 3. Qualitative results. We showcase five categories of spatial reasoning tasks tackled by the proposed SpatialPrompting framework: (1) Small object localization, (2) Complex spatial layout inference, (3) Multi-object relationship reasoning, (4) Relative spatial orientation, and (5) Real-world knowledge integration. Each question–answer pair is generated using minimal keyframes and corresponding camera poses, illustrati… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Effectiveness of Keyframe extraction on Spatial￾Prompting. The proposed keyframe extraction process effectively eliminates redundant views, and the results indicate that images with a wider FOV are preferentially selected. trary, the naive GPT-4o without SpatialPrompti…
Figure 5
Figure 5. Figure 5: Effectiveness of SpatialPrompting in limited images. The use of high-coverage keyframes reduces missing information, whereas camera poses allow the inference of spatial relationships from limited images, enabling the proposed method to achieve complex SpatialQA. as GPT…
Figure 6
Figure 6. Figure 6: Additional Qualitative Result on ScanNet [11] (1). We use GPT-4o [32] as the multimodal LLM and inputs five images for SpatialQA. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Additional Qualitative Result on ScanNet [11] (2). We use GPT-4o [32] as the multimodal LLM and inputs five images for SpatialQA. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Sensitivity Analysis of Parameters α and β 14 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Examples of the Miscount of Objects on SQA3D [27]. S represents the situation, Q represents the question, SpatialPrompting denotes the predicted answer generated by our SpatialPrompting method using GPT-4o [32], and GT denotes the ground-truth answer. Misdirection It h…
Figure 10
Figure 10. Figure 10: Examples of the Misdirection on SQA3D [27]. S represents the situation, Q represents the question, SpatialPrompting denotes the predicted answer generated by our SpatialPrompting method using GPT-4o [32], and GT denotes the ground-truth answer. 15 [PITH_FULL_IMAGE:fi…
Figure 11
Figure 11. Figure 11: Examples of Acceptable Variants on SQA3D [27]. S represents the situation, Q represents the question, SpatialPrompting denotes the predicted answer generated by our SpatialPrompting method using GPT-4o [32], and GT denotes the ground-truth answer. 16 [PITH_FULL_IMAGE…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 35 canonical work pages

  1. [1]

    Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In ECCV, pages 422–440. Springer, 2020. 2

    work page 2020

  2. [2]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In CVPR, pages 19129–19139, 2022. 1, 2, 5, 6, 8, 13, 17

    work page 2022

  3. [3]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 33:1877– 1901, 2020. 2

    1901

  4. [4]

    Scanrefer: 3d object localization in rgb-d scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In ECCV, pages 202–221. Springer, 2020. 2

    work page 2020

  5. [5]

    Language conditioned spatial relation reasoning for 3d object grounding

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. Advances in neural information processing systems, 35:20522–20535,

  6. [6]

    End-to-end 3d dense captioning with vote2cap-detr

    Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Gang Yu, and Tao Chen. End-to-end 3d dense captioning with vote2cap-detr. In CVPR, pages 11124–11133, 2023. 2

    work page 2023

  7. [7]

    Ll3da: Visual interactive instruction tuning for omni-3d understand- ing reasoning and planning

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. Ll3da: Visual interactive instruction tuning for omni-3d understand- ing reasoning and planning. In CVPR, pages 26428–26438,

    work page

  8. [8]

    V ote2cap-detr++: Decoupling localization and describing for end-to-end 3d dense captioning

    Sijin Chen, Hongyuan Zhu, Mingsheng Li, Xin Chen, Peng Guo, Yinjie Lei, YU Gang, Taihao Li, and Tao Chen. V ote2cap-detr++: Decoupling localization and describing for end-to-end 3d dense captioning. IEEE TPAMI, 2024. 2

    2024

Show all 57 references
  1. [9]

    Scan2cap: Context-aware dense captioning in rgb-d scans

    Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2cap: Context-aware dense captioning in rgb-d scans. In CVPR, pages 3193–3203, 2021. 2

    2021

  2. [10]

    Zero-shot video question answering with pro- cedural programs

    Rohan Choudhury, Koichiro Niinuma, Kris M Kitani, and L´aszl´o A Jeni. Zero-shot video question answering with pro- cedural programs. arXiv preprint arXiv:2312.00937, 2023. 2

    2023 arXiv

  3. [11]

    Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner

    Angela Dai, Angel X. Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017. 5, 11, 12

    2017

  4. [12]

    Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration

    Angela Dai, Matthias Nießner, Michael Zollh ¨ofer, Shahram Izadi, and Christian Theobalt. Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics (ToG), 36(4): 1, 2017. 3

    2017

  5. [13]

    Lan- guage to map: Topological map generation from natural lan- guage path instructions

    Hideki Deguchi, Kazuki Shibata, and Shun Taguchi. Lan- guage to map: Topological map generation from natural lan- guage path instructions. In 2024 IEEE International Confer- ence on Robotics and Automation (ICRA), pages 9556–9562. IEEE, 2024. 8

    2024

  6. [14]

    Scene-llm: Extending language model for 3d visual understanding and reasoning

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wen- han Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401, 2024. 1, 2, 6

    2024 arXiv

  7. [15]

    Gemini 2.0

    Google. Gemini 2.0. https://gemini.google/ ,

  8. [16]

    3d-llm: Inject- ing the 3d world into large language models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Inject- ing the 3d world into large language models. NeurIPS, 36: 20482–20494, 2023. 2, 6

    2023

  9. [17]

    Vtimellm: Empower llm to grasp video moments

    Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower llm to grasp video moments. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14271–14280, 2024. 2

    2024

  10. [18]

    Chat-scene: Bridging 3d scene and large language models with object identifiers

    Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. In NeurIPS,

  11. [19]

    An embodied generalist agent in 3d world

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In ICML, 2024. 6

    2024

  12. [20]

    Multi- view transformer for 3d visual grounding

    Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi- view transformer for 3d visual grounding. In CVPR, pages 15524–15533, 2022. 2

    2022

  13. [21]

    More: Multi-order relation mining for dense captioning in 3d scenes

    Yang Jiao, Shaoxiang Chen, Zequn Jie, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. More: Multi-order relation mining for dense captioning in 3d scenes. In ECCV, pages 528–545. Springer, 2022. 2

    2022

  14. [22]

    Chat-univi: Unified visual representation em- powers large language models with image and video under- standing

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video under- standing. In CVPR, pages 13700–13710, 2024. 2

    2024

  15. [23]

    Context-aware alignment and mutual masking for 3d- language pre-training

    Zhao Jin, Munawar Hayat, Yuwei Yang, Yulan Guo, and Yin- jie Lei. Context-aware alignment and mutual masking for 3d- language pre-training. In CVPR, pages 10984–10994, 2023. 2, 6

    2023

  16. [24]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. NeurIPS, 35:22199–22213, 2022. 2

    2022

  17. [25]

    Videoin- sta: Zero-shot long video understanding via informa- tive spatial-temporal reasoning with llms

    Ruotong Liao, Max Erler, Huiyu Wang, Guangyao Zhai, Gengyuan Zhang, Yunpu Ma, and V olker Tresp. Videoin- sta: Zero-shot long video understanding via informa- tive spatial-temporal reasoning with llms. arXiv preprint arXiv:2409.20365, 2024. 2

    2024 arXiv

  18. [26]

    Video-llava: Learning united visual rep- resentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual rep- resentation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 2

    2023 arXiv

  19. [27]

    Sqa3d: Situated question answering in 3d scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yi- tao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022. 1, 2, 5, 6, 8, 13, 15, 16, 17, 18

    2022 arXiv

  20. [28]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 2 9

    2023 arXiv

  21. [29]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. NeurIPS, 36:46212– 46244, 2023. 2

    2023

  22. [30]

    Large language models know your contextual search intent: A prompting framework for conversational search

    Kelong Mao, Zhicheng Dou, Fengran Mo, Jiewen Hou, Hao- nan Chen, and Hongjin Qian. Large language models know your contextual search intent: A prompting framework for conversational search. arXiv preprint arXiv:2303.06573 ,

    arXiv

  23. [31]

    Morevqa: Exploring modular reason- ing models for video question answering

    Juhong Min, Shyamal Buch, Arsha Nagrani, Minsu Cho, and Cordelia Schmid. Morevqa: Exploring modular reason- ing models for video question answering. In CVPR, pages 13235–13245, 2024. 2

    2024

  24. [32]

    OpenAI. Gpt-4o. https://openai.com/index/ hello- gpt- 4o/, 2024. Accessed: 2025-02-25. 2, 5, 6, 11, 12, 13, 15, 16

    2024

  25. [33]

    Clip-guided vision-language pre-training for question answering in 3d scenes

    Maria Parelli, Alexandros Delitzas, Nikolas Hars, Geor- gios Vlassis, Sotirios Anagnostidis, Gregor Bachmann, and Thomas Hofmann. Clip-guided vision-language pre-training for question answering in 3d scenes. In CVPR, pages 5607– 5612, 2023. 1, 2

    2023

  26. [34]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763. PMLR, 2021. 3, 5

    2021

  27. [35]

    Traveler: A multi-lmm agent frame- work for video question-answering

    Chuyi Shang, Amos You, Sanjay Subramanian, Trevor Dar- rell, and Roei Herzig. Traveler: A multi-lmm agent frame- work for video question-answering. arXiv e-prints , pages arXiv–2404, 2024. 2

    2024

  28. [36]

    Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras

    Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. NeurIPS, 34: 16558–16569, 2021. 3

    2021

  29. [37]

    Four ways to improve verbo-visual fusion for dense 3d visual grounding

    Ozan Unal, Christos Sakaridis, Suman Saha, and Luc Van Gool. Four ways to improve verbo-visual fusion for dense 3d visual grounding. In ECCV, pages 196–213. Springer, 2024. 2

    2024

  30. [38]

    Vamos: Versatile action models for video understanding

    Shijie Wang, Qi Zhao, Minh Quan Do, Nakul Agarwal, Kwonjoon Lee, and Chen Sun. Vamos: Versatile action models for video understanding. In ECCV, pages 142–160. Springer, 2024. 2

    2024

  31. [39]

    Videoagent: Long-form video understanding with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung- Levy. Videoagent: Long-form video understanding with large language model as agent. In ECCV, pages 58–76. Springer, 2024. 2

    2024

  32. [40]

    Language models with im- age descriptors are strong few-shot video-language learners

    Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chen- guang Zhu, Derek Hoiem, et al. Language models with im- age descriptors are strong few-shot video-language learners. NeurIPS, 35:8483–8497, 2022. 2

    2022

  33. [41]

    3drp-net: 3d relative position-aware network for 3d visual grounding

    Zehan Wang, Haifeng Huang, Yang Zhao, Linjun Li, Xize Cheng, Yichen Zhu, Aoxiong Yin, and Zhou Zhao. 3drp-net: 3d relative position-aware network for 3d visual grounding. arXiv preprint arXiv:2307.13363, 2023. 2

    2023 arXiv

  34. [42]

    Distill- ing coarse-to-fine semantic matching knowledge for weakly supervised 3d visual grounding

    Zehan Wang, Haifeng Huang, Yang Zhao, Linjun Li, Xize Cheng, Yichen Zhu, Aoxiong Yin, and Zhou Zhao. Distill- ing coarse-to-fine semantic matching knowledge for weakly supervised 3d visual grounding. In ICCV, pages 2662–2671,

  35. [43]

    Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes

    Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308.08769, 2023. 2

    2023 arXiv

  36. [44]

    Videotree: Adaptive tree-based video representation for llm reasoning on long videos

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for llm reasoning on long videos. arXiv preprint arXiv:2405.19209,

    arXiv

  37. [45]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 ,

    arXiv

  38. [46]

    Retrieval-based video language model for efficient long video question answering

    Jiaqi Xu, Cuiling Lan, Wenxuan Xie, Xuejin Chen, and Yan Lu. Retrieval-based video language model for efficient long video question answering. arXiv preprint arXiv:2312.04931,

  39. [47]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, pages 10371–10381, 2024. 3

    2024

  40. [48]

    Self-chained image-language model for video localization and question answering

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. NeurIPS, 36:76749–76771, 2023. 2

    2023

  41. [49]

    X-trans2cap: Cross-modal knowledge transfer using transformer for 3d dense caption- ing

    Zhihao Yuan, Xu Yan, Yinghong Liao, Yao Guo, Guanbin Li, Shuguang Cui, and Zhen Li. X-trans2cap: Cross-modal knowledge transfer using transformer for 3d dense caption- ing. In CVPR, pages 8563–8573, 2022. 2

    2022

  42. [50]

    3dgraphllm: Combin- ing semantic graphs and large language models for 3d scene understanding, 2024

    Tatiana Zemskova and Dmitry Yudin. 3dgraphllm: Combin- ing semantic graphs and large language models for 3d scene understanding, 2024. 1, 2, 6

    2024

  43. [51]

    A sim- ple llm framework for long-range video question-answering

    Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A sim- ple llm framework for long-range video question-answering. arXiv preprint arXiv:2312.17235, 2023. 2

    2023 arXiv

  44. [52]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 2

    2023 arXiv

  45. [53]

    Multi3drefer: Grounding text description to multiple 3d ob- jects

    Yiming Zhang, ZeMing Gong, and Angel X Chang. Multi3drefer: Grounding text description to multiple 3d ob- jects. In ICCV, pages 15225–15236, 2023. 2

    2023

  46. [54]

    3dvg- transformer: Relation modeling for visual grounding on point clouds

    Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg- transformer: Relation modeling for visual grounding on point clouds. In ICCV, pages 2928–2937, 2021. 2

    2021

  47. [55]

    Lscenellm: Enhancing large 3d scene understand- ing using adaptive visual preferences

    Hongyan Zhi, Peihao Chen, Junyan Li, Shuailei Ma, Xinyu Sun, Tianhang Xiang, Yinjie Lei, Mingkui Tan, and Chuang Gan. Lscenellm: Enhancing large 3d scene understand- ing using adaptive visual preferences. arXiv preprint arXiv:2412.01292, 2024. 6

    2024 arXiv

  48. [56]

    Is,” “Can,

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In ICCV, pages 2911–2921, 2023. 2, 6 10 Supplementary Materials A. Additional Qualitative Results In this section, we present further...

    2023

  49. [2024]

    Accessed: 2025-02-25. 5, 6

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.