{"id":"bcc1f472-0599-4081-aaef-50e08fd273be","arxiv_id":"2505.04911","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SpatialPrompting shows that keyframe selection plus camera-pose prompting can make a pre-trained multimodal LLM competitive on 3D question answering without 3D-specific fine-tuning.","lead":"SpatialPrompting lets off-the-shelf multimodal AI models answer questions about 3D indoor scenes by selecting a few camera images and feeding them along with camera positions to the model. It reports near-state-of-the-art scores on the ScanQA and SQA3D benchmarks without any task-specific fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot claim is undermined by benchmark answer-list prior; annotation, not spatial prompting, drives the reported SOTA.","rationale":"The central contribution is advertised as zero-shot SOTA. The only component that produces a large gain is the top-20 answer list from the training set. The paper's own ablation demonstrates this. This is not merely a definitional quibble: the method without the prior is not competitive with even the naive baseline. Thus the main claim is not robust to a stricter, standard reading of zero-shot. The reader's conditional verdict is appropriate; the paper should either clearly label the method few-shot, or re-run and report the zero-shot numbers and temper the SOTA claim. Also noted, but secondary, are the lack of code and error bars; the answer-prior issue is the decisive concern.","tokens_in":15340,"tokens_out":4645,"duration_ms":46707,"concrete_test":"A single decisive check: run the full SpatialPrompting pipeline on the ScanQA validation set with the answer-list annotation replaced by a generic concise-answer instruction (the paper's own 'w/o Annotation' condition), and report EM@1 and ROUGE-L against the GPT-4o baseline on the same keyframes and camera poses. If EM@1 is at or below baseline (Table 2 already indicates 19.83 vs 21.43), the zero-shot SOTA claim must be withdrawn or re-framed as few-shot with benchmark answer priors; also report the result averaged over at least three GPT-4o runs with fixed temperature to rule out API stochasticity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's core claim is state-of-the-art zero-shot spatial reasoning without 3D-specific fine-tuning. The prompt, however, includes an 'annotation' component (Sec. 4.3, Supp. C) that lists the top 20 most frequent ground-truth answers from each benchmark's training split, organized by question type. This is a benchmark-specific, training-derived prior. The ablation in Table 2 shows it is the dominant term: removing it drops ScanQA EM@1 from 27.34 to 19.83, ROUGE-L from 43.39 to 31.89, CIDEr from 87.69 to 55.06, and SQA3D average from 52.74 to 47.77. More strikingly, the 'w/o Annotation' row (19.83) falls below the 'GPT-4o baseline' (21.43), meaning the keyframe/camera-pose prompt without the answer list provides no benefit over raw GPT-4o and may slightly hurt. Consequently, the 'state-of-the-art zero-shot' headline is not a property of the proposed keyframe/prompt mechanism; it is largely a property of the leaked answer distribution. The paper itself calls this 'few-shot prompting' (Sec. 4.3), which conflicts with the zero-shot terminology used in the title, abstract, and conclusion. Under a definition of zero-shot that excludes access to the benchmark's training answer distribution, the central claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SpatialPrompting, a training-free framework for 3D question answering. It selects a diverse set of keyframes from RGB-D video using CLIP-based vision-language similarity, Mahalanobis distance on point-cloud distributions, field-of-view spread, and image sharpness, then feeds these keyframes together with camera poses and a structured prompt to off-the-shelf multimodal LLMs (GPT-4o and Gemini-2.0). The prompt includes a preamble, keyframe/camera-pose blocks, an 'annotation' component, and the user query. The authors report state-of-the-art zero-shot performance on ScanQA (EM@1 27.34, ROUGE-L 43.39, SPICE 20.49) and competitive results on SQA3D without any 3D-specific fine-tuning. The contribution is framed as replacing expensive 3D-specific training with a scalable, prompt-based alternative.","tokens_in":15596,"tokens_out":4506,"duration_ms":41293,"significance":"If the central claim held, the paper would make a useful practical contribution: demonstrating that off-the-shelf MLLMs can perform 3D spatial QA from keyframes and camera poses would open a low-cost alternative to specialized 3D-VQA models. The keyframe selection algorithm is clearly described, the ablations are informative, and the qualitative failure analysis (e.g., miscounting, direction confusion) is honest and useful. However, the headline 'zero-shot SOTA' claim is compromised by the fact that the prompt's annotation component is built from the top-20 most frequent answers in each benchmark's training split, and the ablation in Table 2 shows that this component is responsible for most of the reported gain. The proposed keyframe/camera-pose mechanism alone, without the answer prior, does not outperform the raw GPT-4o baseline on ScanQA. The paper would need to either reframe its claims as few-shot assisted performance or demonstrate an independent benefit of the prompt mechanism without benchmark-derived answer lists.","major_comments":[{"comment":"The paper's 'zero-shot' claim is contradicted by the few-shot annotation. Section 4.3 explicitly states that the annotation for benchmark datasets is 'the top 20 common answers for each question type' obtained from the training dataset. Removing this annotation drops ScanQA EM@1 from 27.34 to 19.83 and ROUGE-L from 43.39 to 31.89, which is below the GPT-4o baseline (21.43 and 32.18). Thus the reported SOTA is largely driven by a benchmark-specific answer prior, not by the proposed keyframe and camera-pose prompting. The title, abstract, and conclusion call the method 'zero-shot,' while the text self-describes the annotation as 'few-shot prompting.' This is a load-bearing inconsistency. Please either (a) revise the claim to 'few-shot assisted' and provide an evaluation where the few-shot demonstrations are not drawn from the benchmark's training answers, or (b) remove the answer-list component and report the resulting performance, tempering the SOTA statement accordingly.","section":"Sec. 4.3, 'Prompt generation', and Table 2"},{"comment":"The comparison with previous methods is not apples-to-apples. Table 1 compares SpatialPrompting exclusively against methods that are fine-tuned on 3D data; there is no zero-shot baseline in the table (the GPT-4o baseline appears only in the ablation Table 2). Moreover, the reported margins over the strongest fine-tuned baselines are extremely small (EM@1 27.34 vs 27.2; SPICE 20.49 vs 20.44). Because the evaluation uses a stochastic API model (GPT-4o), the paper should report repeated runs, confidence intervals, or a statistical significance test before claiming 'state-of-the-art zero-shot performance.'","section":"Table 1 and Sec. 4.3"},{"comment":"The ablation shows that the core mechanism is not independently effective. The 'w/o Annotation' row gives ScanQA EM@1 of 19.83 and ROUGE-L of 31.89, both below the GPT-4o baseline of 21.43 and 32.18. This means that keyframe extraction plus camera poses, without the benchmark answer prior, provides no benefit over raw GPT-4o and may even hurt. The paper should explicitly discuss this fact and report what fraction of the total gain is attributable to the prompt structure versus the answer prior. Without such a decomposition, the claim that 'keyframe-driven prompt generation' enables zero-shot spatial reasoning is not supported.","section":"Table 2 and Sec. 4.3.2"}],"minor_comments":[{"comment":"Section 4.1 appears as a heading with no content; the datasets and metrics are only described in Sec. 4.2. Please renumber the sections so the 'Quantitative Results' heading is not empty.","section":"Sec. 4.2"},{"comment":"In the row for '5' images, the value '39,07' uses a comma as a decimal separator; it should be '39.07' for consistency with the rest of the table.","section":"Table 5"},{"comment":"The sentence 'The annotation is consists of the top 20 frequent answers' should be 'The annotation consists of the top 20 frequent answers.'","section":"Supp. C"},{"comment":"The camera-pose template has an inconsistent space in 'Camera rotation : [x= {roll}°, y= {pitch}°, z={yaw}°]'. Please make the formatting uniform and check that the rendered prompt matches the intended syntax.","section":"Sec. 3.3"},{"comment":"The term 'annotation' is used for two different things: the instruction not to refer to images, and the benchmark-specific few-shot answer list. Please disambiguate these two uses to avoid confusion.","section":"Sec. 4.3"}],"recommendation":"major_revision","confidential_remarks":"The central claim of state-of-the-art zero-shot spatial reasoning is not supported by the evidence as presented, because the dominant component of the reported gain is the benchmark-specific answer prior. This is fixable within the scope of the manuscript by reframing the contribution as few-shot assisted prompting and adding experiments that separate the effect of the answer prior from the keyframe/camera-pose mechanism. I recommend major revision rather than rejection, since the method and problem are interesting and the authors have provided informative ablations and an honest failure analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result in this paper doesn't hold up. SpatialPrompting reports state-of-the-art zero-shot performance on ScanQA, but the main driver is an \"annotation\" that lists the top 20 most frequent training-set answers per question type. The ablation (Table 2) shows removing it drops ScanQA EM@1 from 27.34 to 19.83—below the plain GPT-4o baseline (21.43). So the keyframe and camera-pose prompts, without the answer prior, actually hurt or at best do nothing. The paper calls this few-shot prompting in Sec. 4.3, which contradicts the zero-shot framing in the title, abstract, and conclusion.\n\nWhat's genuinely new is the combination of keyframe selection using Mahalanobis distance on point-cloud distributions plus CLIP similarity, a quality score favoring wide field of view and sharpness, and camera poses encoded as Euler angles in the prompt. That's a reasonable, training-free way to give an MLLM spatial context, and the ablations around these components are helpful. The paper is clearly written and the qualitative examples are illustrative.\n\nThe soft spots, in order of severity: (1) the zero-shot claim is misleading as explained above; (2) no error bars or multiple runs, despite using stochastic API models—the reported margins over strong baselines are small (EM@1 27.34 vs. 27.2; SPICE 20.49 vs. 20.44); (3) all baselines are fine-tuned 3D models, with no zero-shot baselines from other prompt-based methods, making \"competitive\" hard to interpret; (4) no code or prompt templates released.\n\nThe central mechanism—keyframe selection plus camera pose—contributes only a modest gain (a few points over uniform sampling or no pose). The few-shot answer list is the dominant term. That doesn't make the method useless, but it makes the current title and claims untenable.\n\nWho gets value from this? Researchers working on training-free 3D-VQA and prompt engineering for MLLMs. It deserves peer review because the method is worth examining and the failure mode is instructive, but a serious referee should push for re-framing, proper zero-shot baselines, error bars, and code release.\n\nMy recommendation: send it to review, but expect heavy revision. The core idea is there; the evaluation and framing are not.","headline":"The paper's zero-shot SOTA claim is driven by a training-answer prior, not by spatial prompting, though the keyframe/camera-pose method itself has merit.","tokens_in":16175,"tokens_out":2672,"would_cite":true,"duration_ms":23943,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SpatialPrompting shows that keyframes plus camera poses can replace 3D fine-tuning for spatial question answering.","keywords":["spatial question answering","zero-shot learning","multimodal large language models","keyframe selection","camera pose prompting","3D visual question answering","ScanQA","SQA3D"],"falsifier":"Take the ScanQA validation set and replace the annotation's top-20 training answers with a permuted or unrelated list (for example, the same words in random order); if EM@1 falls toward the 19.83 no-annotation ablation, the reported gain is a training-answer prior rather than a spatial-reasoning effect, whereas if it stays near 27.34 the model is using the list only as an output-format cue.","tokens_in":15117,"feed_emoji":"🧭","tokens_out":7897,"duration_ms":66634,"temperature":0.7,"pith_summary":"SpatialPrompting claims that off-the-shelf multimodal large language models can answer questions about 3D scenes without any 3D-specific training, provided they receive a small set of well-chosen keyframes plus the camera pose of each frame. On the ScanQA benchmark this prompt-only approach reaches the state of the art on exact match, ROUGE-L, and SPICE, and it stays competitive on the situated-question benchmark SQA3D, against models that were fine-tuned on point clouds or voxel features. The proposal matters because it replaces an expensive training pipeline with a prompt-engineering recipe built from standard components: vision-language features, a spatial-distance metric, and a pose-annotated image template. The paper also shows that most of the gain depends on a few-shot annotation listing the most frequent training answers, which is the part a stricter reading of 'zero-shot' would question.","feed_headline":"Keyframes and camera poses beat 3D-tuned models on ScanQA","feed_subtitle":"No point clouds or fine-tuning: off-the-shelf multimodal LLMs answer 3D spatial questions from a few posed images.","key_machinery":"The machinery is the keyframe-selection algorithm together with the pose-aware prompt template. For every pair of frames, the combined distance $d'(i,j)=d(i,j)+\\alpha(1-S(i,j))$ mixes a Mahalanobis distance $d$ between the mean and covariance of the frames' back-projected point clouds with a CLIP cosine-similarity term $S$; the algorithm repeatedly removes the frame with the lower quality score $q_t=\\det|\\Sigma_t|+\\beta\\,\\mathrm{Var}(\\nabla^2 I_t)$, which prefers wider point-cloud spread and sharper images. The prompt then attaches to each surviving image a camera pose given as position $[x,y,z]$ in meters and rotation as Euler angles, a representation the paper found easier for LLMs to interpret than quaternions or matrices. This combination is what lets the LLM reconstruct spatial layout without ever seeing a point cloud.","core_discovery":"The central claim is that spatial reasoning in 3D environments is largely an emergent capability of modern multimodal LLMs, and that the role of the vision stack is to expose the right views with the right positional context. SpatialPrompting selects keyframes by combining CLIP-based semantic similarity with the Mahalanobis distance between per-frame point-cloud statistics, then scores candidates by a quality term favoring wide field of view and sharpness. The surviving keyframes are placed in a prompt template that states each camera's position and Euler-angle rotation, followed by a task annotation and the user question. With GPT-4o this pipeline obtains 27.34 EM@1, 43.39 ROUGE-L, and 20.49 SPICE on ScanQA validation, ahead of all compared 3D-tuned systems on those metrics, and an average of 52.74 on SQA3D, competitive with but slightly below the specialized state of the art. The authors conclude that expensive 3D-specific fine-tuning is not necessary for strong spatial QA, and that the remaining weaknesses are directional questions that depend on the user's orientation and precise metric estimation.","pith_inferences":["Under a stricter definition of zero-shot that forbids using benchmark training statistics, the method is better described as few-shot with a benchmark-specific answer prior: the annotation lists the top 20 training answers per question type, and removing it drops EM@1 from 27.34 to 19.83.","The same keyframe-and-pose recipe could be adapted to embodied tasks such as navigation or manipulation, where poses come from SLAM in real time; the SQA3D failure analysis suggests that transforming poses into the user's coordinate frame, not the camera's, would be a necessary adaptation for situated questions.","A testable extension is to replace the static top-20 answer list with a per-scene prior generated from object labels detected in the keyframes, which could preserve the output-formatting benefit without leaking benchmark answer frequencies.","Since the method delegates reasoning to the off-the-shelf LLM, its ceiling is set by that model's spatial priors; running the same prompt on smaller open-weight models would reveal how much of the result is prompting versus model scale."],"forward_implications":["3D visual question answering no longer requires a dedicated 3D encoder or fine-tuning; any multimodal LLM with strong spatial priors can be pointed at a scene through this prompt recipe.","The method's ScanQA exact-match score (27.34) and ROUGE-L (43.39) set a new prompt-only reference, so future work can benchmark vision-language prompting against 3D-tuned models without training.","The ablation shows that camera pose contributes about 1.6 EM@1 points on ScanQA and keyframe selection adds roughly one more point over uniform sampling, so both components are load-bearing for the reported gains.","On SQA3D, removing the camera pose slightly improves the average score, indicating that in situated questions the camera coordinate frame can conflict with the user's orientation; the paper treats this as a limitation, not a universal advantage.","The approach inherits the counting and orientation weaknesses of the underlying LLM, as documented in the paper's failure analysis, so applications needing exact counts or user-centric directions will require supplemental modules."],"supporting_citations":[{"why":"Supplies the ScanQA benchmark and evaluation metrics (EM@1, ROUGE-L, SPICE) that the method tops.","marker":"[2]"},{"why":"Supplies the SQA3D benchmark and its question-type taxonomy, including the orientation-dependent questions where the method fails.","marker":"[27]"},{"why":"Provides the CLIP vision-language features used for semantic keyframe selection.","marker":"[34]"},{"why":"DROID-SLAM estimates the camera poses that the prompts attach to each keyframe.","marker":"[36]"},{"why":"Depth Anything supplies monocular depth estimates when only RGB frames are available for back-projection.","marker":"[47]"},{"why":"The primary multimodal LLM (GPT-4o) whose spatial reasoning the prompts exploit.","marker":"[32]"},{"why":"Gemini-2.0 serves as the second tested LLM, showing the recipe transfers across models.","marker":"[15]"},{"why":"Grounds the in-context-learning premise that few-shot prompting adapts an LLM to a task without fine-tuning.","marker":"[3]"}],"fun_headline_variants":["Zero-shot 3D reasoning from keyframes and camera poses","Keyframes alone: off-the-shelf LLMs rival 3D-tuned systems","No 3D inputs, no fine-tuning: SpatialPrompting tops ScanQA","Prompt with posed images, not point clouds, for 3D QA","SpatialPrompting: emergent 3D reasoning in multimodal LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method counts as zero-shot only if feeding the model the most common answers from each benchmark's training set is a permissible prompt aid rather than a benchmark-specific prior.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot 3D reasoning from keyframes and camera poses","Keyframes alone: off-the-shelf LLMs rival 3D-tuned systems","No 3D inputs, no fine-tuning: SpatialPrompting tops ScanQA","Prompt with posed images, not point clouds, for 3D QA","SpatialPrompting: emergent 3D reasoning in multimodal LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000964,"raw_usage":{"total_tokens":4120,"prompt_tokens":980,"completion_tokens":3140,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":3040}},"tokens_in":596,"tokens_out":3140,"duration_ms":21172,"temperature":1.0,"reasoning_tokens":3040,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:17:56.868833+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the ScanQA validation set and replace the annotation's top-20 training answers with a permuted or unrelated list (for example, the same words in random order); if EM@1 falls toward the 19.83 no-annotation ablation, the reported gain is a training-answer prior rather than a spatial-reasoning effect, whereas if it stays near 27.34 the model is using the list only as an output-format cue.","supporting_citations":[{"cited_title":"Scanqa: 3d question answering for spatial scene understanding","cited_arxiv_id":null,"evidence_quote":"Supplies the ScanQA benchmark and evaluation metrics (EM@1, ROUGE-L, SPICE) that the method tops."},{"cited_title":"Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras","cited_arxiv_id":null,"evidence_quote":"DROID-SLAM estimates the camera poses that the prompts attach to each keyframe."},{"cited_title":"Depth anything: Unleashing the power of large-scale unlabeled data","cited_arxiv_id":null,"evidence_quote":"Depth Anything supplies monocular depth estimates when only RGB frames are available for back-projection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The primary multimodal LLM (GPT-4o) whose spatial reasoning the prompts exploit."},{"cited_title":"Gemini 2.0","cited_arxiv_id":null,"evidence_quote":"Gemini-2.0 serves as the second tested LLM, showing the recipe transfers across models."}],"review_version":1}