{"id":"381a0482-84cf-472c-9de2-c51275626897","arxiv_id":"2505.12194","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Spatial-LLaVA, fine-tuned on the SUN-Spot v2.0 dataset with set-of-marks prompts, reports 56.14% accuracy on Visual Spatial Reasoning, about 3.8 points above LLaVA v1.5 13B.","lead":"Spatial-LLaVA introduces SUN-Spot v2.0, an RGB-D dataset with spatial referring expressions and landmark annotations, then fine-tunes a vision-language model with set-of-marks prompts. The model reports gains on spatial benchmarks, but the evaluation protocol needs a fairer baseline before the gain is fully established.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VSR benchmark claim rests on a rewritten task with SoM prompting and no matched baseline; the +3.77 gain may be protocol-dependent.","rationale":"The central claim requires that the VSR accuracy gain measure spatial referring-expression understanding on the actual VSR benchmark. The weakest link is the evaluation: the authors changed the task format and injected SoM markers at test time. For the claim to hold, the gain needs to survive both the original True/False caption-label format and removal of SoM from the fine-tuned model; neither condition is reported. This is not a disagreement with the value of the dataset or the SoM fine-tuning approach, which are plausible and potentially useful. The concern is specifically that the headline number is protocol-bound. I agree with the reader's conditional verdict: controlled comparisons and artifact release are needed before the accuracy claim is taken as established. The proposed check, official-format VSR for both models, directly tests whether the +3.77 point delta persists when the benchmark is not rewritten. Other concerns, such as Table III being evaluated on a held-out split of the training distribution, are real but secondary; the VSR result is the primary external validation claim.","tokens_in":10907,"tokens_out":7542,"duration_ms":70173,"concrete_test":"Re-run the VSR evaluation for Spatial-LLaVA 13b and LLaVA-v1.5 13b under the original VSR protocol: official caption-label True/False pairs, no question rewriting, no SoM markers, same zero-shot setting. If the accuracy gap is materially smaller than the 3.77 points in Table IV or reverses, the headline claim is protocol-dependent and must be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing claim is the zero-shot VSR improvement in Section IV-D2 and Table IV. The evaluation protocol differs from the benchmark as published: VSR is an image-caption entailment task with True/False labels, but Section IV-D2 says the authors reconstructed the captions into questions and incorporated SoM prompting into the questions. Spatial-LLaVA was fine-tuned on single-turn yes/no spatial questions over SoM-marked images (Sections III-C and III-D), so the reported 56.14% vs 52.37% for LLaVA-v1.5-13b may reflect alignment between the modified test format and the training format, plus SoM-provided grounding, rather than improved spatial referring expression understanding. The paper provides no control that removes SoM from Spatial-LLaVA at inference, no control that adds SoM to a non-spatial-tuned LLaVA, and no statement of whether every baseline received the identical SoM overlay. Even if all baselines did receive identical inputs, the rewritten task is no longer the published VSR benchmark, so the abstract claim of a 3.15% improvement on the zero-shot VSR benchmark is not established as a claim about the original VSR task.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SUN-Spot v2.0, a dataset of RGB-D images with spatial referring expressions and landmark-object annotations, and proposes Spatial-LLaVA, an MLLM obtained by fine-tuning LLaVA on GPT-4o-generated conversational data that incorporates Set-of-Marks (SoM) prompting. The model is evaluated on a held-out split of SUN-Spot v2.0 Expert and on the Visual Spatial Reasoning (VSR) benchmark, reporting accuracy gains over several baselines, including a 3.15% improvement on VSR claimed in the abstract.","tokens_in":11140,"tokens_out":4260,"duration_ms":40914,"significance":"If the results hold, the dataset contribution is potentially valuable: SUN-Spot v2.0 provides landmark annotations for spatial referring expressions, and the SoM-based fine-tuning idea is plausible for reducing semantic bias. The paper includes a human annotation pipeline with second-round review, which is a strength. However, the evidence for the central claims is weakened by evaluation-protocol issues: the VSR benchmark is reformatted and augmented with SoM markers without matched baselines, the SUN-Spot Expert comparison is between a fine-tuned model and zero-shot baselines, and no ablation isolates the effect of SoM. The machine-generated component of the dataset is described but does not appear to be used in training or evaluation. These issues are addressable with additional experiments and clarifications, so the underlying contribution is defensible but not yet convincingly supported.","major_comments":[{"comment":"The VSR evaluation does not follow the original benchmark protocol: the authors reconstruct the caption-label True/False entailment pairs into yes/no questions and add SoM markers to the images. No baseline is evaluated under the same rewritten format, so the reported gain of 56.14% versus 52.37% for LLaVA-13b may reflect alignment between the modified test format and the model's single-turn yes/no fine-tuning data, plus the extra SoM grounding, rather than improved spatial reasoning. Please run all baselines on the same rewritten questions with the same SoM overlay, and also report results on the original True/False entailment format.","section":"Section IV-D2, Table IV"},{"comment":"Table III compares Spatial-LLaVA, which is fine-tuned on 75k question-answer pairs derived from SUN-Spot v2.0 Expert and SUNRefer, against baselines that are evaluated zero-shot (no fine-tuning on the SUN-Spot training split). The claimed 7.5% (7b) and 9.91% (13b) improvements over LLaVA v1.5 are therefore not like-for-like comparisons. The authors should fine-tune at least one baseline on the same training conversations, or re-frame the claim as 'zero-shot baselines vs. supervised fine-tuning on the same distribution.'","section":"Section IV-D1, Table III"},{"comment":"The SUN-Spot v2.0 Machine-generated dataset is introduced as a contribution and described in detail, but the training pipeline in Section III-C and Section IV-B uses only SUN-Spot Expert and SUNRefer to generate 75k question-answer pairs. The machine-generated captions never appear in training or evaluation. Please clarify whether this component is used at all; if not, it should be repositioned as a dataset resource for future work rather than as part of the method's training data.","section":"Section III-A2, Section III-C, Section IV-B"},{"comment":"The abstract claims 'Spatial-LLaVA outperforms previous methods by 3.15% on the zero-shot Visual Spatial Reasoning benchmark dataset,' but Table IV reports 56.14% for Spatial-LLaVA-13b versus 52.37% for LLaVA-13b (a 3.77-point difference) and 53.60% versus 52.95% for the 7b models (a 0.65-point difference). No comparison in Table IV yields 3.15%. Please state precisely how the 3.15% figure is computed, or correct the abstract and any related claims.","section":"Abstract and Table IV"},{"comment":"The paper credits Set-of-Marks prompting with improving spatial understanding, but no ablation isolates its contribution. There is no experiment removing SoM from Spatial-LLaVA at inference, no experiment adding SoM to a baseline LLaVA model, and no comparison between training with and without SoM. Without such controls, the observed gains cannot be attributed to SoM rather than to the fine-tuning data or the question format.","section":"Section III-B and Section IV-D2"}],"minor_comments":[{"comment":"Typo: 'prepossessing' should be 'preprocessing'.","section":"Section IV-D2"},{"comment":"The table header contains 'avgerage' (should be 'average'), and the number of expert descriptions is inconsistent between the text (7,987) and the table (7,990).","section":"Table I"},{"comment":"The machine-generated row reads '10, 333' with an extra space; also, the entry '90k image-caption pairs' in the abstract should be reconciled with the detailed counts in Table I and Table II.","section":"Table I"},{"comment":"The model name is inconsistently rendered as 'Spatial-LLaV A' with a space before the final 'A' in several places; please standardize to 'Spatial-LLaVA'.","section":"Throughout"},{"comment":"The sentence 'Thus, to perform well on these benchmarks, a trained model does not necessarily need to fully understand the spatial relationships' appears without context and seems to contradict the motivation of the paper; please clarify or remove it.","section":"Section II-B"}],"recommendation":"major_revision","confidential_remarks":"The core issue is whether the headline VSR improvement is an artifact of the reformatted task and SoM overlay. The authors should be encouraged to run matched baselines and ablations; the dataset contribution itself has value, but the current evaluation does not cleanly separate the effects of data, SoM, and fine-tuning. The abstract's 3.15% discrepancy should also be checked against the actual table values."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: take the dataset, hold the headline number. The SUN-Spot v2.0 contribution is real and likely useful — landmark annotations on both target and landmark objects, with Set-of-Marks markers aligned to caption mentions. That combination isn't in the prior work I know, and the authors are candid that machine-generated captions degrade on complex prompts. For a robotics or vision lab working on spatial referring expressions, this is a plausible training resource.\n\nThe soft spots are in the evaluation, and they matter as much as the reader's report suggests. The VSR result in Table IV is the paper's hook, but the protocol is not the VSR benchmark as published. The authors converted the original True/False caption entailment pairs into binary yes/no questions and then inserted SoM prompting into the questions. That is a new task, not the original benchmark. The abstract's claim of a 3.15% improvement over previous methods on \"the zero-shot VSR benchmark\" is therefore not supported as a statement about the original benchmark. The 3.77 point gain over LLaVA v1.5 13b might be real under the modified protocol, but it could just reflect alignment between the test format and the model's single-turn yes/no SoM-style training.\n\nThere's also no ablation to separate the contributions. No baseline gets SoM prompts without spatial fine-tuning, no version of Spatial-LLaVA is evaluated without SoM at inference, and Table III is on a held-out split of the same SUN-Spot v2.0 distribution used for training — so the fine-tuned model winning there is unsurprising. One more thing: the paper introduces the 93k-caption machine-generated dataset, but Section III-C shows the fine-tuning conversations were generated only from SUN-Spot v2.0 Expert and SUNRefer. The machine-generated data doesn't appear to be used in the model at all. That's not fatal, but it makes the abstract's \"90k image-caption pairs\" framing feel oversold.\n\nNone of this kills the core dataset contribution. The annotation protocol for landmarks and the SoM-aligned captions are worth a serious look. But the experimental comparison needs rework: run the original VSR task as-is, add SoM to all baselines, and ablate the SoM signal on both sides. The paper deserves peer review, not desk rejection, but an editor should expect major revisions.\n\nWould I bring it to reading group? Maybe. It's more likely to be a useful citable dataset than a reliable method claim.","headline":"The SUN-Spot v2.0 dataset is a real resource, but the headline VSR gain is not established because the benchmark was rewritten and no SoM-matched baseline exists.","tokens_in":11670,"tokens_out":2859,"would_cite":true,"duration_ms":30293,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a vision-language model on captions with marked landmarks lifts zero-shot Visual Spatial Reasoning accuracy to 56.14%, a reported 3.15-point gain over prior methods.","keywords":["Spatial referring expressions","Set-of-Marks prompting","Multimodal large language models","Visual spatial reasoning","SUN-Spot v2.0 dataset","RGB-D","Visual grounding"],"falsifier":"Run Spatial-LLaVA 13b on the original Visual Spatial Reasoning test protocol (caption-label pairs with True/False labels and no set-of-marks tags) and compare with the rewritten-protocol score; a large drop indicates the headline gain depends on the question reformatting rather than spatial understanding.","tokens_in":10721,"feed_emoji":"📍","tokens_out":3809,"duration_ms":33415,"temperature":0.7,"pith_summary":"This paper seeks to show that multimodal large language models can be taught spatial referring expressions by training on a dataset that annotates not just the target object but also the landmark objects used in descriptions. The authors build SUN-Spot v2.0, 90,000 image-caption pairs over 10,000 RGB-D indoor images, and augment every caption with Set-of-Marks tags that tie each mention to a marked region. Fine-tuning LLaVA v1.5 on 75,000 question-answer pairs generated from this data yields 56.14% accuracy on the zero-shot Visual Spatial Reasoning benchmark, which the paper reports as a 3.15% gain over previous methods. A sympathetic reading is that the gain comes from aligning language to marked landmarks, not from object semantics.","feed_headline":"Spatial-LLaVA lifts zero-shot spatial reasoning to 56%","feed_subtitle":"Marking landmarks in images and captions helps a multimodal LLM reason about object positions, beating prior models on VSR.","key_machinery":"The load-bearing mechanism is the Set-of-Marks (SoM) prompt: small alphanumeric markers are placed at the center of each object in an image, and the same markers are appended to object mentions in captions and questions. This creates an explicit pointer from language to visual region, which the paper argues weakens reliance on semantic cues such as object class and color. Around it sits the SUN-Spot v2.0 dataset, built by extending SUN RGB-D with spatial referring expressions and landmark annotations, and conversation data generated from those captions by a language model. The fine-tuning keeps the visual encoder frozen and updates only the projector and language model on single-turn conversations so answers do not leak between questions.","core_discovery":"The central claim is that object-level grounding through Set-of-Marks prompting, applied at training and evaluation, lets an MLLM learn spatial relations with less bias from object identity. On the paper's own benchmark, SUN-Spot v2.0 Expert, Spatial-LLaVA 13b reaches 76.13% accuracy versus 66.22% for LLaVA v1.5 13b, and on the Visual Spatial Reasoning test set it reaches 56.14% versus 52.37%. This is presented as evidence that landmark-level annotations in the training data, not larger backbones or additional semantic cues, drive the improvement.","pith_inferences":["The Visual Spatial Reasoning evaluation rewrote the benchmark's 'True'/'False' captions into 'Yes'/'No' questions and inserted SoM tags; if that reformatting changes difficulty, the 3.15-point gain may partly measure format compatibility rather than spatial reasoning.","The reported improvement should be tested on the untouched Visual Spatial Reasoning protocol, with no question reconstruction and no SoM tags, before treating 56.14% as a true zero-shot spatial reasoning score.","Since the dataset is RGB-D, the same fine-tuning pipeline could be applied to depth-based tasks such as grasping or navigation instruction following, which the paper names as future work.","A cheaper test of the SoM hypothesis would be to train the same model on SUN-Spot v2.0 captions with markers removed and compare Visual Spatial Reasoning accuracy; if the gap disappears, the markers, not the captions, carry the benefit."],"forward_implications":["A dataset with landmark-level annotations is enough to improve zero-shot spatial reasoning in a general MLLM without retraining from scratch.","SoM prompting can be applied at test time in the same form used in training, making the improvement applicable to any image with detected or ground-truth regions.","Single-turn conversation fine-tuning reduces the model's ability to infer answers from previous questions, so reported gains are more likely to reflect per-question spatial judgment.","The 56.14% Visual Spatial Reasoning result suggests spatial language understanding can be improved by training data rather than by larger visual encoders or larger language backbones."],"supporting_citations":[{"why":"Supplies the original SUN-Spot dataset and annotation procedure that v2.0 extends with landmark object annotations.","marker":"[35]"},{"why":"Introduces Set-of-Marks prompting, the visual grounding method used to align object mentions with image regions.","marker":"[47]"},{"why":"Provides the LLaVA v1.5 base MLLM and pre-trained weights that Spatial-LLaVA fine-tunes.","marker":"[29]"},{"why":"Defines the Visual Spatial Reasoning benchmark and test set used for the zero-shot evaluation.","marker":"[26]"},{"why":"Adds SUNRefer spatial referring expressions used as supplementary training data.","marker":"[27]"},{"why":"Provides the SUN RGB-D images and ground-truth segmentation that the dataset is built on.","marker":"[41]"},{"why":"Supplies the Vicuna v1.5 language backbone used inside the MLLM.","marker":"[8]"}],"fun_headline_variants":["Spatial-LLaVA: Landmark annotations boost VSR to 56%","Set-of-Marks prompting sharpens multimodal spatial skills","Marking objects helps MLLMs reason about spatial relations","SUN-Spot v2.0 + Spatial-LLaVA: better spatial understanding","New MLLM with landmark cues beats prior VSR scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that rewriting Visual Spatial Reasoning captions into yes/no questions with set-of-marks tags leaves the task's difficulty unchanged.","fun_headline_variants_meta":{"raw":{"variants":["Spatial-LLaVA: Landmark annotations boost VSR to 56%","Set-of-Marks prompting sharpens multimodal spatial skills","Marking objects helps MLLMs reason about spatial relations","SUN-Spot v2.0 + Spatial-LLaVA: better spatial understanding","New MLLM with landmark cues beats prior VSR scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1460,"prompt_tokens":958,"completion_tokens":502,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":407}},"tokens_in":574,"tokens_out":502,"duration_ms":5365,"temperature":1.0,"reasoning_tokens":407,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:38:24.679102+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Spatial-LLaVA 13b on the original Visual Spatial Reasoning test protocol (caption-label pairs with True/False labels and no set-of-marks tags) and compare with the rewritten-protocol score; a large drop indicates the headline gain depends on the question reformatting rather than spatial understanding.","supporting_citations":[{"cited_title":"Sun-spot: An rgb-d dataset with spatial referring expressions","cited_arxiv_id":null,"evidence_quote":"Supplies the original SUN-Spot dataset and annotation procedure that v2.0 extends with landmark object annotations."},{"cited_title":"Refer-it-in-rgbd: A bottom-up approach for 3d visual grounding in rgbd images","cited_arxiv_id":null,"evidence_quote":"Adds SUNRefer spatial referring expressions used as supplementary training data."}],"review_version":1}