{"id":"1309da81-f782-4589-a490-f8f40d121a18","arxiv_id":"2602.04635","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Adding spatial relationship edges to 3D scene graphs improves LLM-based object grounding, with no statistically significant advantage for open-vocabulary relations over closed-vocabulary ones.","lead":"This paper tests whether adding spatial relations between objects to 3D scene graphs helps AI models pick out the object a person refers to in a command. It finds explicit spatial edges improve grounding accuracy, but open-vocabulary relations generated by a vision model do not clearly beat fixed-relation labels.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VLA-3D support for RQ1 is potentially circular: referential statements are generated from the very edges added in G_P,E, so gains may reflect format matching rather than improved grounding.","rationale":"The reader identified the load-bearing concern as the view-independence and entity-coverage of VLM-generated open-vocabulary edges, which affects RQ2 (open vs closed vocabulary). However, the central claim of the paper is RQ1: spatial edges improve grounding. The most load-bearing threat to RQ1 is the circular dependency between the evaluation statements and the added edges on VLA-3D. Since the statements are derived from the edges, the G_P,E condition can be solved by lexical matching, while G_P requires geometric reasoning. This confound is present in the strongest experimental condition (GPT-4o on VLA-3D, the only significant GPT-4o edge effect). The REACT dataset includes human commands that avoid this issue, but the reported aggregate results do not isolate them, and the overall GPT-4o REACT edge gain is not statistically significant. This does not overturn the paper's conclusion—GPT-5 on REACT does show a significant edge benefit—but it means the central claim is less robust than the raw VLA-3D numbers suggest. The paper is transparent and appropriately hedged, so a conditional verdict remains appropriate. The reader's concern about RQ2 is valid but secondary relative to this confound in RQ1.","tokens_in":11595,"tokens_out":5341,"duration_ms":56822,"concrete_test":"Run the G_P vs G_P,E comparison restricted to the 119 REACT human-command statements for both GPT-4o and GPT-5, applying McNemar's test. If neither model shows a significant improvement on this non-edge-derived subset, then the central claim rests solely on VLA-3D statements generated from the graph edges, indicating a circular evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that explicit spatial edges improve object grounding is primarily established on the VLA-3D dataset, where each referential statement is generated from a specific spatial edge in the scene's 3DSG (Section V-B.1). In the G_P,E condition, the LLM sees those exact edges as text, allowing direct pattern-matching between the statement's relation phrase and the edge. In G_P, it must infer the same relation from bounding boxes—a substantially harder reasoning task. The difference may therefore measure an advantage of explicit surface-form alignment, not a general improvement in grounding. The REACT human-command subset is not edge-derived and provides a fairer test, but the paper does not report G_P vs G_P,E results separately for that subset. Overall, the only significant GPT-4o edge effect is on VLA-3D (Table III), so the main non-circular evidence for the central claim is limited to GPT-5 on REACT. This does not invalidate RQ1, but it weakens its generalizability to natural, non-edge-derived commands.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether adding explicit spatial edges to 3D scene graphs helps LLMs ground natural-language referential statements to objects, and whether open-vocabulary edges generated from images by a VLM perform differently from closed-vocabulary edges. The authors build an LLM-based serialization/query pipeline, a VLM-based edge-generation pipeline, and evaluate GPT-4o and GPT-5 on VLA-3D and REACT scenes. They report accuracy gains for the graph with edges (G_{P,E}) over the position-only graph (G_P) and the node-only baseline (G), with McNemar significance in most comparisons, and conclude that open-vocabulary relations are feasible but show no significant difference from closed-vocabulary relations.","tokens_in":11846,"tokens_out":5339,"duration_ms":59168,"significance":"If the central claim is supported, the result is practically useful for robot scene understanding and for deciding how to enrich 3DSGs with relational edges. The paper's strengths are its use of standard off-the-shelf models, the inclusion of two datasets and two LLMs, the use of McNemar's test for paired comparisons, a human-authored command set, and a candid discussion of failure cases in VLM-generated edges. The public release of prompts is also a positive feature. However, the evidence for the main generalization is weakened by the fact that the primary evaluation uses statements generated from the very edges being tested, and the RQ2 result is based on a very small and potentially biased sample. The contribution is therefore more of a qualified empirical study than a definitive demonstration, though the approach is worth publishing after revision.","major_comments":[{"comment":"The main evidence for the central claim is partly circular. The VLA-3D referential statements are generated from the specific spatial edges present in the 3DSG, and the REACT generated statements are created with the same VLA-3D codebase. In G_{P,E}, the LLM is given those exact edges as text, so it can match the relation phrase directly; in G_P it must infer the relation from bounding boxes. The improvement may therefore reflect surface-form matching rather than improved grounding. The 119 human-authored statements in Section V-B.5 are a fairer test, but Table II reports only the 74 generated REACT statements (the accuracy values are consistent with denominators of 74), and no G_P vs G_{P,E} comparison is reported for the human-command set. For GPT-4o, the only significant edge effect is on VLA-3D; on REACT the G_P vs G_{P,E} cell is blank, i.e., not significant. Please report RQ1 on th","section":"V-B.1, V-C, Tables II-III"},{"comment":"The RQ2 conclusion that open- and closed-vocabulary edges perform comparably is not well supported because many of the generated open-vocabulary edges are invalid relative to the 3DSG. The paper documents camera-relative descriptions ('left side of the table, closer to the camera'), references to objects absent from the graph ('glass wall and stairs'), and attributes not represented in the graph ('red chair'). Including such edges in the evaluation means the null result may reflect generation unreliability rather than vocabulary expressiveness. In addition, only 26 generated-command statements are tested, of which 22 are 'near' and 4 are 'on', and no McNemar statistics or power analysis are reported. The conclusion should be stated as inconclusive rather than as evidence of equivalence.","section":"V-D.2, V-E"},{"comment":"The selection procedure for the RQ2 test set introduces a bias that limits external validity. Edges for which no fitting image was available were replaced with the original closed-vocabulary edge, and only the 26 statements whose edges were newly generated were tested. The image-selection criterion (highest combined pixel count of the two object masks) can also favor large objects. Because the tested set is small and skewed toward the 'near' relation, the absence of a statistically significant difference in Table IV is not informative about the general question of vocabulary choice. The paper should report edge-level statistics, exact p-values, and ideally a larger or more balanced sample.","section":"V-D.1, Table IV"}],"minor_comments":[{"comment":"The notation G_{P,E} is used for both the closed- and open-vocabulary edge graphs in Experiment 2; this is confusing. Use distinct subscripts, e.g., G_{P,E}^{closed} and G_{P,E}^{open}.","section":"V-D.1, notation"},{"comment":"Table III has an empty cell for GPT-4o G vs G_P, and no p-values are reported. Please include exact McNemar p-values (or a dash with 'n.s.') for every comparison, and state the number of statements used in each test.","section":"Tables II-III"},{"comment":"The description of the human-command collection is slightly ambiguous: 104 images, two annotations each, leading to 208 statements, of which 119 survive after ambiguity filtering. It would help to state how many objects and scenes these correspond to, and whether any scene was excluded.","section":"V-B.5"},{"comment":"Some example generated edges are long and would consume significant token budget; consider discussing whether edge length or graph total serialization size affects model performance, especially for larger scenes.","section":"V-D.2"},{"comment":"The scalability discussion is welcome but is only qualitative. A short analysis of token counts for the graphs used would strengthen the claim that token limits are a practical barrier.","section":"VI"}],"recommendation":"major_revision","confidential_remarks":"The central claim is currently supported mainly by edge-derived statements, and the human-command dataset is not used for RQ1. If the authors can rerun the G_P vs G_{P,E} comparison on the 119 human-command statements, or clearly restrict their claims to procedurally generated statements, the paper would be much stronger. The RQ2 experiment is too small to support a general conclusion and should be reframed as a feasibility study with explicit statistical limitations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper you're asking about is the first systematic test I know of for whether explicit spatial edges in 3D scene graphs actually help LLMs ground referential statements. That is a real gap, and the authors fill it with a competently run study: two models, two datasets, McNemar tests, and an honest discussion of failure cases. The VLM pipeline that generates open-vocabulary edges from images with color-outlined objects is a modest method contribution. The paper also gets credit for reporting the GPT-3.5 degradation and for acknowledging the small scale of Experiment 2.\n\nThe soft spots are real, though. The stress-test note about circularity on VLA-3D holds up. Each referential statement there is generated from a specific spatial edge in the graph. In the G_P,E condition, the LLM sees that exact edge as text and can pattern-match the relation phrase; in G_P, it has to infer the relation from bounding boxes. That is not a clean test of whether edges improve grounding. The REACT human-command subset is the fair test, but the paper never reports G_P vs. G_P,E on those 119 statements. So the non-circular evidence for the edge increment reduces to GPT-5 on REACT generated commands, which are themselves edge-derived. The abstract's stronger phrasing overstates what is actually shown.\n\nRQ2 is underpowered—only 26 statements for the open-vs-closed comparison—and the open-vocabulary edges have documented quality problems: camera-relative relations, references to objects absent from the graph, and outline colors leaking into the text. The null result is appropriately hedged, but it can't support a firm conclusion. Also, the baseline G is computed as random chance, not an LLM baseline, so G vs. G_P,E conflates adding position with adding edges.\n\nWho gets value from this? Robotics researchers working with 3DSGs for language grounding. It is a useful data point, not a breakthrough. The authors think clearly and engage honestly with their limitations, so I would send this to peer review with the expectation of major revision: release code and data, add confidence intervals, and report the human-command subset for the G_P vs. G_P,E comparison. That would let the edge effect stand on non-circular ground.\n\nI'd bring it to a reading group interested in empirical methodology in embodied AI, and I'd cite it cautiously in my own work as early evidence, not proof.\n\nBest.","headline":"A genuinely useful, transparent empirical study on whether spatial edges in 3DSGs help LLM object grounding, but the headline result leans on a dataset where the testing procedure may inflate the benefit.","tokens_in":12327,"tokens_out":2762,"would_cite":true,"duration_ms":29444,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding explicit spatial relations between objects in a 3D scene graph measurably improves a language model's ability to ground natural-language commands to the right object.","keywords":["3D scene graphs","object grounding","natural language commands","spatial relations","large language models","vision-language models","open-vocabulary","robot perception"],"falsifier":"Run the same grounding task on a dataset where the spatial edges are randomly shuffled among object pairs while keeping the graph otherwise identical; if LLM accuracy stays at the same level as with correct edges, the edges are not carrying the claimed information.","tokens_in":11493,"feed_emoji":"🤖","tokens_out":6268,"duration_ms":60489,"temperature":0.7,"pith_summary":"This paper asks whether a simple graph augmentation—adding labeled spatial edges between objects in a 3D scene graph—helps large language models resolve natural-language commands about objects. The authors build a pipeline that serializes scene graphs into text, then let two LLMs pick the referenced object ID. They find that adding position attributes and, more so, spatial edges consistently raises grounding accuracy across two datasets and both models, with gains up to 26 percentage points and most comparisons statistically significant. They also test whether open-vocabulary edges generated by a vision-language model from robot images outperform fixed closed-vocabulary edges; they find no significant difference, and they document generation errors that suggest reliability, not expressiveness, is the current bottleneck.","feed_headline":"Spatial edges lift LLM object grounding by up to 26 points","feed_subtitle":"Adding explicit relations between objects to a robot's scene graph makes commands far more reliable, and stronger models gain more.","key_machinery":"The key object is the spatial edge added to the 3D scene graph: a directed relation between two object IDs, expressed either as a closed-vocabulary term like 'on' or as an open-vocabulary sentence. These edges are serialized into text along with object nodes and fed to an LLM that must return the target object's ID. The edges are what convert a collection of object positions into a relational description that matches how humans talk about scenes, and their presence—not just object positions—drives the accuracy gains.","core_discovery":"The central claim is that explicit spatial relations in a 3D scene graph are not decorative—they carry information that LLMs use to disambiguate objects. On a large indoor-scene benchmark, grounding accuracy for GPT-4o rose from 76.97% with positions only to 84.27% with spatial edges; for GPT-5 it rose from 98.17% to 99.58%, and from a 73.0% baseline to 99.58% overall. On a robot-captured dataset with human-authored commands, both models improved substantially when edges were added. The paper further demonstrates that open-vocabulary edges generated from images by a VLM are feasible, but these generated edges sometimes describe camera-relative layouts or entities outside the graph, and on a","pith_inferences":["The absence of a statistically significant gap between open- and closed-vocabulary edges is likely power-limited; a larger benchmark with dozens of scenes and hundreds of generated edges could reveal a real difference once generation errors are filtered.","The documented failures—camera-relative descriptions and references to colors not in the graph—suggest a concrete fix: constrain VLM generation to the object set and to a fixed camera-independent reference frame, which could turn open-vocabulary edges into a strict improvement.","The same edge-augmentation idea could transfer to other downstream tasks such as spatial navigation instructions or manipulation target selection, where relational descriptions are equally natural.","Because the paper shows edges help even when they are occasionally noisy, robust relation generation may matter less than always ensuring edges are present."],"forward_implications":["Robots can substantially improve command understanding by storing explicit spatial relations in their scene graphs, even without retraining the language model.","The benefit of spatial edges grows with model capability: the stronger of the two LLMs gained more from the edges in every comparison, suggesting future models will rely even more on this kind of structured relational input.","Open-vocabulary edges generated from robot images are usable, but their quality must be checked for view-independence and in-graph references before deployment.","For large scenes, the token cost of serializing all edges is a bottleneck; approaches that select relevant subgraphs or retrieve only nearby objects will be needed."],"fun_headline_variants":["Spatial edges in scene graphs lift LLM grounding by 26 points","Adding explicit spatial relations to robot scene graphs boosts grounding","LLM grounding accuracy jumps when scene graphs add spatial links","Robot command grounding improved by spatial edges in 3D scene graphs","Spatial relations in scene graphs key to better robot grounding"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire comparison rests on the assumption that the spatial edges actually describe the scene the graph represents; if generated edges reference things outside the graph or depend on the camera's viewpoint, the observed effects and the open-vs-closed comparison become unreliable.","fun_headline_variants_meta":{"raw":{"variants":["Spatial edges in scene graphs lift LLM grounding by 26 points","Adding explicit spatial relations to robot scene graphs boosts grounding","LLM grounding accuracy jumps when scene graphs add spatial links","Robot command grounding improved by spatial edges in 3D scene graphs","Spatial relations in scene graphs key to better robot grounding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000262,"raw_usage":{"total_tokens":1485,"prompt_tokens":846,"completion_tokens":639,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":552}},"tokens_in":590,"tokens_out":639,"duration_ms":7439,"temperature":1.0,"reasoning_tokens":552,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T04:29:19.920984+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same grounding task on a dataset where the spatial edges are randomly shuffled among object pairs while keeping the graph otherwise identical; if LLM accuracy stays at the same level as with correct edges, the edges are not carrying the claimed information.","supporting_citations":[],"review_version":1}