{"id":"f697dd93-643d-43a1-9d29-7716fd195b59","arxiv_id":"2506.13629","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FreeQ-Graph builds a semantically aligned 3D scene graph using CLIP, LLaVA, and an LLM, and reports strong zero-shot results on 3D visual grounding, segmentation, and scene graph prediction.","lead":"This paper presents FreeQ-Graph, a system that builds a 3D scene graph from images and point clouds and uses a language model to answer free-form queries such as 'find the pillow near the table lamp' without task-specific training. It reports strong zero-shot results on 3D grounding, segmentation, and scene graph generation across six benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The superpoint labels used in Eq. 2 appear to come from the same LVLM-generated object list, so the semantic-alignment module may be injecting category text into node features rather than providing true 3D consistency.","rationale":"The reader's weakest assumption is the same load-bearing concern I identify: the semantic labels used in the superpoint merging and node-feature fusion are underspecified and potentially circular. The paper's strongest claim rests on these 'semantic aligned features', and the ablation in Table VI shows they drive a decisive part of the improvement. Because Eq. 2 computes superpoint similarity from text-encoded labels, and the figure shows a 'free-form vocabulary' feeding the label generation, the most economical explanation of the alignment gain is that node features are enriched with the same category names that later match the query's CLIP text embedding. This directly threatens the central contribution, not just the comparison fairness. The condition is testable: swapping the text-label features for pure visual features, or hiding the label, would reveal whether any consistency signal beyond category-text matching remains. I therefore keep the reader's conditional verdict; the paper is not yet acceptable as-is, but the issue is methodological transparency and reproducibility rather than an evident fabrication.","tokens_in":20625,"tokens_out":3887,"duration_ms":40914,"concrete_test":"Re-run Sr3D and Nr3D grounding with the semantic-alignment module modified so that f_Qi in Eqs. 2 and 4 is replaced by a CLIP visual descriptor of the superpoint's top-5 2D masks (or by a constant 'unknown' token), keeping the LVLM node generation, captions, edges, and two-stage LLM reasoning unchanged. If the Acc@0.25 gap between rows 2 and 3 of Table VI collapses, the claimed semantic-consistency gain is dominated by category-name injection rather than genuine 3D alignment. Separately, release the exact prompt and labeling process used to assign 'desk', 'chair', etc. to superpoints before Eq. 2, and verify those labels are independent of the node object list from Eq. 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central innovation is the 'semantic consistent' alignment in Sec. III-C, but the provenance of the superpoint semantic labels is never specified. Fig. 3 shows a 'free-form vocabulary' feeding '3D semantic instance labels', and Eq. 2 defines the superpoint similarity A_ij using f_Qi and f_Qj, 'obtained by encoding their label into feature vectors using a text encoder.' If those labels come from the same LVLM/LLM object list that produced the graph nodes in Eq. 1, then spectral clustering in Eq. 3 groups superpoints by pre-existing category text, and Eq. 4's mean-pooling of f_Qi into f_hat_i injects the category name into every node feature. Under this reading, the large Table VI gains attributed to semantic alignment (rows 2-3: Sr3D Acc@0.1 33.7 to 43.3, Nr3D 29.7 to 38.9) are explainable as label-text leakage: grounding and segmentation comparisons to CLIP query embeddings succeed because the node feature already contains the matching category text, not because the 3D representation is semantically consistent. The manuscript must state exactly how each superpoint obtains its label before Eq. 2 and show that this process does not reuse the LVLM object list or query vocabulary.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"FreeQ-Graph proposes a training-free pipeline for 3D scene understanding that builds a scene graph from multi-view RGB-D observations using an LVLM and a 2D instance segmentation model, then aligns graph nodes with so-called 3D semantic-consistent features obtained by superpoint clustering, and finally answers free-form queries via a two-stage LLM reasoning procedure. The paper reports zero-shot results on ScanRefer, Sr3D, Nr3D, Replica, ScanNet, and 3DSSG, claiming state-of-the-art performance among zero-shot methods and competitive performance with fully supervised approaches.","tokens_in":20889,"tokens_out":5592,"duration_ms":55927,"significance":"If the label-provenance concern is resolved, this is a substantial empirical contribution: the paper covers six benchmarks, includes ablations and efficiency comparisons, and would demonstrate that an open-vocabulary 3D grounding system can operate without training. The authors also state that code and datasets will be released, which would support reproducibility. The main risk is that the key semantic-alignment module may be injecting category text from the same LVLM vocabulary into node features, which would make the reported gains in grounding and segmentation less informative.","major_comments":[{"comment":"The provenance of the superpoint semantic labels is never specified, and this is load-bearing for the paper's central claim. Eq. (2) defines f_Qi and f_Qj as features \"obtained by encoding their label into feature vectors using a text encoder,\" but no source for these labels is stated before the similarity matrix is constructed. Eq. (3) produces only cluster indices via spectral clustering; it does not produce label strings such as \"desk\" or \"chair.\" If the label strings are taken from the LVLM/LLM object list of Eq. (1) or from the query vocabulary, then the clustering in Eq. (3) groups superpoints by pre-existing category text, and the mean-pooling in Eq. (4) injects that same category text into every node feature. Query-to-node matching would then succeed because the node feature contains the matching text embedding, not because the 3D representation is semantically consistent. Please specify exactly how each superpoint obtains its label, verify that the labels are independent of the LVLM object list and query vocabulary, and provide an ablation that removes the text-feature fusion in Eq. (4) to demonstrate that the gains in Table VI are not an artifact of label-text leakage.","section":"III-C, Eqs. (2)-(4)"},{"comment":"The ablation labeled \"SA\" bundles superpoint clustering, label assignment, and text-feature fusion, and row 3 improves Sr3D Acc@0.1 from 33.7 to 43.3 and Nr3D Acc@0.1 from 29.7 to 38.9. Because Eq. (4) fuses the label text embedding into the node feature, this improvement could be explained by feature construction alone rather than by a genuinely 3D-consistent semantic representation. Please break out the contribution of geometric superpoint merging using only visual CLIP features from the contribution of text-embedding fusion, and report both configurations on the same evaluation protocol.","section":"Table VI and Sec. IV-C2"},{"comment":"The implementation details report consistent thresholds tau_iou=0.9 and tau_sim=0.9, but Eq. (2) uses only tau_iou; tau_sim never appears in any equation. In addition, the procedure for converting the clusters produced by Eq. (3) into semantic label strings (e.g., \"desk\") is not described anywhere in Sec. III-C. Without a precise definition of these steps and thresholds, the method is not reproducible and the claimed \"semantic consistency\" cannot be independently verified. Please define tau_sim formally and specify the cluster-to-label mapping, including how label names are obtained and whether they are verified against the 3D scene.","section":"IV-A and III-C"}],"minor_comments":[{"comment":"There are several typos, including \"accuray\" for \"accuracy\" and \"groud-truth\" for \"ground-truth\" in Sec. IV-A2 and Sec. IV-B1.","section":"IV-A and IV-B"},{"comment":"The ablation text refers to \"Table 5\" when the referenced ablation table is actually Table VI; please correct the cross-reference.","section":"IV-C"},{"comment":"Fig. 4 is described as a \"quantitative comparison,\" but the figure shows qualitative grounding examples; please change to \"qualitative.\"","section":"IV-B1"},{"comment":"The notation in Eq. (1), including the expression \"V,{Mi}M i=1,\" is syntactically unclear; please rewrite the equation and define all symbols before first use.","section":"III-B, Eq. (1)"},{"comment":"The text cites \"CLIP [7]\" but reference [7] is Clip-FO3D; the actual CLIP paper appears as reference [64]. Please correct the citation.","section":"Sec. I and References"},{"comment":"The \"out-of-word queries\" evaluation is mentioned as validated with manually annotated ground truth, but the query generation process, the number of queries, and the annotation protocol are not described; please provide these details.","section":"IV-A2"},{"comment":"The metric \"fmIoU\" is used but never defined; please define it in Sec. IV-A2 alongside mAcc and mIoU.","section":"Table IV"}],"recommendation":"major_revision","confidential_remarks":"The central concern is the circularity risk in Sec. III-C: if the superpoint labels used in Eq. (2) and Eq. (4) originate from the same LVLM/LLM object list or query vocabulary, the reported improvements over ConceptGraphs and BBQ are not interpretable as gains in 3D semantic consistency. The authors need to state the label source, demonstrate independence from the query vocabulary, and provide a visual-features-only ablation. This is a fixable issue within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a real system paper, not a hallucination. The integration of LVLM/LLM scene graph construction, CLIP-based superpoint merging, and two-stage LLM reasoning is new relative to ConceptGraphs, BBQ, PoLo, and VL-SAT, and the zero-shot numbers on Sr3D, Nr3D, and ScanRefer are genuinely ahead of the baselines compared. The authors did broad experiments across six datasets and included ablations showing each component contributes. That part is solid.\n\nThe soft spot is exactly where the reader and stress-test point: the semantic alignment module. Eq. 2 builds a superpoint similarity matrix using features 'obtained by encoding their label into feature vectors using a text encoder,' and Eq. 4 fuses that same text feature into the final node feature. The paper never says where the superpoint labels come from. If they come from the same LVLM/LLM object list that generated the graph nodes, then the category name is injected into every node feature, and the Table VI gains from semantic alignment (33.7 to 43.3 on Sr3D, for example) are plausibly text-label leakage rather than genuine 3D consistency. This is load-bearing because semantic consistency is the paper's main claimed contribution. The authors need to specify an external label source, or demonstrate that labels are derived from geometry and visual features alone, and ideally release code or intermediate features so reviewers can check.\n\nOther issues are minor: the Sr3D/Nr3D evaluation uses a filtered subset, which is non-standard; there are no error bars; and the Table V claim that the method 'outperforms most fully-supervised models' is not supported by the table, since SGRec3D and VL-SAT beat it on every reported metric. The prose has typos and some rough edges, but that is fixable.\n\nThe central argument holds up as a system claim; the flaw is a missing specification, not an internal contradiction. This paper deserves a serious referee, not a desk reject. I would send it out with a clear request to resolve the label provenance and release code. If the labels turn out to be the LVLM list, the ablations need to be rerun with a clean separation. If that can be ruled out, this is a useful contribution to open-vocabulary 3D grounding.","headline":"A useful training-free 3D grounding system built from known parts, but the core semantic-alignment module is underspecified and may leak category text into node features — worth peer review with a demand to fix that.","tokens_in":21438,"tokens_out":4194,"would_cite":false,"duration_ms":35946,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free pipeline that builds a semantically consistent 3D scene graph can answer free-form and relational queries in 3D scenes without any predefined vocabulary or training.","keywords":["3D scene understanding","free-form querying","3D scene graph","3D visual grounding","zero-shot learning","semantic alignment","superpoint merging","large multimodal models"],"falsifier":"Take a ScanNet or Replica scene, run FreeQ-Graph twice with the same visual inputs but with the LVLM object-list prompt restricted to exclude the query category (e.g., remove 'pillow' from the list while querying 'the pillow near the lamp'). If Acc@0.5 drops toward the visual-only baseline, the claimed 3D semantic alignment is not the source of the gains; if accuracy holds, the superpoint labels provide genuinely independent 3D semantics.","tokens_in":20395,"feed_emoji":"🎯","tokens_out":5917,"duration_ms":54591,"temperature":0.7,"pith_summary":"The paper claims that a fully training-free pipeline can answer free-form and relation-aware queries in 3D scenes at state-of-the-art zero-shot accuracy, provided it builds a scene graph whose nodes carry 3D-consistent semantic labels rather than relying on the ungrounded object lists that vision-language models produce. It argues that existing open-vocabulary methods are limited by predefined vocabularies, while LLM/LVLM-based methods lack 3D scene awareness and generate inconsistent labels. FreeQ-Graph instead combines an LLM/LVLM-generated 3D scene graph with superpoint-based semantic alignment and a two-stage chain-of-thought reasoning algorithm. If the claim holds, a robot or embodied agent could ground arbitrary natural-language queries in scans without any task-specific training.","feed_headline":"Free-form 3D queries answered by training-free scene graph","feed_subtitle":"Superpoint merging plus two-stage LLM reasoning aligns 3D objects with semantic labels, beating zero-shot baselines on six benchmarks.","key_machinery":"The load-bearing object is the semantic-consistent 3D scene graph, whose nodes are objects with point clouds, bounding boxes, captions, and fused features, and whose edges encode spatial relations with distances. The argument's engine is the superpoint alignment module: graph cuts produce superpoints; a similarity matrix $A_{ij}$ scores how strongly two superpoints co-occur in projected 2D masks and agree in text-encoder representation; spectral clustering on the normalized Laplacian turns those scores into a small set of 3D semantic instance labels. Fusing each node's CLIP visual feature with its assigned label's text feature gives the 'semantic-aligned' representation $\\hat{f}_i$, and the two-stage LLM chain-of-thought reasoning then operates on these aligned nodes. The authors also use an eigengap heuristic to choose the number of semantic clusters automatically.","core_discovery":"FreeQ-Graph's central claim is that 3D semantic consistency, not stronger language models, is what unlocks free-form 3D querying. The method builds a scene graph $G=(V,E)$ by prompting an LVLM for object categories, grounding them with a 2D instance segmenter, and asking an LLM for captions and pairwise spatial relations. It then segments the point cloud into superpoints, merges them by a similarity matrix combining 2D mask overlap and text-feature agreement, and performs spectral clustering to obtain 3D semantic instance labels. Each graph node's final feature is the mean-pooled combination of its CLIP visual feature and its superpoint label's text feature ($\\hat{f}_i = \\varphi(f_{Q_i}, f_i)$), which the authors argue corrects the semantic misalignment that plagues pure LLM/LVLM scene graphs. Free-form queries are answered by a two-stage LLM reasoner that first identifies candidate targets and relations from object- and scene-level descriptions, then selects the final target. On six datasets the authors report that this training-free pipeline exceeds all zero-shot baselines on 3D visual grounding, semantic segmentation, and scene graph generation, and approaches fully supervised methods.","pith_inferences":["A direct test of the alignment claim: censor the LVLM's object-list prompt so it never names the query category, then re-run grounding; if accuracy collapses, the superpoint labels are leaking language-side category information rather than contributing independent 3D consistency.","The framework is naturally a module for embodied tasks like object navigation ('find a place to sit near the window'), since it already outputs object identities, spatial relations, and distances.","The graph representation could also serve as a memory store for multi-turn dialogue about a scene, with the semantic-aligned features enabling queries phrased in terms of function ('something to dispose of wastepaper') rather than object names.","If label leakage is controlled for and the gains persist, the superpoint-merging idea transfers to other fusion tasks, such as aligning 2D masks from multiple views or regularizing open-vocabulary detectors with geometric consistency."],"forward_implications":["A robot or agent can ground arbitrary natural-language queries in a 3D scene without being retrained on that query type or vocabulary.","The approach's accuracy is not tied to one LLM: the authors show consistent results across GPT-3.5, GPT-4, GPT-4o, Vicuna, and Qwen2-VL backbones.","Zero-shot 3D semantic segmentation and scene-graph generation improve to the point of competing with fully supervised models, suggesting open-vocabulary 3D perception no longer needs large curated training sets.","Because the pipeline is training-free, adding a new scene or sensor stream reduces to running the same graph construction and reasoning stages, with no weight update required.","Complex relational queries, such as 'the gray pillow between two orange pillows', are decomposed into candidate-and-relation subtasks, which is the paper's explanation for its lead on hard splits."],"supporting_citations":[{"why":"ConceptGraphs is the main zero-shot baseline and the source of the caption-generation and scene-graph construction pattern that the method extends.","marker":"[13]"},{"why":"BBQ is the strongest zero-shot baseline that also builds LLM-based scene graphs, used for comparison and for ablating reasoning algorithms.","marker":"[14]"},{"why":"LLaVA is the LVLM whose prompted responses provide the initial free-form object lists and node captions.","marker":"[38]"},{"why":"Grounded-SAM is the 2D instance segmenter used to ground LVLM object lists into 2D masks and correct hallucinations.","marker":"[39]"},{"why":"PoLo supplies the graph-cuts superpoint segmentation idea that feeds the semantic alignment module.","marker":"[40]"},{"why":"ReferIt3D (Sr3D/Nr3D) provides datasets and the evaluation protocol for top-1 and IoU-based grounding.","marker":"[41]"},{"why":"ScanRefer is the benchmark for 3D visual grounding with unique, multiple, and overall splits.","marker":"[42]"},{"why":"CLIP ViT-L/14 is the visual and text encoder used for node features and superpoint label features.","marker":"[64]"}],"fun_headline_variants":["LLM-built scene graph answers free-form 3D queries","Training-free 3D semantic querying via LLM scene graph","Semantic consistency beats bigger LLMs for 3D querying","Zero-shot 3D understanding with LLM+LVLM scene graphs","Free-form 3D questions solved by scene graph reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each superpoint can be assigned a semantic label that reflects the true 3D object identity before the similarity matrix is built; if those labels come from the same vision-language model that named the scene graph's objects, the feature fusion in Eq. (4) can make category information circular rather than adding genuine 3D consistency.","fun_headline_variants_meta":{"raw":{"variants":["LLM-built scene graph answers free-form 3D queries","Training-free 3D semantic querying via LLM scene graph","Semantic consistency beats bigger LLMs for 3D querying","Zero-shot 3D understanding with LLM+LVLM scene graphs","Free-form 3D questions solved by scene graph reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0004,"raw_usage":{"total_tokens":2166,"prompt_tokens":1101,"completion_tokens":1065,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":717,"completion_tokens_details":{"reasoning_tokens":976}},"tokens_in":717,"tokens_out":1065,"duration_ms":9339,"temperature":1.0,"reasoning_tokens":976,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:57:53.993571+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a ScanNet or Replica scene, run FreeQ-Graph twice with the same visual inputs but with the LVLM object-list prompt restricted to exclude the query category (e.g., remove 'pillow' from the list while querying 'the pillow near the lamp'). If Acc@0.5 drops toward the visual-only baseline, the claimed 3D semantic alignment is not the source of the gains; if accuracy holds, the superpoint labels provide genuinely independent 3D semantics.","supporting_citations":[{"cited_title":"Vocabulary-Free 3D Instance Segmentation with Vision and Language Assistant","cited_arxiv_id":"2408.10652","evidence_quote":"PoLo supplies the graph-cuts superpoint segmentation idea that feeds the semantic alignment module."},{"cited_title":"Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes,","cited_arxiv_id":null,"evidence_quote":"ReferIt3D (Sr3D/Nr3D) provides datasets and the evaluation protocol for top-1 and IoU-based grounding."},{"cited_title":"Scanrefer: 3d object local- ization in rgb-d scans using natural language,","cited_arxiv_id":null,"evidence_quote":"ScanRefer is the benchmark for 3D visual grounding with unique, multiple, and overall splits."}],"review_version":2}