Pith. sign in

REVIEW 3 major objections 7 minor 73 references

FreeQ-Graph: Free-form Querying with Semantic Consistent Scene Graph for 3D Scene Understanding

T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A training-free pipeline that builds a semantically consistent 3D scene graph can answer free-form and relational queries in 3D scenes without any predefined vocabulary or training.

desk verdict A useful training-free 3D grounding system built from known parts, but the core semantic-alignment module is underspecified and may leak category text into node features — worth peer review with a demand to fix that. read the letter →

arxiv 2506.13629 v2 pith:3DZOOHB2 submitted 2025-06-16 cs.CV

Chenlu Zhan , Yufei Zhang , Gaoang Wang , Hongwei Wang This is my paper · ORCID
classification cs.CV
keywords 3Dsceneunderstandingfree-formqueryinggraphvisualgroundingzero-shotlearningsemanticalignmentsuperpointmerginglargemultimodalmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a fully training-free pipeline can answer free-form and relation-aware queries in 3D scenes at state-of-the-art zero-shot accuracy, provided it builds a scene graph whose nodes carry 3D-consistent semantic labels rather than relying on the ungrounded object lists that vision-language models produce. It argues that existing open-vocabulary methods are limited by predefined vocabularies, while LLM/LVLM-based methods lack 3D scene awareness and generate inconsistent labels. FreeQ-Graph instead combines an LLM/LVLM-generated 3D scene graph with superpoint-based semantic alignment and a two-stage chain-of-thought reasoning algorithm. If the claim holds, a robot or embodied agent could ground arbitrary natural-language queries in scans without any task-specific training.

What carries the argument

The load-bearing object is the semantic-consistent 3D scene graph, whose nodes are objects with point clouds, bounding boxes, captions, and fused features, and whose edges encode spatial relations with distances. The argument's engine is the superpoint alignment module: graph cuts produce superpoints; a similarity matrix $A_{ij}$ scores how strongly two superpoints co-occur in projected 2D masks and agree in text-encoder representation; spectral clustering on the normalized Laplacian turns those scores into a small set of 3D semantic instance labels. Fusing each node's CLIP visual feature with its assigned label's text feature gives the 'semantic-aligned' representation $\hat{f}_i$, and the two-stage LLM chain-of-thought reasoning then operates on these aligned nodes. The authors also use an eigengap heuristic to choose the number of semantic clusters automatically.

What would settle it

Take a ScanNet or Replica scene, run FreeQ-Graph twice with the same visual inputs but with the LVLM object-list prompt restricted to exclude the query category (e.g., remove 'pillow' from the list while querying 'the pillow near the lamp'). If Acc@0.5 drops toward the visual-only baseline, the claimed 3D semantic alignment is not the source of the gains; if accuracy holds, the superpoint labels provide genuinely independent 3D semantics.

Watch

Extended reading notes

Core claim

FreeQ-Graph's central claim is that 3D semantic consistency, not stronger language models, is what unlocks free-form 3D querying. The method builds a scene graph $G=(V,E)$ by prompting an LVLM for object categories, grounding them with a 2D instance segmenter, and asking an LLM for captions and pairwise spatial relations. It then segments the point cloud into superpoints, merges them by a similarity matrix combining 2D mask overlap and text-feature agreement, and performs spectral clustering to obtain 3D semantic instance labels. Each graph node's final feature is the mean-pooled combination of its CLIP visual feature and its superpoint label's text feature ($\hat{f}_i = \varphi(f_{Q_i}, f_i)$), which the authors argue corrects the semantic misalignment that plagues pure LLM/LVLM scene graphs. Free-form queries are answered by a two-stage LLM reasoner that first identifies candidate targets and relations from object- and scene-level descriptions, then selects the final target. On six datasets the authors report that this training-free pipeline exceeds all zero-shot baselines on 3D visual grounding, semantic segmentation, and scene graph generation, and approaches fully supervised methods.

Load-bearing premise

The load-bearing premise is that each superpoint can be assigned a semantic label that reflects the true 3D object identity before the similarity matrix is built; if those labels come from the same vision-language model that named the scene graph's objects, the feature fusion in Eq. (4) can make category information circular rather than adding genuine 3D consistency.

Editorial extensions

If this is right

  • A robot or agent can ground arbitrary natural-language queries in a 3D scene without being retrained on that query type or vocabulary.
  • The approach's accuracy is not tied to one LLM: the authors show consistent results across GPT-3.5, GPT-4, GPT-4o, Vicuna, and Qwen2-VL backbones.
  • Zero-shot 3D semantic segmentation and scene-graph generation improve to the point of competing with fully supervised models, suggesting open-vocabulary 3D perception no longer needs large curated training sets.
  • Because the pipeline is training-free, adding a new scene or sensor stream reduces to running the same graph construction and reasoning stages, with no weight update required.
  • Complex relational queries, such as 'the gray pillow between two orange pillows', are decomposed into candidate-and-relation subtasks, which is the paper's explanation for its lead on hard splits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the alignment claim: censor the LVLM's object-list prompt so it never names the query category, then re-run grounding; if accuracy collapses, the superpoint labels are leaking language-side category information rather than contributing independent 3D consistency.
  • The framework is naturally a module for embodied tasks like object navigation ('find a place to sit near the window'), since it already outputs object identities, spatial relations, and distances.
  • The graph representation could also serve as a memory store for multi-turn dialogue about a scene, with the semantic-aligned features enabling queries phrased in terms of function ('something to dispose of wastepaper') rather than object names.
  • If label leakage is controlled for and the gains persist, the superpoint-merging idea transfers to other fusion tasks, such as aligning 2D masks from multiple views or regularizing open-vocabulary detectors with geometric consistency.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. FreeQ-Graph proposes a training-free pipeline for 3D scene understanding that builds a scene graph from multi-view RGB-D observations using an LVLM and a 2D instance segmentation model, then aligns graph nodes with so-called 3D semantic-consistent features obtained by superpoint clustering, and finally answers free-form queries via a two-stage LLM reasoning procedure. The paper reports zero-shot results on ScanRefer, Sr3D, Nr3D, Replica, ScanNet, and 3DSSG, claiming state-of-the-art performance among zero-shot methods and competitive performance with fully supervised approaches.

Significance. If the label-provenance concern is resolved, this is a substantial empirical contribution: the paper covers six benchmarks, includes ablations and efficiency comparisons, and would demonstrate that an open-vocabulary 3D grounding system can operate without training. The authors also state that code and datasets will be released, which would support reproducibility. The main risk is that the key semantic-alignment module may be injecting category text from the same LVLM vocabulary into node features, which would make the reported gains in grounding and segmentation less informative.

major comments (3)
  1. [III-C, Eqs. (2)-(4)] The provenance of the superpoint semantic labels is never specified, and this is load-bearing for the paper's central claim. Eq. (2) defines f_Qi and f_Qj as features "obtained by encoding their label into feature vectors using a text encoder," but no source for these labels is stated before the similarity matrix is constructed. Eq. (3) produces only cluster indices via spectral clustering; it does not produce label strings such as "desk" or "chair." If the label strings are taken from the LVLM/LLM object list of Eq. (1) or from the query vocabulary, then the clustering in Eq. (3) groups superpoints by pre-existing category text, and the mean-pooling in Eq. (4) injects that same category text into every node feature. Query-to-node matching would then succeed because the node feature contains the matching text embedding, not because the 3D representation is semantically consistent. Please specify exactly how each superpoint obtains its label, verify that the labels are independent of the LVLM object list and query vocabulary, and provide an ablation that removes the text-feature fusion in Eq. (4) to demonstrate that the gains in Table VI are not an artifact of label-text leakage.
  2. [Table VI and Sec. IV-C2] The ablation labeled "SA" bundles superpoint clustering, label assignment, and text-feature fusion, and row 3 improves Sr3D Acc@0.1 from 33.7 to 43.3 and Nr3D Acc@0.1 from 29.7 to 38.9. Because Eq. (4) fuses the label text embedding into the node feature, this improvement could be explained by feature construction alone rather than by a genuinely 3D-consistent semantic representation. Please break out the contribution of geometric superpoint merging using only visual CLIP features from the contribution of text-embedding fusion, and report both configurations on the same evaluation protocol.
  3. [IV-A and III-C] The implementation details report consistent thresholds tau_iou=0.9 and tau_sim=0.9, but Eq. (2) uses only tau_iou; tau_sim never appears in any equation. In addition, the procedure for converting the clusters produced by Eq. (3) into semantic label strings (e.g., "desk") is not described anywhere in Sec. III-C. Without a precise definition of these steps and thresholds, the method is not reproducible and the claimed "semantic consistency" cannot be independently verified. Please define tau_sim formally and specify the cluster-to-label mapping, including how label names are obtained and whether they are verified against the 3D scene.
minor comments (7)
  1. [IV-A and IV-B] There are several typos, including "accuray" for "accuracy" and "groud-truth" for "ground-truth" in Sec. IV-A2 and Sec. IV-B1.
  2. [IV-C] The ablation text refers to "Table 5" when the referenced ablation table is actually Table VI; please correct the cross-reference.
  3. [IV-B1] Fig. 4 is described as a "quantitative comparison," but the figure shows qualitative grounding examples; please change to "qualitative."
  4. [III-B, Eq. (1)] The notation in Eq. (1), including the expression "V,{Mi}M i=1," is syntactically unclear; please rewrite the equation and define all symbols before first use.
  5. [Sec. I and References] The text cites "CLIP [7]" but reference [7] is Clip-FO3D; the actual CLIP paper appears as reference [64]. Please correct the citation.
  6. [IV-A2] The "out-of-word queries" evaluation is mentioned as validated with manually annotated ground truth, but the query generation process, the number of queries, and the annotation protocol are not described; please provide these details.
  7. [Table IV] The metric "fmIoU" is used but never defined; please define it in Sec. IV-A2 alongside mAcc and mIoU.

Circularity Check

1 steps flagged · score 6.0 of 10

Superpoint features in Eqs. (2)-(4) are built from the very semantic labels whose '3D consistency' they are supposed to establish; without an external label source the semantic-alignment module injects category text into node features, making query matching partially self-fulfilling.

  1. self definitional [Sec. III-C, Eqs. (2)-(4), Fig. 3]
    "fQi and fQj are the semantic representations of Qi and Qj, obtained by encoding their label into feature vectors using a text encoder. ... we encode the superpoint’s semantic label Qi through a encoder to obtain its semantic feature fQi, and fuse it with the visual feature via meanpooling φ to obtain the final semantically aligned representation ˆfi."

    The semantic label is an input to the feature, not an output. Eq. (2) builds the superpoint similarity matrix from text embeddings of superpoint labels, and Eq. (4) mean-pools that same label embedding into the final node feature. The paper never states where each superpoint's label comes from; the only object-category source in the pipeline is the LVLM/LLM list in Eq. (1), supplemented by the 'free-form vocabulary' shown in Fig. 3. If those same category names supply the labels, then the spectral clustering in Eq. (3) groups superpoints by pre-existing category text, and every node feature already contains its category name. Retrieval and segmentation that match node features to CLIP text embeddings of the query class (Sec.

full rationale

The main circularity is localized to the semantic-alignment module, which is the paper's stated central contribution. Eq. (2) requires each superpoint to already have a semantic label so that its text embedding fQi can be computed, yet the provenance of those labels is never specified. If, as Fig. 3 and the preceding node-generation step suggest, the labels are the same LVLM/LLM free-form vocabulary used to construct the object list, then Eq. (4)'s mean-pooling of fQi into the node feature makes category-level query matching self-fulfilling: the node feature contains the category name by construction, so matching it to CLIP text embeddings of the query is not independent evidence of 3D semantic consistency. The large gains attributed to the semantic-alignment ablation (Table VI, rows 2-3) are then explainable as label-text leakage rather than as a genuinely 3D-consistent representation. The other components—3D scene graph construction, edge generation, and the two-stage LLM reasoning in Eqs. (5)-(6)—do not show the same circularity, and there is no load-bearing self-citation chain. Because the semantic-alignment claim is central but the graph and reasoning components retain independent content, a partial-circularity score of 6 is appropriate rather than a higher score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

FreeQ-Graph does not introduce new physical entities or fitted physics constants. Its free parameters are hand-set thresholds and view counts that affect clustering and captioning. The load-bearing axioms are the assumed completeness of the LVLM object list, the assumed availability and correctness of superpoint semantic labels, and the assumption that CLIP text-image alignment is sufficient for open-vocabulary semantic matching. The most fragile item is the unstated origin of superpoint labels, which is central to the claimed semantic consistency.

free parameters (4)
  • tau_iou = 0.9
    Threshold for 2D mask IoU projection in the superpoint similarity matrix in Eq. 2; set consistently across experiments without sensitivity analysis.
  • tau_sim = 0.9
    Threshold for semantic similarity in superpoint merging; set by hand, not optimized, and no ablation is reported.
  • association threshold = 1.1
    Inherited from ConceptGraph for object association; used in the pipeline but not analyzed.
  • number of captioning views = top-10
    Selected image crops from the top-10 best views for node caption generation; no ablation on this choice is provided.
assumptions (5)
  • domain assumption CLIP text and image encoders provide a shared semantic space that aligns free-form language with 2D and 3D visual features.
    Invoked in Eq. 2, Eq. 4, and the segmentation evaluation; if CLIP alignment is weak for abstract or relational queries, the method degrades.
  • domain assumption The LVLM object list, after Grounded-SAM verification, contains all relevant objects in the scene.
    Sec. III-B describes 'complete and accurate nodes generation without priors'; missed objects cannot be retrieved or reasoned about.
  • ad hoc to paper Superpoints can be assigned semantic labels, and those labels are consistent with true 3D object semantics.
    Sec. III-C uses f_Q that encodes 'their label', but the paper never specifies how labels are produced or verified before Eq. 2 and Eq. 4.
  • standard math Spectral clustering with the eigengap heuristic yields the correct number and grouping of 3D instances.
    Eq. 3 assumes the cluster structure in the similarity matrix reflects object instances rather than arbitrary groupings.
  • domain assumption LLM-generated spatial relations and captions from node coordinates are sufficiently accurate for reasoning.
    Sec. III-B and III-D rely on LLM outputs for edges and captions; the paper's own Limitations section admits node captions are prone to errors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FreeQ-Graph: Free-form Querying with Semantic Consistent Scene Graph for 3D Scene Understanding." pith.science (2026). https://pith.science/paper/3DZOOHB2

@misc{pith2026250613629,
  author       = {Pith},
  title        = {Pith review of: FreeQ-Graph: Free-form Querying with Semantic Consistent Scene Graph for 3D Scene Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3DZOOHB2}},
  note         = {Machine review of arXiv:2506.13629}
}
read the original abstract

Semantic querying in complex 3D scenes through free-form language presents a significant challenge. Existing 3D scene understanding methods use large-scale training data and CLIP to align text queries with 3D semantic features. However, their reliance on predefined vocabulary priors from training data hinders free-form semantic querying. Besides, recent advanced methods rely on LLMs for scene understanding but lack comprehensive 3D scene-level information and often overlook the potential inconsistencies in LLM-generated outputs. In our paper, we propose FreeQ-Graph, which enables Free-form Querying with a semantic consistent scene Graph for 3D scene understanding. The core idea is to encode free-form queries from a complete and accurate 3D scene graph without predefined vocabularies, and to align them with 3D consistent semantic labels, which accomplished through three key steps. We initiate by constructing a complete and accurate 3D scene graph that maps free-form objects and their relations through LLM and LVLM guidance, entirely free from training data or predefined priors. Most importantly, we align graph nodes with accurate semantic labels by leveraging 3D semantic aligned features from merged superpoints, enhancing 3D semantic consistency. To enable free-form semantic querying, we then design an LLM-based reasoning algorithm that combines scene-level and object-level information to intricate reasoning. We conducted extensive experiments on 3D semantic grounding, segmentation, and complex querying tasks, while also validating the accuracy of graph generation. Experiments on 6 datasets show that our model excels in both complex free-form semantic queries and intricate relational reasoning.

Figures

Figures reproduced from arXiv: 2506.13629 by the authors.

Figure 2
Figure 2. The structure of FreeQ-Graph. We propose FreeQ-Graph to realize free-form semantic querying without any training priors for 3D scene understanding. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Detailed process of 3D semantic instance label generation and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Comparison of 3D object visual grounding task with free-form query. The ground truth box is in green. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Comparison semantic segmentation on the Replica dataset. The semantic map highlights the regions most relevant to the query’s semantic features, [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparison of our semantic consistent scene graph with other scene [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Error analysis of FreeQ-Graph on ScanRefer dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 56 canonical work pages

  1. [1]

    3ur-llm: An end- to-end multimodal large language model for 3d scene understanding,

    H. Xiong, Y . Zhuge, J. Zhu, L. Zhang, and H. Lu, “3ur-llm: An end- to-end multimodal large language model for 3d scene understanding,” IEEE Transactions on Multimedia , vol. 27, pp. 2899–2911, 2025

    work page 2025

  2. [2]

    Language-assisted 3d scene understanding,

    Y . Wu, Q. Gao, R. Zhang, H. Li, and J. Zhang, “Language-assisted 3d scene understanding,” IEEE Transactions on Multimedia , 2025

    work page 2025

  3. [3]

    Openscene: 3d scene understanding with open vocabularies,

    S. Peng, K. Genova, C. Jiang, A. Tagliasacchi, M. Pollefeys, T. Funkhouser et al. , “Openscene: 3d scene understanding with open vocabularies,” in Proc. Computer Vision and Pattern Recognition , 2023, pp. 815–824

    work page 2023

  4. [4]

    Pla: Language- driven open-vocabulary 3d scene understanding,

    R. Ding, J. Yang, C. Xue, W. Zhang, S. Bai, and X. Qi, “Pla: Language- driven open-vocabulary 3d scene understanding,” in Proc. Computer Vision and Pattern Recognition , 2023, pp. 7010–7019

    work page 2023

  5. [5]

    Clip-fields: Weakly supervised semantic fields for robotic memory,

    N. M. M. Shafiullah, C. Paxton, L. Pinto, S. Chintala, and A. Szlam, “Clip-fields: Weakly supervised semantic fields for robotic memory,” arXiv preprint arXiv:2210.05663 , 2022

    arXiv 2022

  6. [6]

    Llm-grounder: Open-vocabulary 3d visual grounding with large language model as an agent,

    J. Yang, X. Chen, S. Qian, N. Madaan, M. Iyengar, D. F. Fouhey, and J. Chai, “Llm-grounder: Open-vocabulary 3d visual grounding with large language model as an agent,” 2023

    work page 2023

  7. [7]

    Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip,

    J. Zhang, R. Dong, and K. Ma, “Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip,” in Proc. of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 2048–2059

    work page 2023

  8. [8]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

Show all 73 references
  1. [9]

    Scene-llm: Extending language model for 3d visual understanding and reasoning,

    R. Fu, J. Liu, X. Chen, Y . Nie, and W. Xiong, “Scene-llm: Extending language model for 3d visual understanding and reasoning,” arXiv preprint arXiv:2403.11401, 2024

    2024 arXiv

  2. [10]

    3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds,

    D. Cai, L. Zhao, J. Zhang, L. Sheng, and D. Xu, “3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds,” in Proc. Computer Vision and Pattern Recognition , 2022, pp. 16 464–16 473

    2022

  3. [11]

    Sceneverse: Scaling 3d vision-language learning for grounded scene understanding,

    B. Jia, Y . Chen, H. Yu, Y . Wang, X. Niu, T. Liu, Q. Li, and S. Huang, “Sceneverse: Scaling 3d vision-language learning for grounded scene understanding,” in European Conference on Computer Vision. Springer, 2024, pp. 289–310

    2024

  4. [12]

    Sgrec3d: Self-supervised 3d scene graph learning via object-level scene reconstruction,

    S. Koch, P. Hermosilla, N. Vaskevicius, M. Colosi, and T. Ropinski, “Sgrec3d: Self-supervised 3d scene graph learning via object-level scene reconstruction,” 2023

    2023

  5. [13]

    Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning,

    Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappa, C. Gan, C. M. de Melo, J. B. Tenenbaum, A. Torralba, F. Shkurti, and L. Paull, “Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning,” 2023

    2023

  6. [14]

    Beyond bare queries: Open- vocabulary object grounding with 3d scene graph,

    S. Linok, T. Zemskova, S. Ladanova, R. Titkov, D. Yudin, M. Monastyrny, and A. Valenkov, “Beyond bare queries: Open- vocabulary object grounding with 3d scene graph,” 2024

    2024

  7. [15]

    Sai3d: Segment any instance in 3d scenes,

    Y . Yin, Y . Liu, Y . Xiao, D. Cohen-Or, J. Huang, and B. Chen, “Sai3d: Segment any instance in 3d scenes,” in Proc. Computer Vision and Pattern Recognition, 2024, pp. 3292–3302

    2024

  8. [16]

    Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance,

    P. Nguyen, T. D. Ngo, E. Kalogerakis, C. Gan, A. Tran, C. Pham, and K. Nguyen, “Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance,” in Proc. Computer Vision and Pattern Recognition , 2024, pp. 4018–4028

    2024

  9. [17]

    Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring,

    Z. Yuan, X. Yan, Y . Liao, R. Zhang, S. Wang, Z. Li, and S. Cui, “Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring,” in Proc. of the IEEE/CVF International Conference on Computer Vision , 2...

    2021

  10. [18]

    Eda: Explicit text-decoupling and dense alignment for 3d visual grounding,

    Y . Wu, X. Cheng, R. Zhang, Z. Cheng, and J. Zhang, “Eda: Explicit text-decoupling and dense alignment for 3d visual grounding,” in Proc. Computer Vision and Pattern Recognition , 2023, pp. 19 231–19 242

    2023

  11. [19]

    Inst3d-lmm: Instance-aware 3d scene understanding with multi-modal instruction tuning,

    H. Yu, W. Li, S. Wang, J. Chen, and J. Zhu, “Inst3d-lmm: Instance-aware 3d scene understanding with multi-modal instruction tuning,” in Proc. Computer Vision and Pattern Recognition , 2025, pp. 14 147–14 157

    2025

  12. [20]

    Lan- guage conditioned spatial relation reasoning for 3d object grounding,

    S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev, “Lan- guage conditioned spatial relation reasoning for 3d object grounding,” Advances in neural information processing systems , vol. 35, pp. 20 522– 20 535, 2022

    2022

  13. [21]

    Liba: Language instructed multi-granularity bridge assistant for 3d visual grounding,

    Y . Wang, Y .-L. Li, W. E. ZY , and S. Wang, “Liba: Language instructed multi-granularity bridge assistant for 3d visual grounding,” in Proc. the AAAI Conference, vol. 39, no. 8, 2025, pp. 8114–8122

    2025

  14. [22]

    3d room layout estimation from a single rgb image,

    C. Yan, B. Shao, H. Zhao, R. Ning, Y . Zhang, and F. Xu, “3d room layout estimation from a single rgb image,” IEEE Transactions on Multimedia , vol. 22, no. 11, pp. 3014–3024, 2020

    2020

  15. [23]

    Beyond 3d: Generic iou for 3d object detection,

    H. Lun, K. Lu, L. Hou, S. Wang, and J. Xue, “Beyond 3d: Generic iou for 3d object detection,” IEEE Transactions on Multimedia , pp. 1–13, 2025

    2025

  16. [24]

    Vlm- grounder: A vlm agent for zero-shot 3d visual grounding,

    R. Xu, Z. Huang, T. Wang, Y . Chen, J. Pang, and D. Lin, “Vlm- grounder: A vlm agent for zero-shot 3d visual grounding,”arXiv preprint arXiv:2410.13860, 2024

    2024 arXiv

  17. [25]

    Seeground: See and ground for zero-shot open-vocabulary 3d visual grounding,

    R. Li, S. Li, L. Kong, X. Yang, and J. Liang, “Seeground: See and ground for zero-shot open-vocabulary 3d visual grounding,” in Proc. Computer Vision and Pattern Recognition , 2025

    2025

  18. [26]

    Multi-modal self- perception enhanced large language model for 3d region-of-interest captioning with limited data,

    L. Shi, S. Kan, Y . Jin, L. Zhang, and Y . Cen, “Multi-modal self- perception enhanced large language model for 3d region-of-interest captioning with limited data,” IEEE Transactions on Multimedia, vol. 27, pp. 2935–2948, 2025

    2025

  19. [27]

    Chat-scene: Bridging 3d scene and JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 10 large language models with object identifiers,

    H. Huang, Y . Chen, Z. Wang, R. Huang, R. Xu, T. Wang, L. Liu, X. Cheng, Y . Zhao, J. Pang et al. , “Chat-scene: Bridging 3d scene and JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 10 large language models with object identifiers,” Proc. the Advances in Neural Info...

    2021

  20. [28]

    Scanreason: Em- powering 3d visual grounding with reasoning capabilities,

    C. Zhu, T. Wang, W. Zhang, K. Chen, and X. Liu, “Scanreason: Em- powering 3d visual grounding with reasoning capabilities,” in European Conference on Computer Vision . Springer, 2024, pp. 151–168

    2024

  21. [29]

    3d scene graph: A structure for unified semantics, 3d space, and camera,

    I. Armeni, Z.-Y . He, J. Gwak, A. R. Zamir, M. Fischer, J. Malik, and S. Savarese, “3d scene graph: A structure for unified semantics, 3d space, and camera,” 2019

    2019

  22. [30]

    3-d scene graph: A sparse and semantic representation of physical environments for intelligent agents,

    U.-H. Kim, J.-M. Park, T.-j. Song, and J.-H. Kim, “3-d scene graph: A sparse and semantic representation of physical environments for intelligent agents,” IEEE Transactions on Cybernetics , vol. 50, no. 12, p. 4921–4933, Dec. 2020

    2020

  23. [31]

    Learning 3d semantic scene graphs from 3d indoor reconstructions,

    J. Wald, H. Dhamo, N. Navab, and F. Tombari, “Learning 3d semantic scene graphs from 3d indoor reconstructions,” in Conference on CVPR , 2020

    2020

  24. [32]

    3d scene graph generation from point clouds,

    W. Wei, P. Wei, J. Qin, Z. Liao, S. Wang, X. Cheng, M. Liu, and N. Zheng, “3d scene graph generation from point clouds,” IEEE Trans- actions on Multimedia , vol. 26, pp. 5358–5368, 2024

    2024

  25. [33]

    Kimera: from slam to spatial perception with 3d dynamic scene graphs,

    A. Rosinol, A. Violette, M. Abate, N. Hughes, Y . Chang, J. Shi, A. Gupta, and L. Carlone, “Kimera: from slam to spatial perception with 3d dynamic scene graphs,” 2021

    2021

  26. [34]

    Hydra: A real-time spatial perception system for 3d scene graph construction and optimization,

    N. Hughes, Y . Chang, and L. Carlone, “Hydra: A real-time spatial perception system for 3d scene graph construction and optimization,” 2022

    2022

  27. [35]

    Exploring hierarchical spatial layout cues for 3d point cloud based scene graph prediction,

    M. Feng, H. Hou, L. Zhang, Y . Guo, H. Yu, Y . Wang, and A. Mian, “Exploring hierarchical spatial layout cues for 3d point cloud based scene graph prediction,” IEEE Transactions on Multimedia , vol. 27, pp. 731–743, 2025

    2025

  28. [36]

    Vl-sat: Visual-linguistic semantics assisted training for 3d semantic scene graph prediction in point cloud,

    Z. Wang, B. Cheng, L. Zhao, D. Xu, Y . Tang, and L. Sheng, “Vl-sat: Visual-linguistic semantics assisted training for 3d semantic scene graph prediction in point cloud,” arXiv preprint arXiv:2303.14408 , 2023

    2023 arXiv

  29. [37]

    Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships,

    S. Koch, N. Vaskevicius, M. Colosi, P. Hermosilla, and T. Ropinski, “Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships,” in Proc. Computer Vision and Pattern Recognition , 2024, pp. 14 183–14 193

    2024

  30. [38]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems , vol. 36, 2024

    2024

  31. [39]

    Grounded sam: Assembling open-world models for diverse visual tasks,

    T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y . Chen, F. Yanet al., “Grounded sam: Assembling open-world models for diverse visual tasks,” arXiv preprint arXiv:2401.14159 , 2024

    2024 arXiv

  32. [40]

    V ocabulary-free 3d in- stance segmentation with vision and language assistant,

    G. Mei, L. Riz, Y . Wang, and F. Poiesi, “V ocabulary-free 3d in- stance segmentation with vision and language assistant,” arXiv preprint arXiv:2408.10652, 2024

    2024 arXiv

  33. [41]

    Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes,

    P. Achlioptas, A. Abdelreheem, F. Xia, M. Elhoseiny, and L. Guibas, “Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes,” in ECCV 2020, Proceedings , 2020, pp. 422–440

    2020

  34. [42]

    Scanrefer: 3d object local- ization in rgb-d scans using natural language,

    D. Z. Chen, A. X. Chang, and M. Nießner, “Scanrefer: 3d object local- ization in rgb-d scans using natural language,” in European conference on computer vision . Springer, 2020, pp. 202–221

    2020

  35. [43]

    The Replica dataset: A digital replica of indoor spaces,

    J. Straub, T. Whelan, L. Ma, Y . Chen, E. Wijmans, S. Green, and J. J. Engel, “The Replica dataset: A digital replica of indoor spaces,” arXiv preprint arXiv:1906.05797, 2019

    1906 arXiv

  36. [44]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes,

    A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in Proc. Computer Vision and Pattern Recognition , 2017

    2017

  37. [45]

    3dvg-transformer: Relation modeling for visual grounding on point clouds,

    L. Zhao, D. Cai, L. Sheng, and D. Xu, “3dvg-transformer: Relation modeling for visual grounding on point clouds,” in Proc. of International Conference on Computer Vision , 2021, pp. 2928–2937

    2021

  38. [46]

    Bottom up top down detection transformers for language grounding in images and point clouds,

    A. Jain, N. Gkanatsios, I. Mediratta, and K. Fragkiadaki, “Bottom up top down detection transformers for language grounding in images and point clouds,” in European Conference on Computer Vision . Springer, 2022, pp. 417–433

    2022

  39. [47]

    Gˆ 3-lq: Marrying hyperbolic alignment with explicit semantic-geometric modeling for 3d visual grounding,

    Y . Wang, Y . Li, and S. Wang, “Gˆ 3-lq: Marrying hyperbolic alignment with explicit semantic-geometric modeling for 3d visual grounding,” in Proc. Computer Vision and Pattern Recognition , 2024, pp. 13 917– 13 926

    2024

  40. [48]

    Multi-branch collaborative learning network for 3d visual grounding,

    Z. Qian, Y . Ma, Z. Lin, J. Ji, X. Zheng, X. Sun, and R. Ji, “Multi-branch collaborative learning network for 3d visual grounding,” in European Conference on Computer Vision . Springer, 2024, pp. 381–398

    2024

  41. [49]

    Four ways to improve verbo-visual fusion for dense 3d visual grounding,

    O. Unal, C. Sakaridis, S. Saha, and L. Van Gool, “Four ways to improve verbo-visual fusion for dense 3d visual grounding,” inEuropean Conference on Computer Vision . Springer, 2024, pp. 196–213

    2024

  42. [50]

    Towards clip-driven language-free 3d visual grounding via 2d-3d relational enhancement and consistency,

    Y . Zhang, H. Luo, and Y . Lei, “Towards clip-driven language-free 3d visual grounding via 2d-3d relational enhancement and consistency,” in Proc. Computer Vision and Pattern Recognition , 2024, pp. 13 063– 13 072

    2024

  43. [51]

    Text-guided sparse voxel pruning for efficient 3d visual grounding,

    W. Guo, X. Xu, Z. Wang, J. Feng, J. Zhou, and J. Lu, “Text-guided sparse voxel pruning for efficient 3d visual grounding,” in Proc. Com- puter Vision and Pattern Recognition , 2025, pp. 3666–3675

    2025

  44. [52]

    Distilling coarse-to-fine semantic matching knowledge for weakly supervised 3d visual grounding,

    Z. Wang, H. Huang, Y . Zhao, L. Li, X. Cheng, Y . Zhu, A. Yin, and Z. Zhao, “Distilling coarse-to-fine semantic matching knowledge for weakly supervised 3d visual grounding,” in Proc. of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 2662–2671

    2023

  45. [53]

    Lerf: Language embedded radiance fields,

    J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik, “Lerf: Language embedded radiance fields,” in Proc. of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 19 729–19 739

    2023

  46. [54]

    Visual programming for zero-shot open-vocabulary 3d visual grounding,

    Z. Yuan, J. Ren, C.-M. Feng, H. Zhao, S. Cui, and Z. Li, “Visual programming for zero-shot open-vocabulary 3d visual grounding,” in Proc. Computer Vision and Pattern Recognition , 2024, pp. 20 623– 20 633

    2024

  47. [55]

    Free-form description guided 3d visual graph network for object grounding in point cloud,

    M. Feng, Z. Li, Q. Li, L. Zhang, X. Zhang, G. Zhu, H. Zhang, Y . Wang, and A. Mian, “Free-form description guided 3d visual graph network for object grounding in point cloud,” in Proc. of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 3722–3731

    2021

  48. [56]

    Look around and refer: 2d synthetic semantics knowledge distillation for 3d visual grounding,

    E. Bakr, Y . Alsaedy, and M. Elhoseiny, “Look around and refer: 2d synthetic semantics knowledge distillation for 3d visual grounding,” Advances in neural information processing systems , vol. 35, pp. 37 146– 37 158, 2022

    2022

  49. [57]

    Sat: 2d semantics assisted training for 3d visual grounding,

    Z. Yang, S. Zhang, L. Wang, and J. Luo, “Sat: 2d semantics assisted training for 3d visual grounding,” in Proc. of the IEEE/CVF Interna- tional Conference on Computer Vision , 2021, pp. 1856–1866

    2021

  50. [58]

    3d-sps: Single-stage 3d visual grounding via referred point progressive selection,

    J. Luo, J. Fu, X. Kong, C. Gao, H. Ren, H. Shen, H. Xia, and S. Liu, “3d-sps: Single-stage 3d visual grounding via referred point progressive selection,” in Proc. Computer Vision and Pattern Recognition , 2022, pp. 16 454–16 463

    2022

  51. [59]

    Multi-view transformer for 3d visual grounding,

    S. Huang, Y . Chen, J. Jia, and L. Wang, “Multi-view transformer for 3d visual grounding,” in Proc. Computer Vision and Pattern Recognition , 2022, pp. 15 524–15 533

    2022

  52. [60]

    Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation,

    S. Y . Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song, “Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation,” in Proc. Computer Vision and Pattern Recognition , 2023, pp. 23 171–23 181

    2023

  53. [61]

    Sort3d: Spatial object-centric reasoning toolbox for zero- shot 3d grounding using large language models,

    N. Zantout, H. Zhang, P. Kachana, J. Qiu, J. Zhang, and W. Wang, “Sort3d: Spatial object-centric reasoning toolbox for zero- shot 3d grounding using large language models,” arXiv preprint arXiv:2504.18684, 2025

    2025 arXiv

  54. [62]

    Solving zero-shot 3d visual grounding as constraint satisfaction problems,

    Q. Yuan, J. Zhang, K. Li, and R. Stiefelhagen, “Solving zero-shot 3d visual grounding as constraint satisfaction problems,” arXiv preprint arXiv:2411.14594, 2024

    2024 arXiv

  55. [63]

    Exploiting edge-oriented reasoning for 3d point-based scene graph analysis,

    C. Zhang, J. Yu, Y . Song, and W. Cai, “Exploiting edge-oriented reasoning for 3d point-based scene graph analysis,” 2021

    2021

  56. [64]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

  57. [65]

    Scenegraph- fusion: Incremental 3d scene graph prediction from rgb-d sequences,

    S.-C. Wu, J. Wald, K. Tateno, N. Navab, and F. Tombari, “Scenegraph- fusion: Incremental 3d scene graph prediction from rgb-d sequences,” 2021

    2021

  58. [66]

    Language-driven semantic segmentation,

    B. Li, K. Q. Weinberger, S. Belongie, V . Koltun, and R. Ranftl, “Language-driven semantic segmentation,” arXiv preprint arXiv:2201.03546, 2022

    2022 arXiv

  59. [67]

    Scaling open-vocabulary image segmentation with image-level labels,

    G. Ghiasi, X. Gu, Y . Cui, and T.-Y . Lin, “Scaling open-vocabulary image segmentation with image-level labels,” 2022

    2022

  60. [68]

    Open-fusion: Real-time open-vocabulary 3d mapping and queryable scene representation,

    K. Yamazaki, T. Hanyu, K. V o, T. Pham, M. Tran, G. Doretto, A. Nguyen, and N. Le, “Open-fusion: Real-time open-vocabulary 3d mapping and queryable scene representation,” in 2024 IEEE Interna- tional Conference on Robotics and Automation , 2024, pp. 9411–9417

    2024

  61. [69]

    Masked point-entity contrast for open-vocabulary 3d scene understanding,

    Y . Wang, B. Jia, Z. Zhu, and S. Huang, “Masked point-entity contrast for open-vocabulary 3d scene understanding,” in Proc. Computer Vision and Pattern Recognition , 2025, pp. 14 125–14 136

    2025

  62. [70]

    Per-pixel classification is not all you need for semantic segmentation,

    B. Cheng, A. G. Schwing, and A. Kirillov, “Per-pixel classification is not all you need for semantic segmentation,” 2021

    2021

  63. [71]

    Conceptfusion: Open-set multimodal 3d mapping,

    K. M. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, T. Chen, A. Maalouf, S. Li, G. Iyer, S. Saryazdi, N. Keetha et al., “Conceptfusion: Open-set multimodal 3d mapping,” arXiv preprint arXiv:2302.07241 , 2023

    2023 arXiv

  64. [72]

    Openmask3d: Open-vocabulary 3d instance segmenta- tion,

    A. Takmaz, E. Fedele, R. W. Sumner, M. Pollefeys, F. Tombari, and F. Engelmann, “Openmask3d: Open-vocabulary 3d instance segmenta- tion,” arXiv preprint arXiv:2306.13631 , 2023

    2023 arXiv

  65. [73]

    Hier- archical open-vocabulary 3d scene graphs for language-grounded robot navigation,

    A. Werby, C. Huang, M. B ¨uchner, A. Valada, and W. Burgard, “Hier- archical open-vocabulary 3d scene graphs for language-grounded robot navigation,” in ICRA 2024 , 2024

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.