Pith. sign in

REVIEW 3 major objections 4 minor 62 references

Unleashing the Multi-View Fusion Potential: Noise Correction in VLM for Open-Vocabulary 3D Scene Understanding

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A training-free pipeline that corrects noise in 2D vision-language features sets a new state of the art for open-vocabulary 3D semantic segmentation, reporting 14.7% mIoU on ScanNet200 and 16.2% on Matterport160.

desk verdict Training-free 2D fusion pipeline likely beats trained 3D nets on large-vocab benchmarks, but the zero-shot SOTA claim is compromised by test-set-tuned hyperparameters. read the letter →

arxiv 2506.22817 v1 pith:7VV2JVE4 submitted 2025-06-28 cs.CV

Xingyilang Yin , Jiale Wang , Xi Yang , Mutian Xu , Xu Gu , Nannan Wang This is my paper
classification cs.CV
keywords open-vocabulary3Dsceneunderstandingmulti-viewfusionvision-languagemodelsnoisecorrectionzero-shotsemanticsegmentationfoundationimagecaptioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the ceiling on open-vocabulary 3D scene understanding, recognizing object classes that never appeared in any 3D training set, is set by noise in 2D vision-language features rather than by the lack of 3D training data. It proposes MVOV3D, a pipeline that performs no 3D training at all: it fuses multi-view 2D features onto 3D points, but first corrects those features by replacing unreliable pixel-level predictions with region-level CLIP image features, adding text features from high-confidence caption-model tags, and finally pooling each point's features within geometrically consistent superpoints. Against strong trained baselines, the paper reports 14.7% mIoU on ScanNet200, 16.2% on Matterport160, 23.9% on Matterport80, and 20.1% on Replica, all without training a 3D network. A sympathetic reader would take the contribution to be a demonstration that 2D foundation-model features, once denoised, transfer to 3D better than learned 3D features do on large-vocabulary benchmarks.

What carries the argument

The load-bearing object is the corrected per-pixel 2D feature map $f^{2D'}_i$, computed at each pixel as the average of the original VLM feature, the region-level CLIP vision feature $f^{2D}_{R_{i,s}}$ propagated from a VFM mask, and the CLIP text feature $f^{1D}_{T_{i,s,select}}$ flood-filled from a caption-model tag, with an indicator function $I(\cdot)$ in the denominator so only available terms count. This map is fused to 3D points by the standard point-pixel alignment of OpenScene, and then geometry enters through superpoints $U_q$ produced by a graph-cut over normal similarity: the final per-point feature $F^{3D''}(p_k)$ is the average of the initially fused features over all points in $p_k$'s superpoint. Each of the three corrections targets a specific failure mode identified in the paper: region isolation fixes co-occurrence confusion, confident text supplies vocabulary beyond what the VLM can name, and superpoint pooling suppresses residual fusion noise.

What would settle it

A concrete check: run MVOV3D on ScanNet200 and Matterport160 with $\delta$ chosen on a held-out validation split rather than the test labels, with the VFM and ICM components fixed to the reported ones, and compare mIoU to the reported 14.7% and 16.2%. If the gap over the leading trained baselines (8.2% RegionPLC, 8.7% OV3D, 9.8% DMA on ScanNet200) narrows to near zero, the claim that training-free corrected fusion is intrinsically superior would collapse.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a training-free combination of off-the-shelf foundation models, orchestrated to cancel each other's noise, can beat 3D networks that are explicitly trained for open-vocabulary segmentation. The pipeline starts from pixel-aligned VLM features (OpenSeg), then corrects each image from three directions: high-confidence masks from vision foundation models isolate objects so the CLIP vision encoder can give clean region features; image caption models supply text tags whose CLIP text embeddings are accepted only above a similarity threshold $\delta$; and after back-projecting these corrected 2D features to points, the features are averaged within superpoints built from normal similarity. The paper reports that this corrected fusion reaches a new state of the art on ScanNet200 and Matterport160 and outperforms current leading trained 3D networks by a significant margin, with the biggest gains on rare, long-tail categories.

Load-bearing premise

The load-bearing premise is that the confidence cutoff and the off-the-shelf component choices keep working on new scenes without retuning, and that those components make fewer mistakes than the vision-language noise they are correcting.

Editorial extensions

If this is right

  • On large-vocabulary benchmarks, ensembling distilled 3D features with 2D multi-view features (OpenScene-2D3D) is worse than using the 2D features alone, which contradicts the pattern reported on 20-class and 21-class benchmarks.
  • A 3D network distilled from the corrected features inherits part of the gain, improving OpenScene-3D by +0.6 mIoU on ScanNet200 and +1.0 on Matterport160, so the correction is useful even when only a 3D model can be deployed.
  • The long-tail breakdown shows the method's advantage is concentrated in rare categories: tail-class mIoU is 18.5 on Replica and 10.8 on Matterport160, whereas distilled 3D features nearly collapse to 1.1 on Matterport160 tail classes.
  • Because no training is involved, swapping in stronger versions of the VFM, ICM, or VLM components should translate directly into higher 3D open-vocabulary accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reported margins survive a protocol where the confidence threshold $\delta$ and the component choices are fixed without any test-set feedback, the result implies that the open-vocabulary 3D bottleneck is currently VLM noise rather than 3D data scarcity, so 2D foundation-model progress would transfer nearly one-for-one to 3D.
  • The ablation order suggests the 1D text correction gives the largest single gain, so the breadth of image-tagging vocabularies may be the binding constraint for recognizing rare classes in 3D.
  • A testable extension would be to replace the global threshold $\delta$ with an adaptive per-region criterion, for example accepting a text tag only when multiple views agree on it; this could reduce mislabeling without retraining.
  • The superpoint pooling assumes geometrically similar points belong to the same class, which may oversmooth scenes where visually similar objects are semantically distinct; combining the geometric pooling with mask boundaries from the VFM could preserve the gain while reducing that risk.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes MVOV3D, a training-free pipeline for open-vocabulary 3D semantic segmentation. Starting from per-pixel OpenSeg features on multi-view RGB-D frames, it refines the features in three stages: (i) it replaces noisy pixel features with CLIP region features computed from high-confidence SAM/Grounding-DINO masks; (ii) it fuses CLIP text embeddings of confident tags produced by image-caption models such as RAM++; and (iii) it smooths the fused point features within 3D superpoints obtained by graph-cut on geometric normals. The paper reports state-of-the-art results on ScanNet200 (14.7 mIoU), Matterport160 (16.2), Matterport80 (23.9), and Replica (20.1), and it extends the features to open-vocabulary 3D instance segmentation by feeding them into SOLE, achieving 20.7 AP on ScanNet200. The central claim is that a training-free 2D multi-view fusion approach with noise correction outperforms trained 3D networks on large-vocabulary open-vocabulary benchmarks.

Significance. If the reported results are reproducible under a genuinely fixed zero-shot configuration, the contribution is significant: it provides a coherent argument that training 3D networks on limited 3D data restricts open-vocabulary generalization, and it demonstrates that a modular, training-free combination of 2D foundation models plus geometric post-processing can beat trained 3D models on large-vocabulary benchmarks. The paper's strengths include a wide set of benchmarks, component ablations, long-tail category analysis, a distillation extension, and an explicit parameter-sensitivity study in the supplementary material. The main caveat is that the evaluation protocol currently selects the confidence threshold and the auxiliary model choices on the test benchmarks used for the headline numbers, and the tag-selection loop shares the same CLIP text encoder between training-free refinement and inference. These issues are fixable with additional control experiments, but they are load-bearing for the zero-shot SOTA claim. I therefore cannot recommend acceptance in the current form.

major comments (3)
  1. [§4.2, Table 2; Supp. Fig. 8; Table 5] The headline zero-shot SOTA claim is undermined by test-set-based selection of free hyperparameters. Supp. Fig. 8 reports that the confidence threshold δ is set to 0.24 for ScanNet200 and Matterport160 but to 0.26 for Replica, and Table 5 selects the SAM & GD combination and the RAM++ caption model by comparing Matterport160 test mIoU. These are the same benchmarks whose numbers appear in Table 2. The per-dataset tuning is visible in the Replica curve, which peaks at a different δ from the other two datasets; if a single δ were fixed across all benchmarks, the Replica number would decline from 20.1 toward roughly 18.2, close to OpenScene-2D's 18.0, and the ScanNet200 and Matterport160 margins could also shrink. To support the central claim, the authors need to report results with a configuration fixed a priori, or at least show that the reported conclusions are unchanged when the same δ and the same component choices are used on every dataset. Without this, the reported significant margin over prior methods is partly an artifact of test-set selection.
  2. [§3.2, Eqs. (2)-(4)] The text-tag selection and the inference share the same CLIP text encoder, creating a self-reinforcing loop that can inflate the reported gains. In Eq. (2), the tag T_{i,s,select} is chosen as the text proposal with the highest CLIP cosine similarity to the CLIP region feature; Eqs. (3)-(4) then flood-fill the region with that tag's embedding and average it into the point feature. At inference, the same CLIP text encoder is used to compare the resulting point feature against the benchmark label names. Selection and evaluation are therefore not independent: a tag that is cosmetically similar to the region in CLIP space may become more similar to the region at inference by construction. The authors should provide a control in which tag selection is done with an independently trained text encoder (or with text features held out), and should report performance separately on categories that never appear in the selected tag vocabulary. Without such a control, the noise-correction story is confounded with self-confirmation.
  3. [§4.3, Table 4] Several ablation differences are within 0.2-0.3 mIoU (e.g., w/o 2D region features: 16.0 vs 16.2 on Matterport160; w/o 3D priors: 15.9 vs 16.2), and no error bars or significance tests are reported. Since the pipeline is deterministic, the relevant variability is across scenes; reporting per-scene standard deviations or paired bootstrap confidence intervals would clarify whether these components actually contribute. The same applies to the component choices in Table 5, where SAM & GD and RAM++ are selected on margins of only 0.2-0.3 mIoU. This issue is secondary to the headline SOTA claim, but it is load-bearing for the paper's decomposition of the gains into 1D, 2D, and 3D corrections.
minor comments (4)
  1. [Abstract and §1] The phrase 'limited amount of 3D data upbound training strong open-vocabulary 3d models' is grammatically incomplete; 'upbound' appears to be a typo for 'limits' or 'bounds'.
  2. [Supp. Fig. 8 and Supp. Sec. D] The parameter-analysis figure is hard to read because the x-axis labels and the mapping of the three series are not explained in the caption, and the sentence beginning 'as illustrated in Fig. 8, when δ is set to 0.24... ' is not a complete sentence.
  3. [§3.2, Eq. (1)] The notation for the image region features is inconsistent: f_{R_{i,s}}^{2D} is introduced as an element of R^{H×W×C}, but Eq. (1) then writes f_{R_{i,s}}^{2D}(h,w,c) as a scalar; the dimensional bookkeeping should be clarified.
  4. [§4.2 and §4.3] No code or pre-trained feature release is mentioned; given the modularity of the pipeline, releasing the evaluation scripts would materially improve reproducibility and make the per-dataset configuration choices auditable.

Circularity Check

1 steps flagged · score 4.0 of 10

The core multi-view fusion pipeline is not circular, but the reported zero-shot SOTA is partially in-sample because the confidence threshold and component choices are selected on the test benchmarks.

  1. fitted input called prediction [Supplementary D (Parameter Analysis, Fig. 8) and Section 4.3 (Table 5)]
    "As illustrated in Fig. 8, when δ is set to 0.24 on Matterport3D and ScanNet200, and 0.26 on Replica, leading to better performance. ... we experiment with various VFM to obtain image regions ... and different image caption models ... on Matterport160."

    The paper reports zero-shot mIoU on Replica, Matterport160, and ScanNet200 as its headline predictions, but the confidence threshold δ is tuned separately on exactly these test splits (0.26 on Replica, 0.24 on the others) and the component choices (SAM & GD, RAM++) are validated on the Matterport160 test split. The reported scores are therefore not out-of-sample predictions: the test labels were used to select the free parameters, so the 'new record' numbers are partly an in-sample fit. This does not make the feature-averaging equations themselves circular, but it undermines the central zero-shot SOTA claim as a fixed-configuration evaluation.

full rationale

The method's derivation chain (Eqs. 1-5) is a training-free feature-averaging pipeline: it combines OpenSeg pixel features, CLIP region features from VFM masks, CLIP text features from ICM proposals, and a superpoint pooling prior. None of these equations is defined in terms of the benchmark labels, and no external mathematical result is imported from the authors' prior work. The self-citations [50, 53, 55] appear only in the related-work survey and are not load-bearing for the central claim. The main circularity concern is the evaluation protocol: δ and the auxiliary model choices are selected using the test benchmarks themselves (Supp. Fig. 8 and Table 5), so the reported 'state-of-the-art' mIoU values are partially the result of test-set selection rather than true predictions. The paper also exhibits a self-reinforcing tendency in which the same CLIP text space selects the caption tag (Eq. 2), injects that tag's embedding into the point feature (Eq. 4), and then scores benchmark queries, but because the selected tag comes from an external captioning model rather than from the benchmark labels, this is a methodological property rather than a formal equivalence. Overall, the pipeline has independent content, but the headline zero-shot claim is moderately compromised by in-sample hyperparameter tuning, yielding a score of 4.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method is an engineering pipeline over frozen foundation models and introduces no new entities. It depends on two domain assumptions about the reliability of external segmentation and captioning models, and on an evaluation-protocol premise that test-set-based hyperparameter selection does not invalidate the zero-shot claim.

free parameters (4)
  • confidence threshold delta = 0.24 (ScanNet200, Matterport160); 0.26 (Replica)
    Selected per dataset based on test performance in Supplementary Fig. 8, so the reported mIoU values are partly fit to the test benchmarks.
  • VFM and ICM component choices = SAM + Grounding-DINO; RAM++
    Model selection is reported on the Matterport160 test set in Table 5, another test-set-dependent choice.
  • superpoint graph-cut parameters = not reported
    The granularity and normal-similarity thresholds for generating superpoints are not specified, but they directly affect Eq. (5).
  • mask confidence thresholds for SAM and Grounding-DINO = not reported
    The confidence levels used to produce high-confidence masks are not stated, affecting the quality of image regions.
assumptions (3)
  • domain assumption Points with similar 3D geometry, measured by normal similarity, belong to the same semantic category.
    Used in Sec. 3.3 Eq. (5) to average features within superpoints; this can corrupt features at object boundaries or in geometrically similar but semantically different regions.
  • domain assumption VFM masks isolate individual entities with high confidence, and CLIP region features are more accurate than pixel-aligned VLM features.
    Core to Sec. 3.2; supported only by qualitative examples and a small ablation, not by systematic analysis.
  • domain assumption The text selected by maximum CLIP cosine similarity above threshold delta is the correct label for the region.
    Eq. (2) assumes the argmax text above threshold is accurate; the authors acknowledge wrong selections such as 'television' for a mirror region.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unleashing the Multi-View Fusion Potential: Noise Correction in VLM for Open-Vocabulary 3D Scene Understanding." pith.science (2026). https://pith.science/paper/7VV2JVE4

@misc{pith2026250622817,
  author       = {Pith},
  title        = {Pith review of: Unleashing the Multi-View Fusion Potential: Noise Correction in VLM for Open-Vocabulary 3D Scene Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7VV2JVE4}},
  note         = {Machine review of arXiv:2506.22817}
}
read the original abstract

Recent open-vocabulary 3D scene understanding approaches mainly focus on training 3D networks through contrastive learning with point-text pairs or by distilling 2D features into 3D models via point-pixel alignment. While these methods show considerable performance in benchmarks with limited vocabularies, they struggle to handle diverse object categories as the limited amount of 3D data upbound training strong open-vocabulary 3d models. We observe that 2D multi-view fusion methods take precedence in understanding diverse concepts in 3D scenes. However, inherent noises in vision-language models lead multi-view fusion to sub-optimal performance. To this end, we introduce MVOV3D, a novel approach aimed at unleashing the potential of 2D multi-view fusion for open-vocabulary 3D scene understanding. We focus on reducing the inherent noises without training, thereby preserving the generalizability while enhancing open-world capabilities. Specifically, MVOV3D improves multi-view 2D features by leveraging precise region-level image features and text features encoded by CLIP encoders and incorporates 3D geometric priors to optimize multi-view fusion. Extensive experiments on various datasets demonstrate the effectiveness of our method. Notably, our MVOV3D achieves a new record with 14.7% mIoU on ScanNet200 and 16.2% mIoU on Matterport160 for challenge open-vocabulary semantic segmentation, outperforming current leading trained 3D networks by a significant margin.

Figures

Figures reproduced from arXiv: 2506.22817 by the authors.

Figure 1
Figure 1. Top: Performance comparison on ScanNet200 and Matterport160 for challenging open-vocabulary 3d semantic seg￾mentation. 2D multi-view fusion methods take precedence over 3D-2D distillation frameworks and 3D-1D contrastive approaches. The proposed MVOV3D achieves a new record with 14.7% mIoU on ScanNet200 and 16.2% on Matterport160. Down: Illustration of different open-vocabulary 3D scene understanding methods. Founda… view at source ↗
Figure 2
Figure 2. Overview of our MVOV3D framework. MVOV3D takes a 3D point cloud [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of 3D predictions and corresponding 2D predictions generated by VLM. Inaccurate 3D predictions arise from noise [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Illustration under different occlusion threshold on [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: An example of correcting the noise in 2D multi-view [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Qualitative results of zero-shot 3D semantic segmentation from our model and OpenScene. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Qualitative results of zero-shot 3D semantic segmenta [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Parameter experiments about confident threshold [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

62 extracted references · 51 canonical work pages

  1. [1]

    Open-yolo 3d: Towards fast and accurate open-vocabulary 3d instance segmentation

    Mohamed El Amine Boudjoghra, Angela Dai, Jean Lahoud, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, and Fahad Shahbaz Khan. Open-yolo 3d: Towards fast and accurate open-vocabulary 3d instance segmentation. arXiv preprint arXiv:2406.02548, 2024. 3

    arXiv 2024

  2. [2]

    Matterport3d: Learning from rgb-d data in indoor environments

    Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Hal- ber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In International Conference on 3D Vision, pages 667–676, 2017. 2, 6, 7

    work page 2017

  3. [3]

    Clip2scene: Towards label-efficient 3d scene understanding by clip

    Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wen- ping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 7020–7030, 2023. 3

    2023

  4. [4]

    Yolo-world: Real-time open-vocabulary object detection

    Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xing- gang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16901–16911, 2024. 1

    work page 2024

  5. [5]

    4d spatio-temporal convnets: Minkowski convolutional neu- ral networks

    Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neu- ral networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3075– 3084, 2019. 2, 6, 7

    work page 2019

  6. [6]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5828–5839, 2017. 2, 7

    work page 2017

  7. [7]

    Pla: Language-driven open- vocabulary 3d scene understanding

    Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Pla: Language-driven open- vocabulary 3d scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7010–7019, 2023. 2, 3, 7

    work page 2023

  8. [8]

    A density-based algorithm for discovering clusters in large spatial databases with noise

    Martin Ester, Hans-Peter Kriegel, J ¨org Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In ACM SIGKDD Con- ference on Knowledge Discovery and Data Mining , pages 226–231, 1996. 7

    work page 1996

Show all 62 references
  1. [9]

    Efficient graph-based image segmentation

    Pedro F Felzenszwalb and Daniel P Huttenlocher. Efficient graph-based image segmentation. International Journal of Computer Vision, pages 167–181, 2004. 6

    2004

  2. [10]

    Scal- ing open-vocabulary image segmentation with image-level labels

    Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scal- ing open-vocabulary image segmentation with image-level labels. In European Conference on Computer Vision, pages 540–557, 2022. 1, 2, 3, 4, 6

    2022

  3. [11]

    Sam-guided graph cut for 3d instance segmentation

    Haoyu Guo, He Zhu, Sida Peng, Yuang Wang, Yujun Shen, Ruizhen Hu, and Xiaowei Zhou. Sam-guided graph cut for 3d instance segmentation. In European Conference on Com- puter Vision, pages 234–251, 2024. 3

    2024

  4. [12]

    spacy 2: Natural lan- guage understanding with bloom embeddings, convolutional neural networks and incremental parsing

    Matthew Honnibal and Ines Montani. spacy 2: Natural lan- guage understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear, pages 411–420, 2017. 5

    2017

  5. [13]

    Open-set image tagging with multi-grained text su- pervision

    Xinyu Huang, Yi-Jie Huang, Youcai Zhang, Weiwei Tian, Rui Feng, Yuejie Zhang, Yanchun Xie, Yaqian Li, and Lei Zhang. Open-set image tagging with multi-grained text su- pervision. arXiv preprint arXiv:2310.15200, 2023. 2, 3, 4, 5, 6, 8

    2023 arXiv

  6. [14]

    Odin: A single model for 2d and 3d segmentation

    Ayush Jain, Pushkal Katara, Nikolaos Gkanatsios, Adam W Harley, Gabriel Sarch, Kriti Aggarwal, Vishrav Chaudhary, and Katerina Fragkiadaki. Odin: A single model for 2d and 3d segmentation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, ...

    2024

  7. [15]

    Con- ceptfusion: Open-set multimodal 3d mapping

    Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, et al. Con- ceptfusion: Open-set multimodal 3d mapping. In Robotics Science and Systems, 2023. 3

    2023

  8. [16]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International Conference on Machine Learning, pages 4904–491...

    2021

  9. [17]

    Pointgroup: Dual-set point group- ing for 3d instance segmentation

    Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi- Wing Fu, and Jiaya Jia. Pointgroup: Dual-set point group- ing for 3d instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4867–4876, 2020. 2

    2020

  10. [18]

    Open-vocabulary 3d semantic segmentation with foundation models

    Li Jiang, Shaoshuai Shi, and Bernt Schiele. Open-vocabulary 3d semantic segmentation with foundation models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21284–21294, 2024. 2, 3, 6, 7

    2024

  11. [19]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 202...

    2023

  12. [20]

    Seg- ment any 3d object with language

    Seungjun Lee, Yuyang Zhao, and Gim Hee Lee. Seg- ment any 3d object with language. arXiv preprint arXiv:2404.02157, 2024. 7

    2024 arXiv

  13. [21]

    Language-driven semantic seg- mentation

    Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic seg- mentation. In International Conference on Learning Rep- resentations, 2022. 1, 2, 3, 4

    2022

  14. [22]

    Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. In In- ternational Conference on Machine Learning, pages 12888– 12900, 2022. 2, 3, 4, 5, 8

    2022

  15. [23]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page...

    2022

  16. [24]

    Decap: Decoding clip latents for zero-shot captioning via text-only 2 training

    Wei Li, Linchao Zhu, Longyin Wen, and Yi Yang. Decap: Decoding clip latents for zero-shot captioning via text-only 2 training. In International Conference on Learning Represen- tations, 2023. 2, 3, 4, 5, 8

    2023

  17. [25]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pag...

    2023

  18. [26]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 2, 3, 4, 5, 6, 8

    2023 arXiv

  19. [27]

    Ovir-3d: Open-vocabulary 3d in- stance retrieval without training on 3d data

    Shiyang Lu, Haonan Chang, Eric Pu Jing, Abdeslam Boular- ias, and Kostas Bekris. Ovir-3d: Open-vocabulary 3d in- stance retrieval without training on 3d data. In Conference on Robot Learning, pages 1610–1620, 2023. 3, 7

    2023

  20. [28]

    An end-to- end transformer model for 3d object detection

    Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to- end transformer model for 3d object detection. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 2906–2917, 2021. 1

    2021

  21. [29]

    Isbnet: a 3d point cloud instance segmentation network with instance- aware sampling and box-aware dynamic convolution

    Tuan Duc Ngo, Binh-Son Hua, and Khoi Nguyen. Isbnet: a 3d point cloud instance segmentation network with instance- aware sampling and box-aware dynamic convolution. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13550–13559, 2023. 2, 7

    2023

  22. [30]

    Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance

    Phuc Nguyen, Tuan Duc Ngo, Evangelos Kalogerakis, Chuang Gan, Anh Tran, Cuong Pham, and Khoi Nguyen. Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 4018–402...

    2024

  23. [31]

    V oxel cloud connectivity segmentation- supervoxels for point clouds

    Jeremie Papon, Alexey Abramov, Markus Schoeler, and Flo- rentin Worgotter. V oxel cloud connectivity segmentation- supervoxels for point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2027–2034, 2013. 6

    2027

  24. [32]

    Openscene: 3d scene understanding with open vocabularies

    Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 815–824, 2023. 2, 3, 4, 5, 6, 7

    2023

  25. [33]

    Kosmos-2: Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023. 2

    2023 arXiv

  26. [34]

    Pointnet++: Deep hierarchical feature learning on point sets in a metric space

    Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Informa- tion Processing Systems, 2017. 1, 2

    2017

  27. [35]

    Pointnext: Revisiting pointnet++ with improved training and scaling strategies

    Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In Advances in Neural Infor- mation Processing Systems, pages 23192–23204, 2022. 2

    2022

  28. [36]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International Conference on Machine Learning...

    2021

  29. [37]

    Language- grounded indoor 3d semantic segmentation in the wild

    David Rozenberszki, Or Litany, and Angela Dai. Language- grounded indoor 3d semantic segmentation in the wild. In European Conference on Computer Vision, pages 125–141,

  30. [38]

    Dense multimodal alignment for open-vocabulary 3d scene understanding

    Li Ruihuang, Zhang Zhengqiang, He Chenhang, Ma Zhiyuan, Patel Vishal M., and Zhang Lei. Dense multimodal alignment for open-vocabulary 3d scene understanding. In European Conference on Computer Vision, 2024. 2, 3, 6, 7

    2024

  31. [39]

    Mask3d: Mask trans- former for 3d semantic instance segmentation

    Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask trans- former for 3d semantic instance segmentation. In 2023 IEEE International Conference on Robotics and Automation, pages 8216–8223, 2023. 2, 7

    2023

  32. [40]

    Pointr- cnn: 3d object proposal generation and detection from point cloud

    Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointr- cnn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 770–779, 2019. 2

    2019

  33. [41]

    The replica dataset: A digital replica of indoor spaces

    Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797,

    1906 arXiv

  34. [42]

    Open- mask3d: open-vocabulary 3d instance segmentation

    Ayc ¸a Takmaz, Elisabetta Fedele, Robert W Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann. Open- mask3d: open-vocabulary 3d instance segmentation. In Neu- ral Information Processing Systems , pages 68367–68390,

  35. [43]

    Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learn- i...

    2022

  36. [44]

    Open vocabulary 3d scene under- standing via geometry guided self-distillation

    Pengfei Wang, Yuxi Wang, Shuai Li, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. Open vocabulary 3d scene under- standing via geometry guided self-distillation. In European Conference on Computer Vision, 2024. 2, 3, 4, 6, 7

    2024

  37. [45]

    Detr3d: 3d object detection from multi-view images via 3d-to-2d queries

    Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning, pages 180–191,

  38. [46]

    Uni3detr: Unified 3d detection trans- former

    Zhenyu Wang, Ya-Li Li, Xi Chen, Hengshuang Zhao, and Shengjin Wang. Uni3detr: Unified 3d detection trans- former. In Advances in Neural Information Processing Sys- tems, 2023. 1, 2

    2023

  39. [47]

    Point transformer v3: Simpler faster stronger

    Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xi- hui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler faster stronger. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4840–4851, 2024. ...

    2024

  40. [48]

    Open-vocabulary panop- 3 tic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiao- long Wang, and Shalini De Mello. Open-vocabulary panop- 3 tic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 2955–2966, 2023. 2

    2023

  41. [49]

    Paconv: Position adaptive convolution with dy- namic kernel assembling on point clouds

    Mutian Xu, Runyu Ding, Hengshuang Zhao, and Xiao- juan Qi. Paconv: Position adaptive convolution with dy- namic kernel assembling on point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3173–3182, 2021. 2

    2021

  42. [50]

    Sampro3d: Locating sam prompts in 3d for zero-shot scene segmentation

    Mutian Xu, Xingyilang Yin, Lingteng Qiu, Yang Liu, Xin Tong, and Xiaoguang Han. Sampro3d: Locating sam prompts in 3d for zero-shot scene segmentation. arXiv preprint arXiv:2311.17707, 2023. 3

    2023 arXiv

  43. [51]

    A unified framework for 3d scene understanding

    Wei Xu, Chunsheng Shi, Sifan Tu, Xin Zhou, Dingkang Liang, and Xiang Bai. A unified framework for 3d scene understanding. In Advances in Neural Information Process- ing Systems, pages 59468–59490, 2024. 7

    2024

  44. [52]

    Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding

    Jihan Yang, Runyu Ding, Weipeng Deng, Zhe Wang, and Xi- aojuan Qi. Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19823–19832, 2024. 2, 3, 6, 7

    2024

  45. [53]

    Sa3dip: Segment any 3d instance with potential 3d priors

    Xi Yang, Xu Gu, Xingyilang Yin, and Xinbo Gao. Sa3dip: Segment any 3d instance with potential 3d priors. In Ad- vances in Neural Information Processing Systems , pages 102568–102586, 2024. 3

    2024

  46. [54]

    Sam3d: Segment anything in 3d scenes.arXiv preprint arXiv:2306.03908, 2023

    Yunhan Yang, Xiaoyang Wu, Tong He, Hengshuang Zhao, and Xihui Liu. Sam3d: Segment anything in 3d scenes.arXiv preprint arXiv:2306.03908, 2023. 3

    2023 arXiv

  47. [55]

    Point deformable network with enhanced nor- mal embedding for point cloud analysis

    Xingyilang Yin, Xi Yang, Liangchen Liu, Nannan Wang, and Xinbo Gao. Point deformable network with enhanced nor- mal embedding for point cloud analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6738– 6746, 2024. 2

    2024

  48. [56]

    Sai3d: Segment any instance in 3d scenes

    Yingda Yin, Yuzheng Liu, Yang Xiao, Daniel Cohen-Or, Jingwei Huang, and Baoquan Chen. Sai3d: Segment any instance in 3d scenes. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 3292–3302, 2024. 3

    2024

  49. [57]

    Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang- Chieh Chen. Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip. In Neural Information Processing Systems, 2023. 1, 2, 3

    2023

  50. [58]

    Recognize anything: A strong image tagging model

    Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 17...

    2024

  51. [59]

    Point transformer

    Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16259–16268, 2021. 1, 2

    2021

  52. [60]

    Se- ssd: Self-ensembling single-stage object detector from point cloud

    Wu Zheng, Weiliang Tang, Li Jiang, and Chi-Wing Fu. Se- ssd: Self-ensembling single-stage object detector from point cloud. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 14494–14503,

  53. [61]

    Detecting twenty-thousand classes using image-level supervision

    Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Kr¨ahenb¨uhl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In European Confer- ence on Computer Vision, pages 350–368, 2022. 1, 2

    2022

  54. [62]

    Open-vocabulary 3d semantic segmentation with text-to-image diffusion models

    Xiaoyu Zhu, Hao Zhou, Pengfei Xing, Long Zhao, Hao Xu, Junwei Liang, Alexander Hauptmann, Ting Liu, and Andrew Gallagher. Open-vocabulary 3d semantic segmentation with text-to-image diffusion models. In European Conference on Computer Vision, pages 357–375, 2024. 3 4

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.