REVIEW 5 major objections 5 minor 22 references
OV-MAP: Open-Vocabulary Zero-Shot 3D Instance Segmentation Map for Robots
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read OV-MAP turns posed RGB-D frames and a point cloud into a text-queryable 3D map of individual objects, without any 3D-supervised segmentation model.
desk verdict Neat zero-shot 3D instance segmentation pipeline, but the headline ScanNet200 number is not yet reproducible because the evaluation protocol is under-specified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a 3D mask voting step over mesh-segmented areas. The input to voting is a set of candidate 3D masks obtained by lifting masks from a class-agnostic 2D segmenter through a supplemented depth image that merges raw sensor depth with synthetic depth computed from the point cloud. Mask groups from adjacent frames are merged only when their overlap rate relative to the larger mask exceeds a threshold, and the merged point set is voxelized at a quarter of the original scale. Each area from an efficient graph-based segmentation of the mesh, computed from surface normals, then receives the dominant group id among the candidate masks that intersect it; that group id is the final 3D instance.
What would settle it
Run OV-MAP on the ScanNet200 validation scenes with the overlap threshold in Eq. (3) fixed at one value across all scenes and then sweep that value; if the reported AP of 11.9 appears only in a narrow tuned band and collapses under small changes, the claimed zero-shot robustness is not supported.
Extended reading notes
Core claim
The paper's central claim is that instance-level precision in open-vocabulary 3D maps does not require a 3D-supervised instance segmentation model. Instead, class-agnostic 2D masks are projected into 3D with a depth image that fills raw-depth gaps using synthetic depth rendered from the reconstructed point cloud. Candidate 3D masks are merged pair-wise by overlap, with the criterion computed against the larger mask so small objects are not swallowed, and each area of a graph-based mesh segmentation is assigned the dominant mask group by voting. That group becomes the instance, and its label comes from the highest-scoring RGB crop embedded with open-vocabulary features. The paper argues this avoids the feature spillover that blurs contiguous objects in per-voxel maps and preserves fine instance boundaries.
Load-bearing premise
The load-bearing premise is that the overlap threshold in the merging step and the mesh-segmentation parameters can be fixed once and still align with real object boundaries in any new scene; if either must be tuned per dataset, the zero-shot claim fails.
Editorial extensions
If this is right
- A robot can segment and name objects in rooms it has never seen using only RGB-D input, because no 3D-labeled training data is needed.
- Performance stays more even across rare (tail) categories than supervised 3D segmenters, which decline on rare classes.
- The supplemented depth image is necessary: using raw depth alone drops AP from 11.9 to 11.1 on ScanNet200, and synthetic depth alone drops it to 10.1.
- Because the final instance proposals are independent of 3D supervision, the method transfers to a new dataset and to real-world captures without retraining.
Reading between the lines
- An implication the paper leaves implicit is that the frame-pair merging and per-area voting are local operations, so the pipeline could be adapted to run incrementally as a robot explores, rather than only on a pre-reconstructed point cloud.
- Because the point cloud is assumed already reconstructed, the method inherits any camera-pose drift; a straightforward stress test is to add pose noise to the Replica inputs and measure how AP degrades.
- The label of each instance comes from a single highest-scoring view, so an object that is well segmented but only visible in a blurry frame could be mislabeled; aggregating features across multiple views would be a natural follow-up.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. OV-MAP proposes a zero-shot, open-vocabulary 3D instance segmentation pipeline for mobile robots. The method takes posed RGB-D frames and a reconstructed point cloud, generates 2D class-agnostic masks with CropFormer, refines the depth image by merging raw and synthetic depth, projects the masks into 3D, merges candidate masks via an overlap criterion, performs dominant-group voting over graph-based mesh segments, and attaches CLIP features to each resulting instance. The paper reports ScanNet200 and Replica instance segmentation results, an ablation of depth types, qualitative comparisons, and real-world demonstrations. The central claim is that OV-MAP achieves accurate zero-shot 3D instance segmentation without relying on 3D supervised segmentation models, with a reported ScanNet200 AP of 11.9 versus 8.4 for SAM3D and 6.6 for OpenScene.
Significance. If the result holds, OV-MAP would be a practically useful step toward zero-shot open-vocabulary 3D instance mapping from RGB-D alone, with clear robotics applications. The core idea of confining CLIP features to instance proposals produced by projecting 2D class-agnostic masks and merging them through mesh-based voting is coherent and does not rely on a fitted constant or on circular benchmark reuse. The paper also includes an ablation study and real-world validation, which are valuable. However, the main quantitative claim rests on an evaluation protocol that is not specified in sufficient detail to be reproducible or even well-defined: the paper does not state how predicted masks are assigned to ScanNet200 class labels, how masks are ranked for average precision when all confidence scores are set to 1.0, or how the per-voxel baselines are converted to instance masks. These issues, together with an internally inconsistent depth-fusion equation and the omission of the main zero-shot baseline from the Replica table, mean that the headline result is not yet established.
major comments (5)
- [IV-A, Table I] The ScanNet200 evaluation protocol is not fully specified, and as written it is not well-defined. Section IV-A states that each predicted mask is assigned a uniform confidence score of 1.0, but average precision requires a ranking of predictions; with identical scores, the AP value depends on tie-breaking order. The paper also never describes how a 3D instance mask is assigned to one of the 200 category labels, even though ScanNet200 AP is class-specific. Without this procedure, the reported 11.9 AP and the comparison with OpenScene and SAM3D cannot be reproduced. This is a load-bearing gap in the central claim of superior zero-shot performance, and it should be fixed by specifying the class-assignment rule and the ranking criterion, or by evaluating with a protocol that does not require a confidence ordering.
- [III-A, Eq. (1)] The depth supplementation rule in Eq. (1) appears logically reversed. The text says the goal is to supplement missing parts of the raw depth image with synthetic depth derived from the point cloud. As written, when I_d(i,j) is nonzero (valid raw depth) and I_d'(i,j) is zero (no synthetic surface), the output is I_d'(i,j) = 0, discarding valid raw depth. A fusion rule that preserves raw depth unless the raw value is missing would be the natural implementation. Because supplemented depth is the proposed contribution and is credited for the improvement in Table III, this inconsistency could affect the reported results and must be corrected or clarified.
- [IV-B, Table II] The Replica comparison omits SAM3D, which is the main zero-shot per-instance baseline used in the ScanNet200 table. The text claims that OV-MAP "significantly outperforms the baseline models" on Replica, but without SAM3D the zero-shot comparison is incomplete. Furthermore, no variance, standard deviation, or significance test is reported for any of the tables; the ScanNet200 margin of 11.9 versus 8.4 is presented as a single number, so it is not possible to assess whether this difference is robust.
- [IV-B, Table I] The claim that supervised methods "tend to falter in the tail category" is contradicted by Table I. Mask3D has a tail AP of 17.9 and OpenMask3D has a tail AP of 14.9, while OV-MAP has a tail AP of 12.7. In fact, OV-MAP is below both supervised methods on the tail subset. The sentence should be revised to match the reported numbers, or the analysis should be reframed around the head/common/tail trade-off rather than a blanket statement about supervised methods failing on tail categories.
- [III-B, Section IV-A] Several parameters that are essential for reproducing the method are unstated or appear to contain typos. The merging threshold in Eq. (3) is described only as a "predefined threshold" and its value is never given. The mesh-segmentation step of [9] is said to be applied to 3D meshes via surface normals, but no implementation details or hyperparameters are provided. In Section IV-A, the merge voxelization is reported as "a radius of 0.05cm," which is likely a typo for 0.05 m or another value; this should be clarified. If the merging threshold is tuned per dataset, the zero-shot claim would be weakened, so the paper should disclose how these parameters were set.
minor comments (5)
- [II] The heading "Per-V oxel 3D Mapping" contains a typo; it should be "Per-Voxel 3D Mapping."
- [IV-A] The implementation details state that the reconstructed point cloud is voxelized with a radius of 2 cm and that merge voxelization uses a radius of 0.05 cm; the latter seems inconsistent and should be corrected to the intended unit or value.
- [III-C, Eq. (5)] The weighting factors alpha and beta in Eq. (5) are never given numerical values or a selection procedure; the authors should state how these were chosen and whether they were fixed across datasets.
- [IV-E] The real-world experiments are only qualitative; reporting quantitative metrics, such as the number of scenes, object categories, and success rates for query-based matching, would strengthen the adaptability claim.
- [IV-A] The paper says that AP is "averaged over a range from 0.5 to 0.95 in increments of 0.05, as per ScanNet's evaluation protocol," but the reference cited is to ScanNet200 [10]; the authors should clarify the exact evaluation code and whether the standard ScanNet AP computation is used.
Circularity Check
No significant circularity: the proposed OV-MAP pipeline constructs 3D instance masks from 2D class-agnostic masks, depth projection, overlap merging, and mesh voting, with no fitted quantity renamed as a prediction and no load-bearing self-citations.
full rationale
The derivation chain in OV-MAP is not circular. The final 3D instance masks are obtained by projecting 2D class-agnostic masks (CropFormer) into 3D using supplemented depth, merging by overlap rate (Eqs. 2-3), and assigning dominant group IDs to mesh segments via voting (Eq. 4). No equation reduces to the ScanNet200 or Replica benchmark labels, and no parameter is fitted to the reported AP values. The per-mask CLIP labeling in Sec. III-C uses a score combining pixel and point counts (Eq. 5) and selects the highest-scoring view, but the paper does not specify how these CLIP features are converted into the class labels needed for average-precision evaluation; that is an evaluation-protocol gap rather than circularity. The method's components are supported by external references (CropFormer, Felzenszwalb graph segmentation, CLIP, SAM3D, OpenMask3D), and no load-bearing self-citation or author-imported uniqueness theorem appears. The uniform confidence score of 1.0 and the unspecified tie-breaking order for AP are validity concerns, but they do not make the claimed prediction equivalent to its inputs by construction. Thus the paper warrants a circularity score of 0.
Assumptions & free parameters
free parameters (4)
- alpha in Eq. (5)
- beta in Eq. (5)
- Merging overlap threshold in Eq. (3)
- Post-processing parameters (DBSCAN, nearest neighbor)
assumptions (4)
- standard math Camera intrinsics, depth scaling, and pose transforms are known and accurate.
- domain assumption CLIP image-text embeddings provide accurate open-vocabulary matching for object crops.
- domain assumption Felzenszwalb-Huttenlocher graph segmentation applies to 3D meshes via surface normals.
- domain assumption The reconstructed point cloud is accurate enough to provide synthetic depth that aligns with camera depth.
Cite this review
Pith. "Pith review of OV-MAP: Open-Vocabulary Zero-Shot 3D Instance Segmentation Map for Robots." pith.science (2026). https://pith.science/paper/FVS57KKA
@misc{pith2026250611585,
author = {Pith},
title = {Pith review of: OV-MAP: Open-Vocabulary Zero-Shot 3D Instance Segmentation Map for Robots},
year = {2026},
howpublished = {\url{https://pith.science/paper/FVS57KKA}},
note = {Machine review of arXiv:2506.11585}
}
read the original abstract
We introduce OV-MAP, a novel approach to open-world 3D mapping for mobile robots by integrating open-features into 3D maps to enhance object recognition capabilities. A significant challenge arises when overlapping features from adjacent voxels reduce instance-level precision, as features spill over voxel boundaries, blending neighboring regions together. Our method overcomes this by employing a class-agnostic segmentation model to project 2D masks into 3D space, combined with a supplemented depth image created by merging raw and synthetic depth from point clouds. This approach, along with a 3D mask voting mechanism, enables accurate zero-shot 3D instance segmentation without relying on 3D supervised segmentation models. We assess the effectiveness of our method through comprehensive experiments on public datasets such as ScanNet200 and Replica, demonstrating superior zero-shot performance, robustness, and adaptability across diverse environments. Additionally, we conducted real-world experiments to demonstrate our method's adaptability and robustness when applied to diverse real-world environments.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[9]
Efficient graph-based image segmentation,
P. F. Felzenszwalb and D. P. Huttenlocher, “Efficient graph-based image segmentation,”International journal of computer vision, vol. 59, pp. 167–181, 2004
work page 2004
-
[1]
Learning transferable visual models from natural language supervision,
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark,et al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning, pp. 8748–8763, PMLR, 2021
2021
-
[2]
Visual language maps for robot navigation,
C. Huang, O. Mees, A. Zeng, and W. Burgard, “Visual language maps for robot navigation,” in2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 10608–10615, IEEE, 2023
work page 2023
-
[3]
Openscene: 3d scene understanding with open vocabularies,
S. Peng, K. Genova, C. Jiang, A. Tagliasacchi, M. Pollefeys, T. Funkhouser,et al., “Openscene: 3d scene understanding with open vocabularies,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 815–824, 2023
work page 2023
-
[4]
Conceptfusion: Open-set multimodal 3d mapping,
K. M. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, T. Chen, A. Maalouf, S. Li, G. Iyer, S. Saryazdi, N. Keetha,et al., “Conceptfusion: Open-set multimodal 3d mapping,”arXiv preprint arXiv:2302.07241, 2023
arXiv 2023
-
[5]
Openmask3d: Open-vocabulary 3d instance segmen- tation,
A. Takmaz, E. Fedele, R. W. Sumner, M. Pollefeys, F. Tombari, and F. Engelmann, “Openmask3d: Open-vocabulary 3d instance segmen- tation,”arXiv preprint arXiv:2306.13631, 2023
arXiv 2023
-
[6]
Sam3d: Segment anything in 3d scenes,
Y . Yang, X. Wu, T. He, H. Zhao, and X. Liu, “Sam3d: Segment anything in 3d scenes,”arXiv preprint arXiv:2306.03908, 2023
arXiv 2023
-
[7]
High-quality entity segmentation,
L. Qi, J. Kuen, W. Guo, T. Shen, J. Gu, J. Jia, Z. Lin, and M.-H. Yang, “High-quality entity segmentation,”arXiv preprint arXiv:2211.05776, 2022
arXiv 2022
Show all 22 references
-
[8]
Segment anything,
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo,et al., “Segment anything,” inProceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015–4026, 2023
2023
-
[10]
Language-grounded indoor 3d semantic segmentation in the wild,
D. Rozenberszki, O. Litany, and A. Dai, “Language-grounded indoor 3d semantic segmentation in the wild,” inProceedings of the European Conference on Computer Vision (ECCV), 2022
2022
-
[11]
The replica dataset: A digital replica of indoor spaces,
J. Straub, T. Whelan, L. Ma, Y . Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma,et al., “The replica dataset: A digital replica of indoor spaces,”arXiv preprint arXiv:1906.05797, 2019
1906 arXiv
-
[12]
Language-driven semantic segmentation,
B. Li, K. Q. Weinberger, S. Belongie, V . Koltun, and R. Ranftl, “Language-driven semantic segmentation,”arXiv preprint arXiv:2201.03546, 2022
2022 arXiv
-
[13]
Open-vocabulary semantic segmentation with mask-adapted clip,
F. Liang, B. Wu, X. Dai, K. Li, Y . Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu, “Open-vocabulary semantic segmentation with mask-adapted clip,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7061–7070, 2023
2023
-
[14]
Freeseg: Unified, universal and open- vocabulary image segmentation,
J. Qin, J. Wu, P. Yan, M. Li, R. Yuxi, X. Xiao, Y . Wang, R. Wang, S. Wen, X. Pan,et al., “Freeseg: Unified, universal and open- vocabulary image segmentation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19446– 19455, 2023
2023
-
[15]
Scaling open-vocabulary image segmentation with image-level labels,
G. Ghiasi, X. Gu, Y . Cui, and T.-Y . Lin, “Scaling open-vocabulary image segmentation with image-level labels,” inEuropean Conference on Computer Vision, pp. 540–557, Springer, 2022
2022
-
[16]
Clip on wheels: Zero-shot object navigation as object localization and exploration,
S. Y . Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song, “Clip on wheels: Zero-shot object navigation as object localization and exploration,”arXiv preprint arXiv:2203.10421, vol. 3, no. 4, p. 7, 2022
2022 arXiv
-
[17]
Pla: Language-driven open-vocabulary 3d scene understanding,
R. Ding, J. Yang, C. Xue, W. Zhang, S. Bai, and X. Qi, “Pla: Language-driven open-vocabulary 3d scene understanding,” inPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7010–7019, 2023
2023
-
[18]
Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models,
H. Ha and S. Song, “Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models,”arXiv preprint arXiv:2207.11514, 2022
2022 arXiv
-
[19]
Mask3d: Mask transformer for 3d semantic instance segmentation,
J. Schult, F. Engelmann, A. Hermans, O. Litany, S. Tang, and B. Leibe, “Mask3d: Mask transformer for 3d semantic instance segmentation,” in2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 8216–8223, IEEE, 2023
2023
-
[20]
Scannet: Richly-annotated 3d reconstructions of indoor scenes,
A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” inProceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828–5839, 2017
2017
-
[21]
A brief review of nearest neighbor algorithm for learning and classification,
K. Taunk, S. De, S. Verma, and A. Swetapadma, “A brief review of nearest neighbor algorithm for learning and classification,” in2019 international conference on intelligent computing and control systems (ICCS), pp. 1255–1260, IEEE, 2019
2019
-
[22]
Dbscan revisited, revisited: why and how you should (still) use dbscan,
E. Schubert, J. Sander, M. Ester, H. P. Kriegel, and X. Xu, “Dbscan revisited, revisited: why and how you should (still) use dbscan,”ACM Transactions on Database Systems (TODS), vol. 42, no. 3, pp. 1–21, 2017
2017
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.