REVIEW 3 major objections 5 minor 22 references
Towards Learning to Complete Anything in Lidar
T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read A Lidar-only model called CAL, trained solely on pseudo-labels mined from unlabeled camera-and-Lidar driving video, completes full amodal object shapes from a single scan and recognizes them from any text vocabulary at test time.
desk verdict First credible zero-shot LiDAR panoptic scene completion; SemanticKITTI results look solid, but KITTI360 numbers need a transparent alignment protocol. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the pseudo-labeling engine coupled to a sparse generative completion architecture. The engine produces training pairs of a sparse scan, a completed voxel instance, and a CLIP feature by tracking video object masks with a video segmentation foundation model over forward and backward temporal windows, lifting them to the Lidar frame, aggregating them with known ego-poses into per-instance occupancy grids, averaging per-instance CLIP features across viewpoints, and refining labels with a dense conditional random field over accumulated 360-degree Lidar occupancy. The model is a sparse generative 3D U-Net with a transformer instance decoder: the generative decoder predicts multi-scale occupancy and a pseudo-semantic prototype head, while the transformer matches queries to instance masks and distills CLIP tokens. The key identity is that per-instance CLIP features and voxel occupancies, both aggregated over multiple views, transfer enough information into the Lidar-only network for full amodal shape prediction from a single scan.
What would settle it
Take a static car whose far side is never visible in any camera frame of its pseudo-label window, then compare CAL's completed far-side voxels against the accumulated 360-degree Lidar ground truth; if the model does not infer the hidden side, the claim that partial pseudo-labels yield full amodal shapes is falsified.
Extended reading notes
Core claim
The paper's central discovery is that a single sparse Lidar scan can be completed into dense, recognized, object-level scenes by a model that has never seen a labeled 3D example. Given one scan, CAL predicts scene occupancy, partitions it into class-agnostic instance masks, and regresses a CLIP feature per instance; at inference a text vocabulary is encoded and each instance is matched by cosine similarity. The training signal comes from a pseudo-labeling engine that uses a video segmentation foundation model to track objects across a temporal window, backprojects the masks into Lidar, aggregates them into per-instance voxel occupancies, averages per-instance CLIP features over time, and refines the masks with a conditional random field over accumulated 360-degree occupancy. The paper argues that partial observations suffice: the distilled model learns full object shapes from many partial pseudo-label observations across the dataset, which is why it can complete objects regardless of category and name them at test time.
Load-bearing premise
Everything depends on the video segmentation model tracking every object consistently across the temporal window, because identity switches and missed detections create the incomplete or noisy pseudo-labels the network must learn from.
Editorial extensions
If this is right
- Zero-shot Panoptic Scene Completion becomes feasible without any labeled 3D data, so a user can prompt the model with a custom vocabulary at test time rather than being limited to fixed dataset classes.
- Raw, unlabeled camera-Lidar sequences are now a sufficient training resource, meaning the approach can scale to new cities and sensor setups without an annotation campaign.
- The model's completed instance shapes can serve as the basis for amodal 3D object detection, as demonstrated qualitatively by fitting bounding boxes to recognized instances.
- Because partial pseudo-labels are enough to learn full shapes, future pipelines can trade tracking horizon for cheaper pseudo-labeling, especially when combined with label-refinement steps.
- The main gap to fully supervised methods is concentrated in zero-shot recognition and rare classes, so stronger vision-language features or distribution-aware clustering should transfer directly to higher PQ†.
Reading between the lines
- If the gap to supervised methods is largely a recognition gap, as the paper's zero-shot-versus-oracle comparison suggests, swapping the CLIP backbone for a stronger vision-language model should raise PQ† without changing the architecture or the pseudo-labeling engine.
- The CRF refinement results imply that coverage of occupied-but-unlabeled voxels matters more than perfect per-frame tracking; this suggests cheaper trackers with aggressive label propagation may close much of the pseudo-label quality gap.
- A direct test of the 'partial observations yield full shapes' claim would isolate one static object, hide its far side from every camera view in the pseudo-label window, and check whether the trained model still completes that hidden side from the single scan.
- The approach's dependence on calibrated multi-modal sensors with known ego-poses could limit transfer to monocular or uncalibrated setups, but any sequence with decent calibration is a candidate source of training signal.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CAL (Complete Anything in Lidar), a method for zero-shot Lidar-based panoptic scene completion. CAL has two main components: (i) a pseudo-labeling engine that mines 3D object shapes and CLIP-based semantic features from unlabeled RGB-Lidar sequences, using SAM2 for video-object segmentation and tracking, temporal aggregation, and CRF-based refinement; and (ii) an instance-level completion network, based on a sparse generative 3D U-Net and a transformer decoder, that predicts class-agnostic instance masks over a completed voxel grid and regresses a CLIP feature per instance. At test time, the model takes a single Lidar scan and can be prompted with free-form text vocabularies to perform semantic/panoptic scene completion or amodal 3D object detection. The method is evaluated on SemanticKITTI and SSCBench-KITTI360, reporting 13.12 PQ† (ZS) on SemanticKITTI, about half of the fully supervised PaSCo baseline, and 8.57 PQ† on KITTI360. The authors also construct zero-shot baselines by combining LODE/LiDiff with SAL and show that CAL outperforms them.
Significance. If the results are verifiable, this is a significant contribution: CAL is, to my knowledge, the first method for zero-shot Lidar panoptic scene completion. The pseudo-labeling pipeline is well-designed and the ablation study is unusually thorough, covering CRF refinement, temporal window sizes, CLIP prototype counts, oracle vs. CLIP semantics, and training-data ablations. The zero-shot baselines are constructed with reasonable criteria, and the analysis of pseudo-label vs. model performance is useful. However, the cross-dataset claim on KITTI360 rests on an empirically determined, unverified calibration shift (Appx. A.4), which is a load-bearing verifiability issue that must be resolved before the results can be accepted as reported.
major comments (3)
- [Appendix A.4, Table 1] The KITTI360 results hinge on an empirically determined transformation vector [0.79, 0.3, -0.25] applied to pseudo-labels before voxelization, because the official Lidar-to-Camera alignment transformation could not be confirmed. This is the only mechanism that aligns CAL's KITTI360 training and evaluation data with the provided ground-truth voxel grids. The manuscript does not state which split was used to tune this vector, whether it was held fixed across all KITTI360 runs, or how sensitive the reported metrics are to perturbations of the shift. Without such details, the KITTI360 rows in Table 1 and the associated 'about 40% of PaSCo' claim are not verifiable. Please provide an independent confirmation of the transformation (e.g., using KITTI-360's official calibration assets) or, failing that, report a sensitivity analysis over plausible shift values and clearly disclose the tuning protocol.
- [Abstract, Section 4.4, Appendix C.3] The central claim that 'our distilled model learns to infer full object shapes from multiple such partial observations across the dataset' is only indirectly supported. The pseudo-label-to-model gap is large (25.90 vs. 17.12 PQ† in the semantic-oracle setting on SemanticKITTI), and the full-grid vs. masked-voxel evaluation in Table 10 shows that label coverage substantially affects the metrics. The current evidence is consistent with the model memorizing common shapes rather than genuinely generalizing from partial observations. I would ask for a more direct analysis, such as measuring completion quality as a function of the number of contributing observations per instance, or evaluating on objects observed from a single viewpoint, to substantiate or soften this claim.
- [Table 5, footnote] The footnote to Table 5 reads: 'The discrepancy likely originated from undocumented point accumulation strategy, which we could not clarify with authors.' This sentence is a self-acknowledged missing verification for the coverage numbers, and it should not appear in a published manuscript. The point accumulation strategy must be documented or the coverage claim must be removed or qualified. This is not just a stylistic issue; it undermines the reliability of the coverage analysis that supports the CRF ablation.
minor comments (5)
- [Section 4.2] The statement that 'the gap between CAL and the supervised baselines is largely due to zero-shot recognition performance' is not fully supported by the numbers: in the semantic-oracle setting, CAL reaches 17.12 PQ† vs. 13.12 with CLIP semantics on SemanticKITTI, while the gap to PaSCo (M=1) is 26.49, so recognition accounts for only part of the gap. Please rephrase or provide a per-class decomposition.
- [Appendix A.4] There is a typo: 'voxalization' should be 'voxelization'.
- [Appendix C.2] In the sentence 'we ablate the affect of the number of frames', 'affect' should be 'effect'.
- [Section 2, Related Work] The word 'segmention' in the sentence 'This is a typo: ... such methods only localize the visible portion...' should be corrected to 'segmentation'.
- [Throughout] The authors should consider adding a reference to the point accumulation strategy used for binary occupancy (Fig. 2, step 4), since the Table 5 footnote indicates this was unclear even to the authors.
Circularity Check
No significant circularity: the derivation is self-contained, GT labels are used only for evaluation, and no reported metric is a renamed fit.
full rationale
The derivation chain is self-contained: CAL's training signal comes from a pseudo-labeling engine (SAM/SAM2/CLIP plus temporal Lidar aggregation), and the paper states explicitly that ground-truth labels are used 'solely for evaluation and ablations' (Sec. 4.1). No reported number is a fitted parameter rewritten as a prediction; the oracle/zero-shot gap on SemanticKITTI (17.12 vs 13.12 PQ†) shows the model does not trivially reproduce its own pseudo-labels. Self-citations to SAL (Osep et al. 2024), OpenMask3D (Takmaz et al. 2023), and Semoli (Seidenschwarz et al. 2024) are component-level (mask refinement, CLIP feature aggregation, registration challenges) and are not invoked to justify the central completion result; SAL is also used as a zero-shot baseline, which points away from self-citation inflation. The paper's own limitations (Sec. 4.4) concede that tracking errors and coverage gaps reduce pseudo-label quality, an acknowledged bottleneck rather than a circular move. One passage flagged by the reviewing rule is Appx. A.4: the authors state they 'were unable to confirm the exact transformation process' for SSCBench-KITTI360 GT and 'empirically determined a transformation vector of [0.79, 0.3, -0.25]' for alignment. This is a missing-support / evaluation-integrity concern, not circular reasoning: the shift is a preprocessing calibration, not a model output, and the claimed capability is not defined in terms of it. It should be weighed under soundness and correctness rather than circularity. No uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. Training and evaluation sharing datasets is not circular here because the training signal is mined from unlabeled sensor data and GT labels are reserved for evaluation.
Assumptions & free parameters
free parameters (5)
- KITTI360 camera-to-Lidar alignment shift =
(0.79, 0.3, -0.25)
- CLIP prototype count C =
18
- Temporal window (T_fw, T_bw, stride) =
(32, 8, 2)
- Inference thresholds (tau_vox, tau_obj, tau_ovr) =
SemanticKITTI (0.1, 0.1, 0.1); KITTI360 (0.3, 0.5, 0.4)
- Loss weights (lambda_mask, lambda_occ, lambda_CLIP, lambda_prot) =
(40, 1, 1, 1)
assumptions (5)
- domain assumption SAM2 provides reliable instance segmentation and tracking of arbitrary objects across the chosen temporal window in urban driving video.
- domain assumption Camera-to-Lidar calibration and ego-poses are accurate enough for lifting and temporal aggregation.
- domain assumption CLIP's text-image embedding space is a sufficient semantic space for zero-shot recognition of completed 3D shapes.
- domain assumption The accumulated 360-degree binary occupancy is a valid training target for completion, with dynamic objects handled by preserving tubes (SemanticKITTI) or removing them with dataset tracks (KITTI360).
- ad hoc to paper The empirically determined KITTI360 alignment shift (0.79, 0.3, -0.25) is correct.
Cite this review
Pith. "Pith review of Towards Learning to Complete Anything in Lidar." pith.science (2026). https://pith.science/paper/NRGMU44K
@misc{pith2026250412264,
author = {Pith},
title = {Pith review of: Towards Learning to Complete Anything in Lidar},
year = {2026},
howpublished = {\url{https://pith.science/paper/NRGMU44K}},
note = {Machine review of arXiv:2504.12264}
}
read the original abstract
We propose CAL (Complete Anything in Lidar) for Lidar-based shape-completion in-the-wild. This is closely related to Lidar-based semantic/panoptic scene completion. However, contemporary methods can only complete and recognize objects from a closed vocabulary labeled in existing Lidar datasets. Different to that, our zero-shot approach leverages the temporal context from multi-modal sensor sequences to mine object shapes and semantic features of observed objects. These are then distilled into a Lidar-only instance-level completion and recognition model. Although we only mine partial shape completions, we find that our distilled model learns to infer full object shapes from multiple such partial observations across the dataset. We show that our model can be prompted on standard benchmarks for Semantic and Panoptic Scene Completion, localize objects as (amodal) 3D bounding boxes, and recognize objects beyond fixed class vocabularies. Our project page is https://research.nvidia.com/labs/dvl/projects/complete-anything-lidar
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[5]
3d-ldm: Neural implicit 3d shape generation with latent diffusion models
Nam, G., Khlifi, M., Rodriguez, A., Tono, A., Zhou, L., and Guerrero, P. 3d-ldm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint arXiv:2212.00842,
-
[7]
Ren, X., Huang, J., Zeng, X., Museth, K., Fidler, S., and Williams, F
URL https:// arxiv.org/abs/2408.00714. Ren, X., Huang, J., Zeng, X., Museth, K., Fidler, S., and Williams, F. Xcube: Large-scale 3d generative modeling using sparse voxel hierarchies. In IEEE Conf. Comput. Vis. Pattern Recog.,
-
[9]
rely on panoptic segmentation labels for identifying and registering such objects. However, (1) we assume no access to semantic labels, and (2) performing such registration is error-prone in our setting. In the original SemanticKITTI completion benchmark (Behley et al., 2019), while performing point level aggregation, spatio-temporal tubes for dynamic obj...
work page 2019
-
[10]
dataset where the 3D object tracks are available as an asset in the dataset, we perform a dynamic object removal operation and only keep the static parts of the scene during Lidar point cloud accumulation. We would like to highlight that while such object tracks are commonly available in most autonomous driving datasets, it is also possible to instead use...
work page 2024
-
[11]
on each masklet across the video sequence, and average-pool normalized CLIP features obtained at each timestamp, per object. The output from this operation is a separate CLIP feature vector of dimension 768, for each object in the scene. B. CAL Model Details B.1. Architecture Details Overview. Our model architecture is built upon a sparse-generative 3D U-...
work page 2024
-
[12]
as in PaSCo (Cao et al., 2024). The voxelized features are then passed through a sparse convolutional encoder that is composed of 4 encoder blocks which gradually upsample the features, resulting in f 1:1 enc, f 1:2 enc, f 1:4 enc, f 1:8 enc. First encoder block consists of 3 residual blocks, and each of the remaining 3 encoder blocks consists of a sparse...
work page 2024
-
[13]
our transformer decoder also includes a CLIP distillation head which aims to predict a CLIP token for each query. This MLP block consists of layers with dimensions [384, 512, 1024, 768, 768], where the final dimension, 768, is the dimensionality of the CLIP embedding space. In essence, for each query, our method regresses a CLIP feature vector. Importantl...
work page 2024
-
[15]
and evaluateCAL on two datasets that provide semantic and instance-level labels for PSC: SSCBench-KITTI360 (Li et al., 2024; Liao et al.,
work page 2024
Show all 22 references
-
[16]
and SemanticKITTI (Behley et al., 2019; Geiger et al., 2012
2019
-
[19]
We train the CAL model using two different sets of data: pseudo-labels w/o CRF refinement, and with CRF refinement
Model ablations for data on SSCBench-KITTI360 (Li et al., 2024). We train the CAL model using two different sets of data: pseudo-labels w/o CRF refinement, and with CRF refinement. We report PSC metrics for both variants individually. We observe that training CAL with data tha...
2024
-
[21]
+MaskPLS (Marcuzzi et al., 2023), SCPNet (Xia et al.,
2023
-
[22]
vehicle” “car
dataset. Per-class scores for the baselines and class-frequencies are taken from (Cao et al., 2024). Method ■ car (3.92%) ■ bicycle (0.03%) ■ motorcycle (0.03%) ■ truck (0.16%) ■ other-veh. (0.20%) ■ person (0.07%) ■ bicyclist (0.07%) ■ motorcyclist (0.05%) ■ road (15.30%) ■ p...
2024
-
[23]
While baselines struggle with coherent structure and semantic accuracy, CAL produces cleaner and more complete outputs that align closely with the ground truth
and LODE (Li et al., 2023b) with SAL (Osep et al., 2024). While baselines struggle with coherent structure and semantic accuracy, CAL produces cleaner and more complete outputs that align closely with the ground truth. 24
2024
-
[32]
CRF-based refinement module In our pseudo-labeling engine (as depicted in Fig
Camera-to-Lidar alignment shift (0.0, 0.0, 0.0) (0.79, 0.3, -0.25) 13 Towards Learning to Complete Anything in Lidar A.1. CRF-based refinement module In our pseudo-labeling engine (as depicted in Fig. 2 of the main paper), we first accumulate two types of information: binary o...
2011
-
[500]
For all settings, the prototypes are fixed at the beginning, and are not updated during training. At inference time, we completely discard our model’s prototype class predictions, and only use the predicted instance masks over the voxel grid as well as the predicted CLIP featu...
2024
-
[2013]
These datasets provide per-voxel semantic labels (SemanticKITTI: 20 classes, 8 are thing; SSCBench-KITTI360: 19 classes, 6 are thing) that we only use during evaluation
whose instance-level labels are provided in (Cao et al., 2024). These datasets provide per-voxel semantic labels (SemanticKITTI: 20 classes, 8 are thing; SSCBench-KITTI360: 19 classes, 6 are thing) that we only use during evaluation. Data was recorded using a 64-beam Velodyne ...
2024
-
[2019]
car” can be addressed by “car, jeep, SUV , van
and keep spatio-temporal moving object tubes for consistency with supervised baseline methods. Data statistics. To have a fair comparison between our method and the supervised panoptic completion methods reported by Cao et al. (2024), we generate pseudo-labels for the same Lid...
2024
-
[2020]
+MaskPLS (Marcuzzi et al., 2023), JS3CNet (Yan et al.,
2023
-
[2021]
Long-tailed 3d detection via 2d late fusion
Ma, Y ., Peri, N., Wei, S., Hua, W., Ramanan, D., Li, Y ., and Kong, S. Long-tailed 3d detection via 2d late fusion. arXiv preprint arXiv:2312.10986,
-
[2022]
and Nichol, A
Jun, H. and Nichol, A. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463,
-
[2023]
Shelf- supervised multi-modal pre-training for 3d object detec- tion
Khurana, M., Peri, N., Ramanan, D., and Hays, J. Shelf- supervised multi-modal pre-training for 3d object detec- tion. arXiv preprint arXiv:2406.10115,
-
[2024]
X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al
Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012,
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.