{"id":"a701a07a-7f7b-4374-ae09-437806aa3aa1","arxiv_id":"2504.12264","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CAL performs zero-shot, open-vocabulary panoptic scene completion from a single Lidar scan using a model distilled from pseudo-labels mined from temporal video and Lidar sequences.","lead":"The authors present CAL, the first zero-shot Lidar panoptic scene completion system: it trains a single-scan Lidar model on pseudo-labels mined from unlabeled camera and Lidar video, then completes occluded object shapes and recognizes them with text prompts. It reaches about half of fully supervised PaSCo's performance without using any manual 3D semantic labels.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"KITTI360 results hinge on an empirically tuned, unverified alignment shift (Appx. A.4); without an independent check, the cross-dataset claim is unsupported.","rationale":"The reader's CONDITIONAL verdict is appropriate. I focus on the KITTI360 alignment because it is a self-admitted missing support that directly controls a reported headline number. By contrast, SAM2 label noise is acknowledged, ablated (Tabs 3-5), and affects the supervised baselines and CAL asymmetrically only in degree; it is a limitation, not a hidden assumption that could invalidate the reported numbers. The calibration shift is different: Appx. A.4 states that the exact Lidar-to-Camera alignment for SSCBench-KITTI360 could not be confirmed, so a shift of [0.79, 0.3, -0.25] was 'empirically determined.' Applying an unverified, possibly test-selected 3D translation to pseudo-labels before voxelization makes the KITTI360 rows of Table 1 non-reproducible and could change PQ† substantially. The paper also does not report sensitivity of results to this shift or state which split was used for tuning. A second, less load-bearing concern is that 'full object shapes' and amodal completion are not directly quantified: no metric isolates voxels unobserved in the input scan or in the pseudo-labels, so the claim that the distilled model learns to infer full object shapes is inferred only from overall PQ†, not from an explicit occluded-voxel or object-completeness evaluation. If the official alignment check passes and a held-out occluded-voxel metric supports amodal completion, the central claim would stand; otherwise the KITTI360 generalization claim should be downgraded. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":31713,"tokens_out":8833,"duration_ms":95739,"concrete_test":"Obtain the official SSCBench-KITTI360 alignment code or transformation (or independently derive it from the dataset calibration files), rerun pseudo-label voxelization and CAL evaluation using the official transform instead of [0.79, 0.3, -0.25], and sweep the shift over a plausible range (e.g., each axis in 0.1 m steps over +/-0.5 m) on the KITTI360 validation split. If PQ† changes by more than about 1-2 points, or if the official transform differs from the tuned vector, the Table 1 KITTI360 results must be recomputed before the cross-dataset generalization claim is accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weak point is the SSCBench-KITTI360 evaluation. Appx. A.4 states that the ground-truth voxel grids 'had undergone a Lidar-to-Camera alignment transformation' that the authors 'were unable to confirm,' and that they therefore 'empirically determined a transformation vector of [0.79, 0.3, -0.25]' to apply to their pseudo-labels before voxelization. This vector is the only mechanism that puts CAL's KITTI360 training and evaluation data into the GT coordinate frame. Because the paper does not report on which split the shift was tuned, whether it was held fixed across all KITTI360 runs, or how sensitive the results are to the shift, the KITTI360 rows of Table 1 and the associated 'about 40% of PaSCo' claim are not verifiable as reported. A wrong or test-tuned shift could either inflate or deflate all KITTI360 metrics. This is a self-acknowledged missing support, not a hypothetical: the authors explicitly state they could not confirm the exact transformation. The central claim of generalizing to a second benchmark therefore rests on an untested calibration assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CAL (Complete Anything in Lidar), a method for zero-shot Lidar-based panoptic scene completion. CAL has two main components: (i) a pseudo-labeling engine that mines 3D object shapes and CLIP-based semantic features from unlabeled RGB-Lidar sequences, using SAM2 for video-object segmentation and tracking, temporal aggregation, and CRF-based refinement; and (ii) an instance-level completion network, based on a sparse generative 3D U-Net and a transformer decoder, that predicts class-agnostic instance masks over a completed voxel grid and regresses a CLIP feature per instance. At test time, the model takes a single Lidar scan and can be prompted with free-form text vocabularies to perform semantic/panoptic scene completion or amodal 3D object detection. The method is evaluated on SemanticKITTI and SSCBench-KITTI360, reporting 13.12 PQ† (ZS) on SemanticKITTI, about half of the fully supervised PaSCo baseline, and 8.57 PQ† on KITTI360. The authors also construct zero-shot baselines by combining LODE/LiDiff with SAL and show that CAL outperforms them.","tokens_in":32004,"tokens_out":5028,"duration_ms":49435,"significance":"If the results are verifiable, this is a significant contribution: CAL is, to my knowledge, the first method for zero-shot Lidar panoptic scene completion. The pseudo-labeling pipeline is well-designed and the ablation study is unusually thorough, covering CRF refinement, temporal window sizes, CLIP prototype counts, oracle vs. CLIP semantics, and training-data ablations. The zero-shot baselines are constructed with reasonable criteria, and the analysis of pseudo-label vs. model performance is useful. However, the cross-dataset claim on KITTI360 rests on an empirically determined, unverified calibration shift (Appx. A.4), which is a load-bearing verifiability issue that must be resolved before the results can be accepted as reported.","major_comments":[{"comment":"The KITTI360 results hinge on an empirically determined transformation vector [0.79, 0.3, -0.25] applied to pseudo-labels before voxelization, because the official Lidar-to-Camera alignment transformation could not be confirmed. This is the only mechanism that aligns CAL's KITTI360 training and evaluation data with the provided ground-truth voxel grids. The manuscript does not state which split was used to tune this vector, whether it was held fixed across all KITTI360 runs, or how sensitive the reported metrics are to perturbations of the shift. Without such details, the KITTI360 rows in Table 1 and the associated 'about 40% of PaSCo' claim are not verifiable. Please provide an independent confirmation of the transformation (e.g., using KITTI-360's official calibration assets) or, failing that, report a sensitivity analysis over plausible shift values and clearly disclose the tuning protocol.","section":"Appendix A.4, Table 1"},{"comment":"The central claim that 'our distilled model learns to infer full object shapes from multiple such partial observations across the dataset' is only indirectly supported. The pseudo-label-to-model gap is large (25.90 vs. 17.12 PQ† in the semantic-oracle setting on SemanticKITTI), and the full-grid vs. masked-voxel evaluation in Table 10 shows that label coverage substantially affects the metrics. The current evidence is consistent with the model memorizing common shapes rather than genuinely generalizing from partial observations. I would ask for a more direct analysis, such as measuring completion quality as a function of the number of contributing observations per instance, or evaluating on objects observed from a single viewpoint, to substantiate or soften this claim.","section":"Abstract, Section 4.4, Appendix C.3"},{"comment":"The footnote to Table 5 reads: 'The discrepancy likely originated from undocumented point accumulation strategy, which we could not clarify with authors.' This sentence is a self-acknowledged missing verification for the coverage numbers, and it should not appear in a published manuscript. The point accumulation strategy must be documented or the coverage claim must be removed or qualified. This is not just a stylistic issue; it undermines the reliability of the coverage analysis that supports the CRF ablation.","section":"Table 5, footnote"}],"minor_comments":[{"comment":"The statement that 'the gap between CAL and the supervised baselines is largely due to zero-shot recognition performance' is not fully supported by the numbers: in the semantic-oracle setting, CAL reaches 17.12 PQ† vs. 13.12 with CLIP semantics on SemanticKITTI, while the gap to PaSCo (M=1) is 26.49, so recognition accounts for only part of the gap. Please rephrase or provide a per-class decomposition.","section":"Section 4.2"},{"comment":"There is a typo: 'voxalization' should be 'voxelization'.","section":"Appendix A.4"},{"comment":"In the sentence 'we ablate the affect of the number of frames', 'affect' should be 'effect'.","section":"Appendix C.2"},{"comment":"The word 'segmention' in the sentence 'This is a typo: ... such methods only localize the visible portion...' should be corrected to 'segmentation'.","section":"Section 2, Related Work"},{"comment":"The authors should consider adding a reference to the point accumulation strategy used for binary occupancy (Fig. 2, step 4), since the Table 5 footnote indicates this was unclear even to the authors.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically strong and the SemanticKITTI results appear carefully evaluated. The main concern is the KITTI360 alignment shift, which is a genuine verifiability issue but likely fixable with additional experiments or an independent calibration check. The Table 5 footnote is also concerning and suggests the manuscript was not fully cleaned; the editor may want to ensure the authors address it explicitly in the revision. Overall, I believe the central claim is defensible and the paper merits a major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The CAL paper is a genuine first: zero-shot LiDAR panoptic scene completion trained from pseudo-labels mined from unlabeled RGB-LiDAR sequences. The SemanticKITTI results are the main evidence, and they look credible—13.12 PQ† zero-shot vs. 26.49 for single-model PaSCo, with the oracle at 17.12 showing the completion is real and the CLIP prompting is the main loss. The ablations are thorough and honest: temporal windows, CRF, prototypes, coverage, and the zero-shot baselines are carefully constructed. I don't see circularity; GT labels are evaluation-only.\n\nThe soft spots are real. The stress-test note is right about Appx. A.4: the KITTI360 numbers rest on an empirically determined alignment shift that the authors admit they could not confirm. They don't say which split was used to fit it, whether it was fixed for all runs, or how sensitive the results are. That makes the KITTI360 rows of Table 1 unverifiable as reported. It doesn't sink the paper—SemanticKITTI doesn't use that shift—but it undercuts the cross-dataset claim, and the authors should fix this before publication: release the alignment procedure, do a sensitivity sweep, or use the official transform.\n\nTwo smaller issues. For KITTI360 they used dataset-provided 3D object tracks for dynamic object removal, a nontrivial supervision signal that slightly tarnishes the 'unlabeled' story for that benchmark, even though they disclose it. And there's no code or data, which matters because the pseudo-label pipeline has many moving parts and the failure modes are exactly what a reader would want to inspect. The 'anything' in the title also overshoots the quantitative evidence: the open-vocabulary evaluation is qualitative, mostly on standard classes.\n\nOverall, the central argument holds, the authors are honest about limitations, and the paper deserves serious refereeing. The reviewers should focus on the alignment protocol, the dynamic-object handling, and artifact release.","headline":"First credible zero-shot LiDAR panoptic scene completion; SemanticKITTI results look solid, but KITTI360 numbers need a transparent alignment protocol.","tokens_in":32583,"tokens_out":4091,"would_cite":true,"duration_ms":39744,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Lidar-only model called CAL, trained solely on pseudo-labels mined from unlabeled camera-and-Lidar driving video, completes full amodal object shapes from a single scan and recognizes them from any text vocabulary at test time.","keywords":["Lidar scene completion","panoptic scene completion","zero-shot recognition","pseudo-label mining","CLIP distillation","amodal object completion","video object segmentation","autonomous driving"],"falsifier":"Take a static car whose far side is never visible in any camera frame of its pseudo-label window, then compare CAL's completed far-side voxels against the accumulated 360-degree Lidar ground truth; if the model does not infer the hidden side, the claim that partial pseudo-labels yield full amodal shapes is falsified.","tokens_in":31507,"feed_emoji":"🚗","tokens_out":8676,"duration_ms":79362,"temperature":0.7,"pith_summary":"The paper sets out to show that Lidar scene completion does not need manually labeled 3D data. CAL mines shape and semantic pseudo-labels from unlabeled camera-and-Lidar sequences by tracking objects in video, lifting their masks into Lidar, and aggregating them over time, then distills them into a Lidar-only completion model. Although the mined shapes are partial and noisy, the trained model learns to output full amodal object extents and to attach a CLIP semantic feature to each instance, so a test-time prompt can name the object. On SemanticKITTI the zero-shot model reaches 13.12 PQ†, roughly half of the fully supervised PaSCo baseline, using no 3D labels. If this holds, scene completion can scale to any dataset with raw sensors, not just datasets with expensive annotations.","feed_headline":"Zero-shot lidar completion reaches half of supervised quality","feed_subtitle":"CAL mines shape and text features from unlabeled camera-lidar video, so test-time prompts name objects never seen in training.","key_machinery":"The load-bearing mechanism is the pseudo-labeling engine coupled to a sparse generative completion architecture. The engine produces training pairs of a sparse scan, a completed voxel instance, and a CLIP feature by tracking video object masks with a video segmentation foundation model over forward and backward temporal windows, lifting them to the Lidar frame, aggregating them with known ego-poses into per-instance occupancy grids, averaging per-instance CLIP features across viewpoints, and refining labels with a dense conditional random field over accumulated 360-degree Lidar occupancy. The model is a sparse generative 3D U-Net with a transformer instance decoder: the generative decoder predicts multi-scale occupancy and a pseudo-semantic prototype head, while the transformer matches queries to instance masks and distills CLIP tokens. The key identity is that per-instance CLIP features and voxel occupancies, both aggregated over multiple views, transfer enough information into the Lidar-only network for full amodal shape prediction from a single scan.","core_discovery":"The paper's central discovery is that a single sparse Lidar scan can be completed into dense, recognized, object-level scenes by a model that has never seen a labeled 3D example. Given one scan, CAL predicts scene occupancy, partitions it into class-agnostic instance masks, and regresses a CLIP feature per instance; at inference a text vocabulary is encoded and each instance is matched by cosine similarity. The training signal comes from a pseudo-labeling engine that uses a video segmentation foundation model to track objects across a temporal window, backprojects the masks into Lidar, aggregates them into per-instance voxel occupancies, averages per-instance CLIP features over time, and refines the masks with a conditional random field over accumulated 360-degree occupancy. The paper argues that partial observations suffice: the distilled model learns full object shapes from many partial pseudo-label observations across the dataset, which is why it can complete objects regardless of category and name them at test time.","pith_inferences":["If the gap to supervised methods is largely a recognition gap, as the paper's zero-shot-versus-oracle comparison suggests, swapping the CLIP backbone for a stronger vision-language model should raise PQ† without changing the architecture or the pseudo-labeling engine.","The CRF refinement results imply that coverage of occupied-but-unlabeled voxels matters more than perfect per-frame tracking; this suggests cheaper trackers with aggressive label propagation may close much of the pseudo-label quality gap.","A direct test of the 'partial observations yield full shapes' claim would isolate one static object, hide its far side from every camera view in the pseudo-label window, and check whether the trained model still completes that hidden side from the single scan.","The approach's dependence on calibrated multi-modal sensors with known ego-poses could limit transfer to monocular or uncalibrated setups, but any sequence with decent calibration is a candidate source of training signal."],"forward_implications":["Zero-shot Panoptic Scene Completion becomes feasible without any labeled 3D data, so a user can prompt the model with a custom vocabulary at test time rather than being limited to fixed dataset classes.","Raw, unlabeled camera-Lidar sequences are now a sufficient training resource, meaning the approach can scale to new cities and sensor setups without an annotation campaign.","The model's completed instance shapes can serve as the basis for amodal 3D object detection, as demonstrated qualitatively by fitting bounding boxes to recognized instances.","Because partial pseudo-labels are enough to learn full shapes, future pipelines can trade tracking horizon for cheaper pseudo-labeling, especially when combined with label-refinement steps.","The main gap to fully supervised methods is concentrated in zero-shot recognition and rare classes, so stronger vision-language features or distribution-aware clustering should transfer directly to higher PQ†."],"supporting_citations":[{"why":"Supplies the video-object segmentation model that propagates masks across the temporal window, generating the masklets that are lifted into Lidar.","marker":"Ravi et al., 2024"},{"why":"Provides the CLIP embedding space used both for per-instance semantic features in pseudo-labels and for zero-shot text prompting at test time.","marker":"Radford et al., 2021"},{"why":"Supplies the sparse generative decoder and transformer instance decoder that CAL is built on, and the fully supervised PaSCo baseline that CAL's performance is compared against.","marker":"Cao et al., 2024"},{"why":"Provides the SemanticKITTI dataset, the completion benchmark formulation, and the ego-poses and calibration used for temporal aggregation.","marker":"Behley et al., 2019"},{"why":"Provides the SSCBench-KITTI360 benchmark and ground-truth labels used for the second zero-shot evaluation.","marker":"Li et al., 2024"},{"why":"Supplies the per-frame 3D mask refinement procedure and the text-prompt augmentation strategy used for zero-shot recognition.","marker":"Osep et al., 2024"},{"why":"Provides the dense CRF inference used to propagate instance pseudo-labels into occupied-but-unlabeled voxels.","marker":"Krähenbühl & Koltun, 2011"},{"why":"Supplies the image segmentation backbone that proposes object masks in the reference frame before video tracking.","marker":"Kirillov et al., 2023"}],"fun_headline_variants":[],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything depends on the video segmentation model tracking every object consistently across the temporal window, because identity switches and missed detections create the incomplete or noisy pseudo-labels the network must learn from.","fun_headline_variants_meta":{"error":"DeepSeek 429: {\"error\":{\"message\":\"Too many requests. Your current concurrency is 137, which exceeds your concurrency limit of 117 based on your remaining balance. Please top up your balance to restore your concurrency.\",\"type\":\"rate_limit_error\",\"param\":null,\"code\":\"invalid_request_error\"}}"},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:34:22.476960+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a static car whose far side is never visible in any camera frame of its pseudo-label window, then compare CAL's completed far-side voxels against the accumulated 360-degree Lidar ground truth; if the model does not infer the hidden side, the claim that partial pseudo-labels yield full amodal shapes is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SemanticKITTI dataset, the completion benchmark formulation, and the ego-poses and calibration used for temporal aggregation."}],"review_version":1}