{"id":"9d97c619-0695-454b-a2d9-4731a24a86d4","arxiv_id":"2411.19551","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FreeGS bootstraps view-consistent semantics and instance indices in 3D Gaussian Splatting without needing 2D masks.","lead":"FreeGS attaches semantic features and instance indices to 3D Gaussians and trains them without task-specific 2D labels by alternating between 3D clustering and CLIP-based distillation. It reports near-parity with supervised methods on segmentation benchmarks while skipping SAM-based preprocessing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Bootstrapping hinges on unsupervised HDBSCAN cluster quality, which is never evaluated; erroneous clusters can self-reinforce and invalidate view-consistency.","rationale":"The reader's weakest assumption identifies the union-space clustering in Section 3.2 as the load-bearing component, and I agree: the entire self-bootstrapping strategy depends on cluster pseudo-labels being accurate enough to guide semantic distillation. My concern sharpens this by pointing to the specific mechanisms that can produce and lock in errors: random initial semantics, unspecified handling of HDBSCAN noise points, and contrastive/smoothing losses that pull features within wrong clusters together rather than correcting the cluster boundaries. The per-dataset tuning of min_samples further suggests that clustering quality is not robust across scenes. This concern is not resolved by the paper's ablations (Table 5), which only remove entire components and do not measure clustering accuracy or its trajectory. Therefore, the central claim of view-consistent 3D understanding without labels remains conditional on clustering success in the tested scenes. The reader's conditional verdict is appropriate; I would not change it, but I would add a required experiment: tracking cluster pseudo-label accuracy against ground truth across training iterations, and an oracle-clustering ablation to bound the achievable performance. This would directly test whether the bootstrapping improves over time or merely reinforces initial errors.","tokens_in":12622,"tokens_out":5787,"duration_ms":54672,"concrete_test":"During the second training phase on LERF-Mask 'figurines' and ScanNet scene0060, render the ID map at 500-iteration intervals and compute instance-level IoU between the cluster pseudo-masks and ground-truth object masks. Plot this over training; if IoU does not improve significantly from its initial value, or if the 'red apple' versus 'green apple' confusion persists, the bootstrap is not correcting clustering errors. Then run the full pipeline with HDBSCAN clusters replaced by ground-truth instance masks as an oracle for the instance-level distillation and contrastive losses, and compare final mIoU. A large performance gap between the oracle and the HDBSCAN-based model would isolate clustering quality as the bottleneck and confirm the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the union-space HDBSCAN clustering in Section 3.2 produces sufficiently accurate object groupings from initially random semantic features, and that the resulting pseudo-masks improve rather than corrupt the distilled semantics. This is the engine of the bootstrap, but three concrete gaps undermine it. First, at the start of the second training phase, the semantic features fi are randomly initialized, so the first clustering is driven almost entirely by position and color, which cannot separate objects with similar appearance or interleaved geometry. Second, HDBSCAN assigns a noise label to low-density points, yet the paper never specifies how noise Gaussians are handled when forming the one-hot instance index d_i or rendering the ID map via Eq. (5); if noise points are treated as a single group, they can pollute the argmax map with spurious instance indices. Third, once a cluster is wrong, both the smoothing loss (Eq. 3) and the contrastive loss (Eq. 8) pull features within that wrong cluster together, and the instance-level distillation (Eq. 6) feeds the wrong mask to CLIP, creating a self-reinforcing error loop. The paper provides no quantitative evaluation of clustering quality, no analysis of how cluster accuracy evolves during training, and no sensitivity study for the HDBSCAN min_samples parameter (set to 20 on LERF-Mask but 60 on 3D-OVS and ScanNet) or for the relative weighting of position, color, and semantic subspaces. Without such evidence, the reported mIoU numbers may reflect scenes where clustering happens to succeed, and the view-consistency claim is not established for general scenes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"FreeGS proposes an unsupervised 3D Gaussian Splatting framework for view-consistent scene understanding. The core idea is an IDentity-coupled Semantic Field (IDSF) that augments each Gaussian with a semantic vector and a scalar instance index. Training alternates between HDBSCAN clustering of Gaussians in a position-color-semantics union space to obtain pseudo-instance indices, and multi-level 2D semantic distillation from frozen CLIP/MaskCLIP features guided by these indices; a 2D-3D contrastive loss is added to stabilize the loop. The paper evaluates on LERF-Mask, 3D-OVS, and ScanNet for novel-view segmentation, 3D detection, and interactive selection, reporting performance comparable to SAM-supervised baselines while eliminating SAM preprocessing.","tokens_in":12970,"tokens_out":7276,"duration_ms":58636,"significance":"If the bootstrapping mechanism is sound, FreeGS is a meaningful advance: it removes the SAM-mask preprocessing bottleneck, introduces an explicit view-consistent instance index in 3D, and unifies several tasks in one field. The paper ships code and includes ablations showing that each component contributes on the tested scenes. The significance is currently tempered by the absence of a direct view-consistency metric, the lack of clustering-quality analysis, and the small evaluation scale; these gaps leave the central claim only partially substantiated.","major_comments":[{"comment":"The handling of HDBSCAN noise labels is unspecified. HDBSCAN assigns a noise label to low-density points, but the paper never states how such Gaussians are treated when constructing the one-hot vector d'_i or when rendering the ID map in Eq. (5). If noise points are assigned a common index, they will appear as a spurious instance in the argmax map; if they are excluded, the definition of the rendered index and of the masks M_i in Eq. (6) is incomplete. This is not a cosmetic detail: Eq. (6) and Eq. (8) use these masks and indices directly, so the ambiguity affects the core bootstrapping loop.","section":"Section 3.2, Eq. (5)"},{"comment":"The paper provides no evaluation of clustering quality or of bootstrap error propagation, despite the method's reliance on union-space clustering as the engine of the alternating optimization. At the start of the second training phase, the semantic features f_i are not yet trained, so the first clustering is driven almost entirely by position and color; the paper does not show that this initialization is sufficient, nor that the smoothing loss (Eq. 3) and contrastive loss (Eq. 8) improve rather than reinforce initial grouping errors. A quantitative analysis of cluster purity/accuracy and of how the pseudo-masks evolve over training iterations is needed to substantiate the claim that the two steps 'bootstrap view-consistent semantics.'","section":"Section 3.2 and Section 4 (Training)"},{"comment":"The view-consistency claim is not directly measured. The reported metrics (per-frame mIoU for segmentation, 3D box mIoU and recall for detection) evaluate average accuracy, not the stability of instance identities across views. To support the title and abstract, the paper should report a direct consistency measure, for example the mean pairwise IoU of rendered ID maps across different viewpoints, or the fraction of Gaussians assigned to the same cluster when the process is repeated with different training seeds. Without such a measure, the qualitative claim of consistency (e.g., Fig. 3) is not quantified.","section":"Section 4, Tables 1-3"},{"comment":"The evaluation is too small to support the strength of the comparisons, and no variance is reported. The benchmarks comprise 3, 5, and 3 scenes, and differences such as Table 1's 69.5 versus 72.8 and Table 2's 77.0 versus 77.0 are within the noise expected for this sample size. Moreover, the HDBSCAN min_samples hyperparameter is tuned per dataset (20 for LERF-Mask, 60 for 3D-OVS and ScanNet) with no sensitivity analysis, even though clustering is the load-bearing component of the method. At minimum, multiple runs with standard deviations and a sensitivity study for min_samples and the loss weights (lambda_C, lambda_S) should be reported.","section":"Section 4, Implementation Details and Tables 1-3"}],"minor_comments":[{"comment":"'Bootstraping' should be 'Bootstrapping' in the title and in the heading of Section 3.3.","section":"Title and Section 3.3"},{"comment":"'Sapp is the gather of view-independent color part c′' should read 'Sapp is the set of view-independent color components c′'.","section":"Section 3.2"},{"comment":"The notation for the positive and negative sets (F Gi+ and F Gi-) is introduced as sets of mean features, but the overbar on F is not defined; please clarify the notation.","section":"Eq. (8)"},{"comment":"The caption says the LERF results are taken from LangSplat, which conflicts with the text stating that methods were retrained on training views; please clarify the protocol.","section":"Table 2"},{"comment":"The interactive object selection task is presented qualitatively; a short protocol description or a small user study would strengthen the claim that the method supports this task.","section":"Fig. 6"},{"comment":"'Gasussians' should be 'Gaussians'; also, 'min samples' should be 'min_samples' for consistency with the HDBSCAN library naming.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for the conference and the ideas are timely. The main uncertainty is whether the bootstrapping loop is robust enough to support the view-consistency claim; I would encourage the authors to add the missing analyses. A potential concern is that the per-dataset tuning of HDBSCAN min_samples may indicate that the method is sensitive to dataset statistics, which should be examined in a sensitivity study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [name],\n\nQuick take on arXiv:2411.19551: the paper has a genuinely new mechanism—the union-space clustering bootstrapped with alternating 2D distillation—and it delivers results that are competitive with SAM-supervised baselines while skipping preprocessing. That is a real practical contribution. The code is out, the ablations show each component pulls weight, and the writing is honest about limitations. I'd send it to referees.\n\nWhat's new: instead of requiring SAM masks or view-consistent labels, they add a per-Gaussian semantic vector and instance index (IDSF) and optimize them alternately. HDBSCAN clusters Gaussians in position/color/PCA-reduced semantic space to get instance indices; those indices are rendered to 2D to produce pseudo-masks that constrain the CLIP distillation, and a joint 2D-3D contrastive loss stabilizes the loop. It's a clean bootstrapping design, and the efficiency gain over LangSplat is real (141 min preprocessing to zero).\n\nThe soft spots are in the empirical support. The central claim is view consistency, but they never measure it directly—no cross-view segmentation stability metric, no cluster accuracy over training. The clustering quality is the engine of the bootstrap, and it's completely unquantified. The stress-test note about noise points is fair: HDBSCAN assigns noise labels, but the paper never says how those Gaussians are treated in the one-hot index or the ID map. That could pollute masks with a spurious \"noise instance.\" Also, there are only 3-5 scenes per benchmark, no error bars or multiple seeds, and the HDBSCAN min_samples is tuned per dataset (20 vs 60), so the method's robustness to its most important hyperparameter is unknown. Those are real gaps, not fatal flaws: the ablations do show the designed components matter, and the qualitative results look consistent.\n\nOverall: this is a solid method paper with a clever core and a fair shot at being useful. A serious referee should require an added cluster-quality analysis, noise-point handling details, and at least a sensitivity study on min_samples. I'd accept it for review rather than desk-reject, and I'd probably cite it if I were working on label-free 3D scene understanding.","headline":"Clever bootstrapping design that delivers competitive label-free 3D segmentation, but the clustering engine is unquantified and the view-consistency claim outruns the evidence.","tokens_in":13480,"tokens_out":2520,"would_cite":true,"duration_ms":20858,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FreeGS learns view-consistent 3D scene semantics from raw multi-view images alone, with no 2D labels, no SAM masks, and no preprocessing.","keywords":["3D Gaussian Splatting","unsupervised scene understanding","view-consistent semantics","instance clustering","open-vocabulary segmentation","3D object detection","bootstrapping","semantic distillation"],"falsifier":"Take a scene with two visually near-identical, touching objects of the same category, and run FreeGS from unlabeled multi-view images. If the union-space clustering initially merges them into one instance and the bootstrapped distillation plus contrastive loss never separates them over the full 7k-iteration second phase, the central claim that clustering and semantics bootstrap toward view-consistent instances fails on that scene. A quantitative version: track cluster purity against ground-truth instance masks during training; the paper's claim predicts a monotone improvement, so a non-increasing or oscillating purity trajectory would contradict it.","tokens_in":12460,"feed_emoji":"🧩","tokens_out":9185,"duration_ms":66285,"temperature":0.7,"pith_summary":"FreeGS claims that a 3D Gaussian scene representation can learn view-consistent semantics and instance identities from raw multi-view images alone, with no 2D labels, no SAM masks, and no tracking-based preprocessing. The paper argues that alternating between clustering Gaussians in a joint geometry–appearance–semantics space and distilling CLIP features into the resulting instance groups lets the two tasks improve each other. If the claim holds, open-vocabulary novel-view segmentation, text- and click-driven object selection, and 3D object detection all become possible using only the images and camera poses already needed for reconstruction. The reported experiments show performance comparable to supervised counterparts on public benchmarks while removing the preprocessing step entirely.","feed_headline":"Zero labels: bootstrapped clustering teaches 3D scenes semantics","feed_subtitle":"An alternating loop of 3D clustering and 2D distillation matches supervised methods without SAM preprocessing.","key_machinery":"The identity-coupled semantic field (IDSF) is the central object: each Gaussian stores a semantic vector $f_i \\in \\mathbb{R}^{128}$ and a scalar instance index $d_i$, so identity and semantics live on the same primitive. The engine is the alternating bootstrap. Union-space clustering runs HDBSCAN on the concatenation of normalized position, color, and PCA-reduced semantics ($S_{\\text{union}} = S_{\\text{pos}} \\cup S_{\\text{app}} \\cup S_{\\text{sem}}$) to produce instance indices; the indices are $\\alpha$-blended and argmaxed into a 2D ID map, refined by DenseCRF, and used to form instance-level CLIP supervision. The multi-level distillation loss $L_F = \\|F_{\\text{pix}} - \\hat{F}\\|_1 + \\gamma \\|F_{\\text{ins}} - F\\|_1$ injects pixel- and instance-level semantics, and the 2D–3D joint contrastive loss $L_C$ re-merges fragments by treating similar groups as positive samples. Feature-field smoothing $L_S$ over $K$-nearest neighbors keeps the field continuous, and together these losses close the loop that lets clustering and semantics refine each other.","core_discovery":"The central discovery is that cross-view instance consistency can be bootstrapped rather than imported from 2D models. FreeGS attaches to each Gaussian an identity-coupled semantic field: a view-independent semantic vector plus a scalar instance index. A clustering pass groups Gaussians in a union space of position, color, and PCA-reduced semantics, assigning each Gaussian an instance index; a distillation pass renders those indices to 2D, uses the resulting masks to extract CLIP features per instance, and aligns them with the rendered semantic field. A 2D–3D joint contrastive loss pulls features of the same clustered group together and pushes different groups apart. The paper argues that this alternating schedule yields view-consistent semantics without any label, because the 3D grouping regularizes the 2D semantic injection and the semantic field in turn improves the grouping.","pith_inferences":["A natural stress test is a scene with two visually near-identical adjacent objects: the union-space clustering has no appearance or semantic signal to separate them initially, so whether the bootstrapping loop can ever split them is an open question not addressed by the chosen benchmarks.","The union-space bootstrapping principle is not tied to Gaussian splatting; any explicit 3D representation with a differentiable renderer and a 2D feature extractor could in principle run the same alternating clustering–distillation loop.","The paper fixes PCA dimension and HDBSCAN min_samples without a sensitivity study; measuring how cluster purity and downstream mIoU vary with those hyperparameters would directly test how much of the result rests on the clustering configuration.","The authors acknowledge 3D detection accuracy has significant room for improvement, so the framework's most immediate practical value is likely segmentation and object selection, with precise 3D localization as the next target for the same bootstrapping idea."],"forward_implications":["On LERF-Mask and 3D-OVS, FreeGS matches the segmentation mIoU of SAM-supervised methods like Gaussian Grouping and LangSplat, without generating any SAM masks.","The same trained field answers text and click prompts by matching in 3D, so novel-view segmentation, object selection, and detection share one inference pipeline.","Removing preprocessing cuts the total pipeline from 92–246 minutes to about 62 minutes on the ramen scene, making the semantic field practical for quick 3D capture.","Because instance indices are stored per Gaussian, 3D editing and prompt-based selection become direct manipulations of the cluster rather than pixel-space postprocessing.","The contrastive loss stabilizes cluster assignments across training iterations, which is what the paper credits for avoiding fragmented objects in cluttered scenes."],"supporting_citations":[{"why":"Supplies the 3D Gaussian Splatting representation and differentiable tile-based rasterizer that FreeGS extends with semantic fields.","marker":"(Kerbl et al. 2023)"},{"why":"Provides HDBSCAN, the hierarchical density clustering algorithm used to group Gaussians in the union space.","marker":"(McInnes et al. 2017)"},{"why":"CLIP is the vision-language model whose features are distilled into Gaussians at pixel and instance levels.","marker":"(Radford et al. 2021)"},{"why":"Provides MaskCLIP, the feature extractor used for pixel-level semantic distillation.","marker":"(Dong et al. 2023)"},{"why":"Provides FeatUP, used to super-resolve CLIP features to the pixel resolution needed by the distillation loss.","marker":"(Fu et al. 2024)"},{"why":"LERF is the NeRF-based baseline and supplies the relevancy-scoring scheme used for text-query matching.","marker":"(Kerr et al. 2023)"},{"why":"Gaussian Grouping is the 3DGS baseline requiring predefined view-consistent labels; FreeGS compares against it and removes that preprocessing.","marker":"(Ye et al. 2023)"},{"why":"LangSplat is the CLIP-in-3DGS baseline with SAM masks; it is the main comparison for segmentation and 3D detection.","marker":"(Qin et al. 2023)"}],"fun_headline_variants":["FreeGS: no labels, no SAM, just bootstrapped clustering","Bootstrapping 3D Gaussians yields view-consistent semantics","Zero-label 3D semantics via alternating clustering and distillation","View-consistent 3D understanding without 2D supervision","Self-supervising 3D scenes by clustering and distilling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that HDBSCAN clustering of position, color, and partially trained semantics produces correct object groupings, and that masks derived from those groupings give supervision that improves the semantics rather than locking in initial clustering errors.","fun_headline_variants_meta":{"raw":{"variants":["FreeGS: no labels, no SAM, just bootstrapped clustering","Bootstrapping 3D Gaussians yields view-consistent semantics","Zero-label 3D semantics via alternating clustering and distillation","View-consistent 3D understanding without 2D supervision","Self-supervising 3D scenes by clustering and distilling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1602,"prompt_tokens":1009,"completion_tokens":593,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":503}},"tokens_in":625,"tokens_out":593,"duration_ms":5075,"temperature":1.0,"reasoning_tokens":503,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:04:03.890012+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a scene with two visually near-identical, touching objects of the same category, and run FreeGS from unlabeled multi-view images. If the union-space clustering initially merges them into one instance and the bootstrapped distillation plus contrastive loss never separates them over the full 7k-iteration second phase, the central claim that clustering and semantics bootstrap toward view-consistent instances fails on that scene. A quantitative version: track cluster purity against ground-truth instance masks during training; the paper's claim predicts a monotone improvement, so a non-increasing or oscillating purity trajectory would contradict it.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides MaskCLIP, the feature extractor used for pixel-level semantic distillation."},{"cited_title":"M.; Goldberg, K.; Kanazawa, A.; and Tancik, M","cited_arxiv_id":null,"evidence_quote":"LERF is the NeRF-based baseline and supplies the relevancy-scoring scheme used for text-query matching."}],"review_version":1}