{"id":"9cae8a30-8ce2-4c46-b29d-003e885ea77a","arxiv_id":"2411.19235","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"InstanceGaussian reports higher 3D instance segmentation accuracy than prior Gaussian-based methods by sharing semantic features across scaffold anchors and merging over-segmented clusters.","lead":"InstanceGaussian attaches a shared semantic label to groups of 3D Gaussians instead of one label per Gaussian, then clusters and merges those groups into whole objects. It reports top results for category-agnostic and open-vocabulary 3D instance segmentation on ScanNet and LeRF benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Appendix A.3 MLP smoothing step may be part of the evaluated ScanNet pipeline; if so, the reported SOTA numbers may not reflect the proposed method as described.","rationale":"The reader's verdict is CONDITIONAL, and I agree that the paper has several addressable issues. However, the single most load-bearing concern is different from the reader's weakest assumption about shared child-Gaussian features. That architectural choice is a plausible design trade-off and is at least tested indirectly by the ablations. The more serious threat is the undisclosed and unablated MLP smoothing step in Appendix A.3. Because it is described in the dataset-setting section and appears to infer the very segmentation labels used for the point-level metrics, it could invalidate the reported comparisons if it is part of the evaluation. The paper provides no code, so this ambiguity cannot be resolved by the reader. The paper deserves credit for internal ablations (Tables 4-5), hyperparameter robustness studies (Tables 10-12), and comparisons to additional baselines in the appendix, which show the method is internally consistent. But the central SOTA claim cannot be verified without clarification of this MLP step and an ablation demonstrating that the reported metrics are produced by the method as described in Sec. 3. Therefore, an UNVERDICTED verdict is more honest than CONDITIONAL, since acceptance should wait until the evaluation pipeline is transparently specified and the effect of the MLP step is quantified.","tokens_in":23068,"tokens_out":6248,"duration_ms":51647,"concrete_test":"Re-run the ScanNet experiments exactly as described in Sec. 3 with the Appendix A.1-A.2 training protocol, but disable the Appendix A.3 MLP smoothing step and evaluate the raw aggregation output of Sec. 3.3 for every point. Compare instance and semantic mIoU on the same 10 scenes used in Tables 1-2. If the scores drop appreciably (e.g., instance mIoU falls below OpenGaussian's 27.32), the SOTA claim depends on an unablated post-processing step. Additionally, request that the authors confirm in a revised draft whether the MLP is used only for visualization, and include a with/without MLP ablation on the reported metrics.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim of SOTA category-agnostic, open-vocabulary point-level segmentation rests on Tables 1-3. Appendix A.3 (\"ScanNet Dataset Setting\") describes an additional post-processing step: \"we identified well-optimized point clouds based on their contributions during rendering and utilized them to train a lightweight MLP. The MLP takes position and color as inputs and predicts segmentation labels as outputs. Subsequently, the trained MLP is employed to infer segmentation results for the point clouds.\" This text appears in the experimental setup, not in a visualization-only section, so it plausibly alters the point-level labels on which mIoU and mAcc are computed. The step is not described in the main method (Sec. 3), is not included in any ablation, and no code is released to confirm whether it is in the evaluation loop. If the MLP is used for evaluation, the reported gains over OpenGaussian could stem from appearance-based label smoothing rather than the proposed joint representation and bottom-up aggregation, making the results not attributable to the method as described. At minimum, the authors must state explicitly whether the MLP output or the raw Sec. 3.3 output is used for evaluation, and provide an ablation over this step.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes InstanceGaussian, a 3D Gaussian Splatting-based method for joint appearance-semantic representation and bottom-up category-agnostic instance segmentation. It introduces a Semantic-Scaffold-GS representation in which anchor-level instance features are shared among child Gaussians, a progressive appearance-semantic joint training strategy, and a graph-connectivity aggregation algorithm built on farthest-point-sampling over-segmentation. Experiments on ScanNet and LeRF report large improvements over Gaussian-based baselines, including OpenGaussian and GaussianGrouping, for instance segmentation, open-vocabulary point cloud understanding, and object selection/rendering.","tokens_in":23321,"tokens_out":3212,"duration_ms":28481,"significance":"If the reported results are reproducible and correctly attributed, the paper makes a useful advance: the anchor-level shared instance feature and the bottom-up aggregation are clean, simple ideas that improve both accuracy and boundary quality over prior Gaussian-based methods. The paper includes helpful ablations (Tables 4, 5, and supplementary C), a hyperparameter robustness study (Table 10), a failure-case analysis, and a comparison with additional baselines in the supplementary material. These are strengths. However, the strength of the central 'state-of-the-art' claim is currently weakened by an ambiguously described post-processing step in the ScanNet evaluation pipeline and by the absence of variance estimates in the main tables.","major_comments":[{"comment":"The MLP smoothing step described in Appendix A.3 ('we identified well-optimized point clouds based on their contributions during rendering and utilized them to train a lightweight MLP... the trained MLP is employed to infer segmentation results for the point clouds') is potentially load-bearing for Tables 1, 2, 4, and 5, since those tables report ScanNet point-level mIoU/mAcc. The manuscript does not state whether the reported numbers come from the raw Sec. 3.3 output or from the MLP-inferred labels, and there is no ablation of this step. The authors must explicitly clarify the evaluation protocol and, if the MLP output is used, provide an ablation that isolates its contribution. Without this, the reported gains over OpenGaussian cannot be attributed to the proposed representation and aggregation method.","section":"Appendix A.3"},{"comment":"All main results are single runs without variance, yet the algorithm depends on randomized FPS initialization and k-means. The paper's own supplementary Table 11 reports that five FPS random initializations yield semantic segmentation mIoU 39.44±1.38 and instance segmentation mIoU 48.57±0.98, indicating that the point estimates in the main tables (for example 50.27 in Table 1) have nontrivial uncertainty. The authors should report mean±std over at least a few runs for the main tables, or explicitly justify why single runs are sufficient.","section":"Tables 1-3, C.3"},{"comment":"The comparison set is too narrow for the claim of state-of-the-art performance. Related works discussed in Sec. 2, including SAGA, GAGA, ClickGaussian, and Feature3DGS, are not evaluated in the main experiments; only the supplementary material (Table 7) adds GAGA and SAGA, without explaining why these are absent from the main comparison or how their metrics were obtained. The authors should either include all feasible Gaussian-based baselines in the main tables or justify their exclusion in terms of input modalities, task definitions, or code availability.","section":"Sec. 4.2, Table 2; Sec. 4.3, Table 3"},{"comment":"The central architectural assumption that five child Gaussians share a single anchor-level instance feature is never ablated. The paper does not report results for different child counts (e.g., 1, 3, 10) or for a variant where each child has an independent instance feature. Since this assumption directly affects boundary precision at object edges, the authors should include an ablation over child count or an independence variant to support the claim that the shared anchor feature is beneficial.","section":"Sec. 3.2(1)"}],"minor_comments":[{"comment":"The paper says in Sec. 4.1 'the 10 scenes selected by OpenGaussian' but in A.3 says 'We randomly selected 10 scenes from ScanNet for evaluation.' These statements are inconsistent and should be reconciled.","section":"Sec. 4.1 and Appendix A.3"},{"comment":"The phrase 'Accurate is measured by mAcc@0.25' is unclear. Please define what mAcc@0.25 means (e.g., whether instances are matched at a 25% IoU threshold) and how it is computed.","section":"Table 1 caption"},{"comment":"The output ordering of Eq. (4) is (f̄, μ̄, Ī) while Algorithm 1 line 4 writes (Ī, f̄, μ̄). Please make the notation consistent.","section":"Eq. (4) and Algorithm 1"},{"comment":"The stage labels '0-10k', '10-20k', '20-30k' should specify the unit (training iterations) and whether these are global steps or per-stage steps.","section":"Fig. 2"},{"comment":"Typo: 'Comparation' should be 'Comparison'.","section":"Table 6 caption"}],"recommendation":"major_revision","confidential_remarks":"The MLP smoothing step in Appendix A.3 is the most serious issue. If the authors clarify the protocol and provide an ablation, the paper may be viable, but the current text does not allow a reviewer to verify that the reported SOTA numbers are due to the proposed method rather than a post-hoc label-smoothing classifier. I also recommend requiring multi-run statistics in the main tables, as the single-run point estimates are not commensurate with the paper's own reported randomness. The paper is otherwise a reasonable contribution to the 3D Gaussian-based perception literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Haijie Li et al. report large gains in 3DGS instance and open-vocabulary segmentation with a plausible combination of existing ingredients. The new bit is sharing a single instance feature across the five child Gaussians decoded from each Scaffold-GS anchor, then running FPS over-segmentation plus graph connectivity merging instead of OpenGaussian's fixed k-means. That is a real, if incremental, contribution, and the ablations in Tables 4-5 and the hyperparameter robustness section in the appendix support it.\n\nWhat the paper gets right: the progressive appearance-semantic training schedule is sensible, the truncated contrastive loss is a reasonable fix for OpenGaussian's instability, and the authors are honest about failure modes (large objects, SAM failures). The GraspNet generalization result is nice bonus evidence.\n\nThe soft spot is load-bearing. Appendix A.3 describes training a lightweight MLP on \"well-optimized point clouds\" and using it to \"infer segmentation results for the point clouds,\" with the stated purpose of enhancing visual quality. This appears in the experimental setup for ScanNet, not in a visualization-only section. If that MLP output is what feeds the mIoU/mAcc tables, then the reported SOTA numbers are not for the method in Section 3.3 and the comparison with OpenGaussian is unfair. The authors need to state plainly whether the MLP is in the evaluation loop, and if it is, ablate it and report both versions. This is not an attack on the core idea; it is a request to know what was measured.\n\nSecondary issues: all main tables are single runs on 10 scenes, with no error bars, and several cited baselines (GAGA, SAGA) appear only in an appendix table with missing metrics. The shared-anchor feature assumption, that five child Gaussians always form one semantic unit, is untested. These are minor if the MLP question is answered.\n\nBottom line: this deserves a serious referee. The component combination is new, the method is coherent, and the evidence is suggestive but not yet clean. I would send it to review with a request for major revision, insisting on a clear statement plus ablation of the MLP step, per-scene breakdowns or variance, and ideally code. If the MLP is only a visualization aid, I'd expect the paper to come out of review in decent shape.","headline":"Plausible incremental method with a serious evaluation-pipeline ambiguity (the MLP smoothing step in Appendix A.3) that must be resolved before the SOTA claim is trusted.","tokens_in":23849,"tokens_out":2976,"would_cite":false,"duration_ms":24808,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"InstanceGaussian jointly learns appearance and instance features in 3D Gaussian Splatting and aggregates instances bottom-up, reaching state-of-the-art category-agnostic segmentation on ScanNet.","keywords":["3D Gaussian Splatting","instance segmentation","open-vocabulary 3D understanding","Semantic-Scaffold-GS","bottom-up instance aggregation","appearance-semantic joint learning","category-agnostic segmentation"],"falsifier":"Run the same pipeline with per-child instance features instead of a shared anchor feature and compare boundary-region segmentation quality on ScanNet; if per-child features improve boundary IoU, the shared-feature assumption is the limiting factor. Alternatively, take a scene where all SAM masks are correct and check whether the large-object aggregation failure persists, which would rule mask quality out as the cause.","tokens_in":22879,"feed_emoji":"🎯","tokens_out":9308,"duration_ms":73250,"temperature":0.7,"pith_summary":"InstanceGaussian tries to make 3D Gaussian Splatting scenes usable for perception, not just rendering, by learning appearance and instance semantics in one representation. It argues that many Gaussians are needed for texture but only one semantic label is needed per region, so it gives each scaffold anchor a single instance feature shared by its five child Gaussians. It couples this with a progressive training schedule that starts with appearance, then adds semantics, then trains both together, and with a bottom-up aggregation step that over-segments the scene and merges sub-objects by feature similarity and spatial adjacency. The payoff, if the method is right, is category-agnostic instance segmentation and open-vocabulary text queries directly on 3D points, with reported gains over prior Gaussian-based approaches on ScanNet.","feed_headline":"Anchor-shared semantics lift 3D instance segmentation to 50.27 mIoU","feed_subtitle":"Five child Gaussians share one instance feature, and graph aggregation merges sub-objects into complete objects.","key_machinery":"The load-bearing object is the anchor-level shared instance feature. In Scaffold-GS, anchors generate child Gaussians; InstanceGaussian gives each anchor one instance feature f that all five of its children inherit, so semantics is learned once per region while appearance remains per-child. The second mechanism is the aggregation graph: nodes are the over-segmented sub-objects from farthest-point-sampling plus k-means, and an edge exists when the L2 distance between sub-object features is below a threshold and the sub-objects share a voxel in a voxelized space; connected components then define the final instances. The third mechanism is the progressive schedule, which starts with appearance-only training, then separates appearance and semantic optimization, then jointly trains both, preventing semantic features from being fitted to a frozen appearance field.","core_discovery":"On the paper's own terms, the central discovery is that separating the granularity of appearance from the granularity of semantics inside a Gaussian scene removes a bottleneck in Gaussian-based perception. In the proposed Semantic-Scaffold-GS representation, each parent anchor holds an appearance embedding and one instance feature, and the five child Gaussians decoded from that anchor share the feature while keeping their own color, position, scale and rotation. Appearance and semantics are then trained in three phases so that neither is frozen while the other learns, using an intra-mask smoothness loss and a contrastive loss truncated at a distance threshold to avoid destabilizing joint training. Instance segmentation is produced bottom-up: farthest point sampling seeds about a thousand sub-object clusters, k-means groups the Gaussians in position-plus-feature space, and a graph whose edges combine feature distance with voxel adjacency merges sub-objects into complete objects with no preset category or object count. On the ten ScanNet scenes used by the baseline, the paper reports 50.27 instance mIoU against 27.32 for OpenGaussian, and 47.94 versus 38.29 for ten-class open-vocabulary semantic segmentation.","pith_inferences":["The paper leaves the shared-per-anchor feature untested: it does not ablate the child count, so a per-child feature variant could preserve or improve boundary precision when anchors cross object edges.","The bottom-up aggregation is modular and could be applied to other Gaussian feature fields, isolating how much of the gain comes from aggregation versus the joint representation.","The reported large-object failure points to a concrete extension: weight the smoothness loss by cross-view observation completeness so rarely seen parts are not pushed apart by the contrastive term.","A mask-free self-supervised consistency signal could replace SAM supervision in scenes where masks are unreliable, testing how much the method depends on mask quality."],"forward_implications":["Category-agnostic instance segmentation on Gaussians no longer requires knowing how many objects are in the scene; the graph aggregation step determines the count adaptively.","A single trained scene supports rendering, instance segmentation, and open-vocabulary text retrieval from the same learned features, as the paper's LeRF object-selection experiments demonstrate.","The method inherits a dependency on SAM mask quality, because the instance-feature losses are supervised by those masks; the paper reports that frequent SAM failures degrade segmentation accuracy.","Training time increases relative to the baselines (about 1847 seconds per ScanNet scene in the paper's measurements) because appearance attributes are decoded through MLPs."],"supporting_citations":[{"why":"Supplies the anchor-to-child scaffold structure that InstanceGaussian extends with a shared instance feature.","marker":"[22]"},{"why":"Provides the instance-feature smoothness and contrast losses, the main Gaussian baseline to beat, and the 2D-mask-to-3D-instance association used for open-vocabulary queries.","marker":"[37]"},{"why":"Supplies the SAM masks that supervise the instance-feature losses.","marker":"[20]"},{"why":"Provides the ScanNet scenes and ground-truth instance annotations on which the main comparisons are run.","marker":"[9]"},{"why":"Defines 3D Gaussian Splatting, the base rendering representation the method builds on.","marker":"[18]"},{"why":"LangSplat is a language-embedded Gaussian baseline compared in open-vocabulary segmentation and object selection.","marker":"[28]"},{"why":"LEGaussians is a language-embedded Gaussian baseline compared in the same tasks.","marker":"[32]"},{"why":"MaskClustering is a depth-assisted reference method whose results frame the paper's comparisons and whose mask-graph idea relates to the graph aggregation.","marker":"[40]"},{"why":"Supplies the LeRF dataset used to evaluate open-vocabulary object selection and rendering.","marker":"[19]"}],"fun_headline_variants":["Joint appearance-semantic Gaussians hit 50.27 mIoU on ScanNet","InstanceGaussian: category-agnostic joint 3D instance segmentation","Decouple appearance and semantics for better 3D instance boundaries","Shared semantics per anchor: bottom-up 3D instance aggregation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes each scaffold anchor lies within a single object, so the shared instance feature inherited by its five child Gaussians cannot blur object boundaries; an anchor straddling two objects sacrifices boundary precision.","fun_headline_variants_meta":{"raw":{"variants":["Joint appearance-semantic Gaussians hit 50.27 mIoU on ScanNet","InstanceGaussian: category-agnostic joint 3D instance segmentation","Decouple appearance and semantics for better 3D instance boundaries","Shared semantics per anchor: bottom-up 3D instance aggregation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000583,"raw_usage":{"total_tokens":2802,"prompt_tokens":1066,"completion_tokens":1736,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":682,"completion_tokens_details":{"reasoning_tokens":1658}},"tokens_in":682,"tokens_out":1736,"duration_ms":12979,"temperature":1.0,"reasoning_tokens":1658,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:22:33.069506+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline with per-child instance features instead of a shared anchor feature and compare boundary-region segmentation quality on ScanNet; if per-child features improve boundary IoU, the shared-feature assumption is the limiting factor. Alternatively, take a scene where all SAM masks are correct and check whether the large-object aggregation failure persists, which would rule mask quality out as the cause.","supporting_citations":[{"cited_title":"Scaffold-gs: Structured 3d gaussians for view-adaptive rendering","cited_arxiv_id":null,"evidence_quote":"Supplies the anchor-to-child scaffold structure that InstanceGaussian extends with a shared instance feature."},{"cited_title":"Opengaussian: Towards point-level 3d gaussian-based open vocabulary understand- ing","cited_arxiv_id":null,"evidence_quote":"Provides the instance-feature smoothness and contrast losses, the main Gaussian baseline to beat, and the 2D-mask-to-3D-instance association used for open-vocabulary queries."},{"cited_title":"Segment any- thing","cited_arxiv_id":null,"evidence_quote":"Supplies the SAM masks that supervise the instance-feature losses."},{"cited_title":"Scannet: Richly-annotated 3d reconstructions of indoor scenes","cited_arxiv_id":null,"evidence_quote":"Provides the ScanNet scenes and ground-truth instance annotations on which the main comparisons are run."},{"cited_title":"3d gaussian splatting for real-time radiance field rendering","cited_arxiv_id":null,"evidence_quote":"Defines 3D Gaussian Splatting, the base rendering representation the method builds on."},{"cited_title":"Langsplat: 3d language gaussian splatting","cited_arxiv_id":null,"evidence_quote":"LangSplat is a language-embedded Gaussian baseline compared in open-vocabulary segmentation and object selection."},{"cited_title":"Language embedded 3d gaussians for open- vocabulary scene understanding","cited_arxiv_id":null,"evidence_quote":"LEGaussians is a language-embedded Gaussian baseline compared in the same tasks."},{"cited_title":"Maskclus- tering: View consensus based mask graph clustering for open-vocabulary 3d instance segmentation","cited_arxiv_id":null,"evidence_quote":"MaskClustering is a depth-assisted reference method whose results frame the paper's comparisons and whose mask-graph idea relates to the graph aggregation."},{"cited_title":"Lerf: Language embedded radiance fields","cited_arxiv_id":null,"evidence_quote":"Supplies the LeRF dataset used to evaluate open-vocabulary object selection and rendering."}],"review_version":1}