{"id":"8830dcdd-6a67-49ae-9f39-c8852acb26fa","arxiv_id":"2411.18066","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"GLS jointly optimizes 3D Gaussian surface reconstruction and open-vocabulary segmentation, reporting improved accuracy on MuSHRoom, ScanNet++ and LERF-OVS.","lead":"This paper presents GLS, a 3D Gaussian Splatting system that reconstructs indoor surfaces and segments objects from free-form text prompts at the same time. It combines geometric normal priors with CLIP and DEVA semantic cues, so that each task helps regularize the other.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported results are internally inconsistent across Tables 1, 2, 4, and 7, so the central SOTA claim is not verifiable as printed.","rationale":"The reader's verdict is CONDITIONAL, and my reading supports that conditionality without escalating it. The reader's stated weakest assumption is the unproven geometric partition behind Ld (Sec. 4.3, Eq. 7); I share that concern, but I regard the internal numerical inconsistencies as the primary load-bearing issue because the paper's central claim is explicitly an empirical SOTA claim. If the tables cannot be reconciled, the claim is unverifiable regardless of how sound the loss design is. The inconsistencies are concrete: Table 4 'All' should equal the full w/o-sensor-depth model, but does not match Table 1; Table 7 per-scene averages do not reproduce Table 1; and the 'kokko' row appears to repeat four of five values from the ScanNet++ row in Table 2. These are exactly the kind of errors that a released codebase and per-scene results would resolve. The paper's contribution remains plausible: the joint optimization is well motivated, and the ablations qualitatively show each loss contributes. But without consistent numbers and code, the SOTA claim should not be accepted at face value. Hence I keep the reader's CONDITIONAL verdict unchanged rather than moving to rejection, because the issues are addressable by the authors.","tokens_in":17320,"tokens_out":6342,"duration_ms":52930,"concrete_test":"Ask the authors for the per-scene metric files and rerun the averaging exactly as printed: compute the mean of Table 7's MuSHRoom sensor-depth rows and compare against Table 1; compute Table 7's ScanNet++ rows against Table 2; and compare Table 4 'All' against the matching 'Ours ×' row of Table 1 or Table 2. Also require the caption of Table 4 to state which scene(s) the ablation is averaged over. The headline SOTA claim is settled only when one consistent set of numbers survives this reconciliation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is empirical: GLS surpasses SOTA on MuSHRoom, ScanNet++, and LERF-OVS. The tables supporting that claim do not cohere, so the claim cannot be checked as printed. (i) Table 4 'All' is described as the full model without sensor depth, but its Accuracy 0.0814 and F-score 0.5127 disagree with Table 1 'Ours ×' (Accuracy 0.0538, F-score 0.6922) for what should be the same setting; the caption does not identify Table 4 as a different split. (ii) Averaging the MuSHRoom per-scene rows of Table 7 with sensor depth gives Accuracy 0.0288, Completion 0.0269, Chamfer 0.0310, Normal Consistency 0.8830, and F-score 0.8796, while Table 1 reports 0.0288, 0.0254, 0.0271, 0.8640, and 0.8924; only Accuracy matches. (iii) The 'kokko' row in Table 7 has Completion 0.0272, Chamfer 0.0444, Normal Consistency 0.9064, and F-score 0.8623, identical to the ScanNet++ row of Table 2 (Accuracy differs: 0.0305 vs 0.0640), indicating a transcription or copy-paste error. Separately, the Sec. 4.3 depth-refinement loss Ld partitions normal space into three sectors without a proof of exhaustiveness; pixels with cos(theta') <= 0 fall outside all masks, and normals not lying in the illustrated plane are unaccounted for. The table inconsistencies are the more load-bearing problem for the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GLS, a 3D Gaussian Splatting framework that jointly optimizes indoor surface reconstruction and 3D open-vocabulary segmentation. It introduces three regularization terms: a normal-prior loss Ln that guides rendered normals with a pretrained normal estimator, a semantic-feature-guided normal smoothing loss Ls that uses CLIP features to smooth large surfaces, and a normal-error-guided depth refinement loss Ld that recombines rendered depth, unbiased depth, and alpha masks according to a three-case partition of the ideal normal direction. The segmentation branch supervises Gaussian semantic features with DEVA masks and CLIP features. The paper claims state-of-the-art results on MuSHRoom, ScanNet++, and LERF-OVS, with ablations showing the contribution of each loss term.","tokens_in":17725,"tokens_out":4079,"duration_ms":34155,"significance":"The core idea of coupling geometric and semantic cues through shared 3D Gaussian primitives is timely and could be practically valuable if the reported gains hold. The paper evaluates on public datasets, reports training times, and includes qualitative demonstrations of interactive applications. The main contribution, however, is empirical, and the printed quantitative evidence is not internally consistent. Several tables that should describe the same configuration disagree, and one per-scene row appears to duplicate another dataset's row. These inconsistencies make the headline SOTA claim unverifiable as printed. The geometric derivation of Ld also lacks an exhaustiveness argument. With corrected and reconciled evaluations, the paper could make a useful contribution; as submitted, the evidence requires substantial clarification.","major_comments":[{"comment":"The full model without sensor depth is reported inconsistently: Table 1 lists Accuracy 0.0538, Normal Consistency 0.8357, and F-score 0.6922 for 'Ours ×', while Table 4's 'All' row, described as the full model without sensor depth, reports Accuracy 0.0814, Normal Consistency 0.8474, and F-score 0.5127. If these are the same setting, the discrepancy is unexplained and prevents verification of the central SOTA claim; if Table 4 uses a different split or protocol, the caption or text must state this explicitly.","section":"§5.2 and §5.4, Tables 1 and 4"},{"comment":"Averaging the five MuSHRoom per-scene rows with sensor depth from Table 7 gives Accuracy 0.0288, Completion 0.0269, Chamfer-L1 0.0310, Normal Consistency 0.8830, and F-score 0.8796, whereas Table 1 reports 0.0288, 0.0254, 0.0271, 0.8640, and 0.8924; only Accuracy agrees. The paper must reconcile these numbers or clearly state which table is the official evaluation result, since the discrepancy directly affects the claimed superiority over DN-Splatter and FDS.","section":"§D, Table 7 vs Table 1"},{"comment":"The 'kokko' row in Table 7 (Accuracy 0.0305, Completion 0.0272, Chamfer-L1 0.0444, Normal Consistency 0.9064, F-score 0.8623) is identical in four of five metrics to the ScanNet++ row in Table 2, with only Accuracy differing. This strongly suggests a transcription or copy-paste error and makes the per-scene evaluation unreliable as printed.","section":"§D, Table 7 'kokko' row and §5.2, Table 2"},{"comment":"The three-case partition of the ideal normal n' is not shown to be exhaustive. The conditions M1: cos(α)>cos(θ')>0, M2: cos(α)<0, and M3: 0<cos(α)≤cos(θ') leave uncovered pixels with cos(θ')≤0, and the boundary cos(α)=0 is assigned to none of the masks. In addition, Fig. 4 presents the argument in a two-dimensional plane, while n' is a unit vector on the sphere; directions outside that plane are not accounted for. Since Ld is a core contribution, the paper needs either a rigorous proof of exhaustiveness or an experiment reporting the fraction of pixels falling into each mask and the behavior of the loss on unassigned pixels.","section":"§4.3, Eq. (7) and Fig. 4"}],"minor_comments":[{"comment":"The sentence 'We and the same tool of DN-Splatter...' is ungrammatical and should be rewritten, for example as 'We use the same evaluation tool as DN-Splatter...'.","section":"§5.1"},{"comment":"The word 'rexonstruction' in the last paragraph is a typo and should be 'reconstruction'.","section":"§5.4"},{"comment":"The rendered mask M used in the cross-entropy loss Lm is not defined alongside rendered color, depth, and features; it should be added to Eq. (3) or defined explicitly in the text.","section":"§3, Eq. (3)"},{"comment":"The threshold N_d^T \\hat N < 0.9 for applying Ld is introduced without justification or ablation; a sensitivity analysis or a brief rationale would help.","section":"§4.3"},{"comment":"The LangSplat row reports only five mIoU values and no per-scene mBIoU values despite the header implying both sets of columns; reformat the table to make the missing entries explicit rather than leaving dashes.","section":"Table 3"},{"comment":"The name 'DEV A' should be written as 'DEVA', and the supplementary text contains the typo 'resconstructed' for 'reconstructed'.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The table inconsistencies are serious enough that I could not verify the paper's central SOTA claim from the submitted evidence. I would ask the authors to provide the raw per-scene numbers, reconcile Tables 1, 4, and 7, and address the exhaustiveness of the Ld partition before the paper is reconsidered. The idea is promising and the presentation is otherwise clear, so I see this as a major revision rather than a rejection, provided the quantitative claims can be substantiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Core idea is worth taking seriously: GLS couples indoor surface reconstruction with open-vocabulary segmentation in one 3DGS pipeline, letting segmentation masks smooth geometry and geometry sharpen masks. The two regularizers, Ls (CLIP-guided normal smoothing on large objects) and Ld (normal-error-based depth refinement), are genuinely new, and the ablations show each term helps both tasks. That is a step beyond treating the two problems as separate stacks.\n\nThe numbers, however, do not hold together. Table 1 gives Ours w/o sensor depth on MuSHRoom as Accuracy 0.0538, F-score 0.6922; Table 4's All row, which is the full model without sensor depth, gives 0.0814 and 0.5127. Those should match unless Table 4 is on a different split, and the caption does not say so. Averaging the MuSHRoom per-scene rows of Table 7 with sensor depth gives 0.0288, 0.0269, 0.0310, 0.8830, 0.8796, while Table 1 reports 0.0288, 0.0254, 0.0271, 0.8640, 0.8924. Only Accuracy matches. And the kokko row of Table 7 is identical to the ScanNet++ row of Table 2 (0.0272, 0.0444, 0.9064, 0.8623) except for Accuracy, which looks like a copy-paste error. These are not cosmetic; they make the headline SOTA claim unverifiable as printed. The fixes are straightforward — rerun the evaluation, release code, report variance — but they are on the authors.\n\nThere is also an unproven geometric partition in Ld: the loss assumes the true normal falls into one of three sectors, and the paper never addresses pixels where cos(theta') <= 0 or normals outside the plane of Fig. 4. Minor relative to the tables, but it should be justified or isolated experimentally.\n\nCredit where due: baselines are re-run from source, the appendix is honest about TSDF limitations and noisy DEVA masks, and the qualitative results look plausible. A serious referee should see this, mostly to push for reconciled tables and code. As printed, I would not cite the numbers.","headline":"The joint reconstruction + segmentation idea is genuinely useful, but the headline numbers don't add up: Tables 1, 4, and 7 disagree with each other, so the SOTA claim needs the authors' own tables reconciled before I'd trust it.","tokens_in":18279,"tokens_out":5280,"would_cite":false,"duration_ms":39176,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One 3D Gaussian model can reconstruct indoor surfaces and segment open-vocabulary objects at the same time, and the paper shows the two tasks improve each other.","keywords":["3D Gaussian Splatting","indoor surface reconstruction","open-vocabulary segmentation","surface normal prior","CLIP features","DEVA masks","depth refinement","TSDF fusion"],"falsifier":"On a synthetic indoor scene with ground-truth normals and depth, disable every loss except $L_d$ and measure whether optimized depth moves toward the ground truth for pixels whose true normal lies near or outside the three angular sectors; if those pixels systematically degrade, the three-case partition is not exhaustive.","tokens_in":17114,"feed_emoji":"🏠","tokens_out":7634,"duration_ms":62958,"temperature":0.7,"pith_summary":"The paper proposes GLS, a single 3D Gaussian Splatting framework that performs indoor surface reconstruction and 3D open-vocabulary segmentation together, rather than as separate tasks. Its central claim is that the two tasks share the same goal—sharp, smooth surfaces—so optimizing them jointly improves both: geometric normal priors sharpen object boundaries for segmentation, while semantic CLIP and DEVA cues smooth texture-less and reflective surfaces for reconstruction. The method introduces three regularization terms: a normal-prior loss $L_n$, a semantic-guided normal smoothing loss $L_s$, and a normal-error-guided depth refinement loss $L_d$. On MuSHRoom, ScanNet++, and LERF-OVS, GLS reports better accuracy than single-task state-of-the-art methods on both tasks, with training time comparable to vanilla 3DGS.","feed_headline":"Joint training sharpens 3D surfaces and text-query masks","feed_subtitle":"Normal priors and CLIP features reinforce each other, beating single-task state of the art on three benchmarks.","key_machinery":"The load-bearing mechanism is the joint optimization objective over Gaussian primitives that carry both geometry and semantic features. Three regularization terms do the work: $L_n$ takes a monocular normal prior and pulls the rendered surface normal toward it, weighted by opacity; $L_s$ uses CLIP feature gradients to decide where to smooth normals, restricted to the largest surfaces via SAM masks; $L_d$ partitions each pixel into one of three angular cases relating the rendered normal to the camera's $-y$ axis, then replaces the rendered depth with a piecewise target depth built from the rendered depth, the unbiased depth, and the rendered alpha, and penalizes the difference with an exponential loss. The semantic branch supervises per-Gaussian features with CLIP features and DEVA-derived view-consistent masks, and meshes are extracted by TSDF fusion of the refined depth.","core_discovery":"The authors aim to establish that surface reconstruction and open-vocabulary segmentation are complementary enough that a single Gaussian representation can be optimized for both at once, and that doing so yields sharper and smoother results than either task alone. Concretely, they show that using a monocular normal prior to regularize the normal derived from rendered depth, using CLIP features to smooth normals only on large surfaces, and using the angular error between rendered and ideal normals to refine the unbiased depth, all improve both reconstruction metrics (accuracy, completion, chamfer-L1, normal consistency, F-score) and segmentation metrics (mIoU and boundary IoU). The same learned semantic features that answer text queries also act as a smoothness prior for geometry, and the cleaned geometry in turn gives segmentation cleaner boundaries.","pith_inferences":["The three-case angular partition in $L_d$ is presented without a proof of exhaustiveness; if real indoor normals frequently fall outside or on the boundaries of these sectors, the depth-refinement loss could push geometry away from true surfaces, so an oracle-normal experiment would isolate this risk.","The same semantic-as-geometric-prior idea could transfer to other explicit representations, such as surfel or mesh-based pipelines, whenever texture-less or specular regions defeat photometric cues.","Since $L_s$ only regularizes the top-k largest objects, small-object scenes may not receive the smoothing benefit; a per-object adaptive weighting scheme is a natural extension.","A direct extension would be to make the normal-prior confidence view-dependent, because the paper notes monocular normal estimates become inconsistent in outdoor scenes; confidence weighting might broaden applicability."],"forward_implications":["A single trained 3DGS model can output a mesh and answer open-vocabulary text queries, so interactive scene understanding and geometry editing no longer require separate 3D reconstructions.","The geometry branch inherits semantic smoothness in texture-less regions, which is where monocular normal priors alone typically fail.","The semantic branch inherits sharper object boundaries from the geometry branch, as shown by gains in boundary IoU.","Because the losses are lightweight regularizers on top of 3DGS, the training cost stays near that of a single-task 3DGS run.","The sensitivity ablations show that each of the five losses contributes to both tasks, supporting the claim that joint optimization is what drives the improvement."],"supporting_citations":[{"why":"Supplies the monocular surface normal prior $\\hat{N}$ used by $L_n$ and $L_d$.","marker":"[3]"},{"why":"Defines the unbiased depth $D_p$ and the gradient-based local normal $N_d$, and provides the codebase GLS builds on.","marker":"[10]"},{"why":"Introduces 2D Gaussian disks, depth distortion, and normal consistency, serving as a baseline and motivating the rendered-depth smoothness needed for reconstruction.","marker":"[24]"},{"why":"Provides the view-consistent object masks used as supervision for the rendered mask loss $L_m$.","marker":"[14]"},{"why":"Supplies the 3D Gaussian Splatting representation, photometric loss, and real-time rendering pipeline that GLS extends.","marker":"[29]"},{"why":"Provides the TSDF fusion procedure that converts refined depth into the final scene mesh.","marker":"[37]"},{"why":"Supplies the CLIP feature extraction scheme and serves as a segmentation baseline and source of object-level CLIP features.","marker":"[39]"},{"why":"Serves as a segmentation baseline and provides the rendering scheme used to evaluate selected objects from Gaussian semantic features.","marker":"[49]"},{"why":"Provides DEVA mask usage, a 3D consistency regularization idea, the mIoU and mBIoU evaluation metrics, and a segmentation baseline.","marker":"[52]"}],"fun_headline_variants":["One Gaussian model nails surfaces and text queries","Joint geometry-and-semantics training sharpens 3D splats","GLS: sharper 3D surfaces and clearer text masks","Surface and semantic priors team up in GLS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The depth-refinement loss $L_d$ assumes that the true surface normal always falls into one of three angular sectors defined around the rendered normal and the camera's $-y$ axis, so any real indoor normal outside those sectors would make the refined depth target wrong.","fun_headline_variants_meta":{"raw":{"variants":["One Gaussian model nails surfaces and text queries","Joint geometry-and-semantics training sharpens 3D splats","GLS: sharper 3D surfaces and clearer text masks","Surface and semantic priors team up in GLS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000294,"raw_usage":{"total_tokens":1680,"prompt_tokens":885,"completion_tokens":795,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":729}},"tokens_in":501,"tokens_out":795,"duration_ms":6818,"temperature":1.0,"reasoning_tokens":729,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:32:38.703895+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a synthetic indoor scene with ground-truth normals and depth, disable every loss except $L_d$ and measure whether optimized depth moves toward the ground truth for pixels whose true normal lies near or outside the three angular sectors; if those pixels systematically degrade, the three-case partition is not exhaustive.","supporting_citations":[{"cited_title":"2d gaussian splatting for geometrically accu- rate radiance fields","cited_arxiv_id":null,"evidence_quote":"Introduces 2D Gaussian disks, depth distortion, and normal consistency, serving as a baseline and motivating the rendered-depth smoothness needed for reconstruction."},{"cited_title":"Tracking anything with de- coupled video segmentation","cited_arxiv_id":null,"evidence_quote":"Provides the view-consistent object masks used as supervision for the rendered mask loss $L_m$."},{"cited_title":"3d gaussian splatting for real-time radiance field rendering","cited_arxiv_id":null,"evidence_quote":"Supplies the 3D Gaussian Splatting representation, photometric loss, and real-time rendering pipeline that GLS extends."},{"cited_title":"Kinectfusion: Real-time dense surface mapping and track- ing","cited_arxiv_id":null,"evidence_quote":"Provides the TSDF fusion procedure that converts refined depth into the final scene mesh."},{"cited_title":"Langsplat: 3d language gaussian splatting","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP feature extraction scheme and serves as a segmentation baseline and source of object-level CLIP features."},{"cited_title":"Gaussian grouping: Segment and edit anything in 3d scenes","cited_arxiv_id":null,"evidence_quote":"Provides DEVA mask usage, a 3D consistency regularization idea, the mIoU and mBIoU evaluation metrics, and a segmentation baseline."}],"review_version":1}