{"id":"7d34f3ce-84e6-407c-aec9-b6bf0275c86e","arxiv_id":"2412.01931","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Planar Gaussian Splatting groups 3D Gaussian primitives into plane instances via a hierarchical Gaussian mixture tree, achieving state-of-the-art 3D planar reconstruction from RGB images without 3D labels or depth supervision.","lead":"This paper introduces Planar Gaussian Splatting, a method that reconstructs 3D planes in a scene from ordinary RGB photos without any 3D plane labels or depth data. It combines 3D Gaussian splatting with a probabilistic tree that groups similar Gaussian points into flat surfaces, and reports top results on indoor benchmarks while running faster than prior optimization methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on plane pseudo-labels generated from the same rendered depth/normals being optimized; no ablation isolates this bootstrap, leaving the core mechanism unverified.","rationale":"The reader's weakest_assumption identifies the same load-bearing dependency: the plane labels used to supervise descriptor learning are generated from rendered depth and normals of the very Gaussian field being optimized. This is the most critical condition for the paper's central claim because the claimed state-of-the-art performance without depth supervision rests on the reliability of this self-generated geometric signal. The paper's own Limitations section concedes that dark regions suffer under-reconstruction, which is exactly where rendered depth can be unreliable, and the RAG merging step in §3.3 has no independent source of depth to correct this. A concrete ablation using ground-truth depth on ScanNet would directly quantify how much the final metrics depend on the bootstrap, and would either strengthen or undermine the claim that the method is a robust unsupervised alternative to supervised baselines. The reader's verdict of CONDITIONAL (accept-shaped with reproducibility and transparency issues) is appropriate; our concern does not escalate the verdict to reject, but it does underline that the missing ablation is a substantive gap, not a minor omission.","tokens_in":14158,"tokens_out":6851,"duration_ms":267637,"concrete_test":"On ScanNet (which provides ground-truth depth), rerun the PGS pipeline in two configurations: (A) original, with rendered depth used in Eq. (7) for RAG merging; (B) same pipeline but substitute ground-truth depth for d(u,v) in Eq. (7) when constructing the SAM merge labels, while keeping all other training losses unchanged. Compare the final 3D plane segmentation metrics (VOI/RI/SC) on the same scenes. If (B) substantially improves over (A), the bootstrap dependency on internally rendered depth is a major source of error and the reported gains over NMF are not robust; if (A)≈(B), the self-supervised loop is stable with respect to depth errors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the accuracy of the plane pseudo-labels used to supervise descriptor learning. In §3.3, SAM segments are merged into planar regions via a Region Adjacency Graph, using surface normals from Eq. (5) and planar distances from Eq. (7), where Eq. (7) requires the depth d(u,v) rendered from the Gaussian field being optimized. Thus the training signal for descriptors (Eq. 6) is derived from geometric quantities (depth and normals) that are themselves outputs of the same jointly optimized field. If the rendered depth is systematically biased — for example, in dark regions where the paper's Limitations section admits under-reconstruction due to sparse Gaussian assignment — the merged SAM labels inherit that bias, and the descriptors learn to reproduce it. The GMT grouping (§3.2) then operates on these descriptors, propagating the error into the final plane instances. No experiment in the paper isolates this dependency: there is no ablation replacing rendered depth with ground-truth depth (available on ScanNet), nor an analysis of how label quality evolves during training. Without such evidence, the reported state-of-the-art scores cannot be attributed to a robust unsupervised mechanism rather than to a noisy self-supervision loop that happens to work on the tested scenes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Planar Gaussian Splatting (PGS), a per-scene optimization method that reconstructs 3D plane instances from posed RGB images using 3D Gaussian Splatting. The method augments each Gaussian with a surface normal and a plane descriptor, lifts 2D SAM masks into 3D via rendered depth and normals, merges them through a Region Adjacency Graph, and groups the resulting segments with a hierarchical Gaussian Mixture Tree. Experiments on 10 ScanNet scenes and Replica report state-of-the-art plane segmentation scores against NMF, PlanarRecon, and RANSAC-based baselines, along with a runtime reduction relative to NMF. An ablation study on two ScanNet scenes attributes performance gains to the SAM-based descriptors, local planar alignment, normal supervision, holistic separability, and Laplacian smoothing.","tokens_in":14462,"tokens_out":7718,"duration_ms":78350,"significance":"If the claims hold, PGS is a meaningful step toward RGB-only, label-free 3D planar reconstruction, combining the geometric convenience of 3D Gaussian primitives with the semantic priors of a foundation model. The per-scene optimization design sidesteps cross-dataset generalization problems that hurt supervised methods, and the reported Replica results support this advantage. The paper is clearly written and includes an ablation study that shows each proposed component contributes, plus a runtime comparison that is favorable. However, the evaluation is thin (10 ScanNet scenes, no error bars), and the core self-supervision loop—where rendered geometry creates the pseudo-labels that train the descriptors—is not controlled experimentally. These gaps currently prevent the SOTA claim from being fully established.","major_comments":[{"comment":"The construction of the Gaussian Mixture Tree is underspecified: Algorithm 1 takes leaf nodes each with a normal n_i and descriptor z_i, but Supplementary A.2 states that leaf nodes are Gaussian distributions fitted to boundary points of merged SAM masks, which have no intrinsic descriptor or normal. The paper does not explain how each leaf node obtains its descriptor and normal from the underlying Gaussian primitives, nor how the plane instances represented by the child nodes of the root are mapped back to individual Gaussians to produce the final plane segmentation used in evaluation. This makes the central grouping mechanism non-reproducible and needs to be clarified.","section":"§3.2 and Supplementary §A.2"},{"comment":"The pseudo-labels that supervise descriptor learning are generated by RAG partitioning that depends on the depth d(u,v) and normals rendered from the very Gaussian field being optimized. The Limitations section acknowledges that dark regions are under-reconstructed due to sparse Gaussian assignment, so these rendered quantities are biased exactly where geometry is hardest. Because no experiment isolates this dependency (e.g., replacing rendered depth/normals with ground-truth values during label generation, or measuring label quality as training progresses), the reported SOTA performance cannot yet be attributed to a robust unsupervised mechanism rather than to a self-supervision loop that reproduces its own geometry biases.","section":"§3.3, Eq. (7)"},{"comment":"The evaluation uses only 10 ScanNet scenes and a single Replica dataset, with no error bars or per-scene breakdown. The margin over NMF on Replica is small (VOI 4.311 to 4.168, SC 0.188 to 0.209), so the claim of 'significantly better' performance is not statistically supported. Reporting variance and per-scene results, or including more scenes, is needed to substantiate the SOTA claim.","section":"§4.1, Tables 1 and 2"},{"comment":"The descriptors are constrained to unit norm (∥z∥=1), but the paper does not state how this constraint is enforced during gradient updates from L_seg or L_n. The closed-form least-squares solution in Eq. (6) is unconstrained and includes a bias term, and the mean-shift update in Eq. (8) relies on the von Mises-Fisher kernel, which assumes unit-norm descriptors. Without a renormalization step (e.g., projection onto the sphere after each optimizer step), the dot-product comparisons and the kernel become scale-dependent; the renormalization procedure must be specified.","section":"§3.3 and §3.5"}],"minor_comments":[{"comment":"The column labeled 'Inference time' is misleading for optimization-based methods: the reported 16 minutes for PGS and 40 minutes for NMF are per-scene optimization/training times, not inference times in the usual sense. Renaming the column to something like 'Per-scene optimization time' would make the comparison clearer.","section":"Table 1"},{"comment":"The method is described as 'unsupervised,' but it relies on pretrained networks (SAM for masks and Omnidata for normals) to provide pseudo-labels. Consider using 'self-supervised' or 'without scene-level plane/depth labels' to avoid overstating the absence of learned priors.","section":"Contributions, line 1"},{"comment":"The thresholds for RAG edge cutting (10 cm planar distance, 20 degrees normal angle) are given only in the supplementary, and the GMT thresholds ϵ_B and ϵ_z are not reported anywhere. Including these thresholds in the main text, together with a sensitivity analysis, would improve reproducibility.","section":"Section 3.3 and Supplementary A"},{"comment":"The matrix dimensions in Y = [Z|1]·W are not explicitly defined; clarifying that Z is the pixel-descriptor matrix, Y is the one-hot segment label matrix, and W includes the bias column would help readers follow the closed-form solution.","section":"Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The paper builds closely on the authors' earlier NMF work and shares authors with it, which is fine; the novelty is in adapting the idea to 3D Gaussian Splatting with a hierarchical merging tree. The main risk is the unvalidated self-supervision loop: if the rendered geometry used to create the training labels is biased in exactly the failure modes the paper's own limitations describe, the reported improvements could be partly self-referential. A control experiment with ground-truth depth/normals for label generation and a more robust evaluation would address this. The manuscript fits the scope of a computer vision venue, though it needs the experiments and clarifications outlined in the major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a genuine step forward for RGB-only 3D planar reconstruction. It replaces the depth-network and post-hoc clustering of the authors' earlier NMF with a hierarchical Gaussian mixture tree and descriptor lifting from SAM masks, and it reports better numbers than supervised methods on both ScanNet and Replica. The central approach is coherent and the engineering looks credible.\n\nWhat's actually new: the GMT grouping over Gaussian primitives, and the closed-form linear regression that lifts variable-length SAM masks into per-Gaussian plane descriptors. The ablation on two scenes shows each component contributes, with SAM masks and local planar alignment carrying the most weight. That is real evidence, not just a narrative.\n\nThe soft spots are reproducibility and the bootstrap. No code is released; several thresholds and loss weights are missing (descriptor dimensionality k, weights for the normal and segmentation terms). More substantively, the pseudo-labels that supervise descriptors are built from rendered depth and normals of the same Gaussian field being optimized. The paper's own limitations admit dark regions under-reconstruct. If the rendered geometry is biased, the SAM merging inherits that bias and the descriptors learn to reproduce it. The stress-test is right that no experiment isolates this: there is no ablation replacing rendered depth with ground-truth depth on ScanNet, even though it is available. That leaves the headline SOTA claim somewhat underdetermined.\n\nI don't think this is fatal. The final evaluation is external ground-truth, and the method generalizes to Replica without retraining, which a degenerate self-supervision loop would be unlikely to do. The real question is attribution: how much of the gain comes from the grouping machinery versus the bootstrap, and how sensitive it is to geometry quality. A serious revision should add an oracle-depth ablation, report error bars on the 10 ScanNet scenes, and release code or a robust supplement.\n\nWho this is for: anyone working in 3D scene understanding, Gaussian splatting, or lifting 2D foundation-model masks into 3D. It deserves a serious referee. I'd conditionally accept after those additions.","headline":"Sound, well-engineered unsupervised planar reconstruction; the SOTA claim is plausible but the self-supervision loop needs an isolating ablation and the paper lacks reproducibility details.","tokens_in":14968,"tokens_out":2355,"would_cite":true,"duration_ms":22814,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A scene's 3D planes can be parsed from posed RGB video by grouping Gaussian splats, with no plane labels or depth supervision.","keywords":["planar reconstruction","3D Gaussian splatting","Gaussian mixture tree","plane instance segmentation","unsupervised 3D understanding","Segment Anything Model","surface normal estimation","neural rendering"],"falsifier":"Run PGS on a scene with a large dark wall beside a lighter wall at the same orientation and compare the recovered plane instances with the ground-truth plane boundaries; the paper's stated limitation predicts the dark wall will be under-reconstructed and split, which would directly test whether the descriptor grouping and region adjacency graph merging hold where rendered depth is weakest.","tokens_in":13961,"feed_emoji":"📐","tokens_out":6097,"duration_ms":53058,"temperature":0.7,"pith_summary":"The paper sets out to show that 3D plane instances in an indoor scene can be reconstructed from posed RGB images together with off-the-shelf 2D segmentation and normal maps, without any 3D plane labels or depth supervision. Instead of training a network to predict planes, Planar Gaussian Splatting (PGS) optimizes a Gaussian splatting field per scene and groups the Gaussian primitives into planes using a probabilistic tree. It does this by adding a surface normal and a learned plane descriptor to each Gaussian, lifting 2D masks from a general segmentation model into 3D through a closed-form linear regression, and merging adjacent segments whose normals and planar distances agree. The payoff, if the reported results hold, is a planar reconstruction pipeline that transfers across datasets without retraining and that the paper reports as more accurate on ScanNet and Replica than supervised baselines, while running in about 16 minutes. It also requires no depth input at all, unlike the closest optimization-based predecessor.","feed_headline":"No plane labels or depth: Gaussian splats parse 3D planes from video","feed_subtitle":"No plane labels, no depth: per-scene optimization beats supervised baselines in 16 minutes.","key_machinery":"The Gaussian Mixture Tree (GMT), a hierarchical tree over Gaussian primitives in which leaf nodes are local clusters of Gaussians, parent nodes are formed by merging nearby Gaussian distributions, and the children of the root are the recovered 3D plane instances. Two extra per-Gaussian parameters carry the argument: a surface normal, supervised by rendered normal maps against an off-the-shelf predictor, and a unit-norm plane descriptor, learned from lifted SAM masks via a per-view closed-form linear regression. Merging is gated by both the Bhattacharyya distance between Gaussian distributions and the cosine similarity of descriptors, while a recurrent mean-shift update applied to the whole field keeps descriptors for different planes separable.","core_discovery":"The central claim is that plane parsing can be folded into the same optimization that reconstructs geometry, so no supervised plane predictor is needed. PGS treats each 3D Gaussian primitive as carrying a unit-norm plane descriptor and a surface normal; the descriptors are learned by rendering them into each camera view and solving a closed-form linear regression that predicts the merged 2D segment labels produced from SAM masks. A region adjacency graph then merges SAM segments that lie on the same plane using normal and planar-distance thresholds, and a tree-structured Gaussian mixture merges Gaussian clusters bottom-up, with the root's children being the final plane instances. The paper reports that this unsupervised, per-scene optimization reaches higher plane segmentation scores on ScanNet and Replica than supervised methods such as PlanarRecon, which were trained on ScanNet and degrade under domain shift, and that it outperforms the optimization-based NMF while taking 16 minutes instead of 40.","pith_inferences":["The closed-form lifting of 2D masks to 3D descriptors suggests PGS could be converted into an online system that updates plane labels incrementally as new frames arrive, rather than re-solving the linear regression over the full image set.","The same tree-merging machinery could be applied to non-planar structure by replacing the planar-distance criterion with a curvature-aware geometric agreement measure, which the paper does not explore.","Because the descriptors are unit-norm vectors in $\\mathbb{R}^3$, the method effectively learns a mapping from Gaussian position to a low-dimensional plane embedding; one could test whether that embedding geometry corresponds directly to plane parameters such as offset and orientation."],"forward_implications":["Plane instances come out directly from the optimized Gaussian field, so the usual post-processing chain of meshing, Marching Cubes, and Sequential RANSAC is bypassed.","Because the optimization is per scene, the method does not suffer the domain-shift failure mode of supervised plane predictors; the paper demonstrates this by transferring from ScanNet to Replica.","The reported 16-minute runtime, against 40 minutes for NMF, makes scan-to-plane reconstruction practical on a single consumer GPU.","Any improvement in off-the-shelf 2D segmentation or normal prediction should translate directly into better 3D plane labels within the same pipeline.","The method's plane count is not fixed in advance, so it can parse scenes with an arbitrary number of planar surfaces."],"supporting_citations":[{"why":"Supplies the 3D Gaussian splatting representation and differentiable rendering that PGS extends with normals and descriptors.","marker":"[16]"},{"why":"Provides the SAM 2D masks that are lifted to 3D to supervise the plane descriptors.","marker":"[17]"},{"why":"Provides the off-the-shelf normal predictions used to supervise the rendered normal field.","marker":"[8]"},{"why":"The closest optimization-based baseline; PGS is compared against it for plane quality and runtime.","marker":"[46]"},{"why":"The supervised baseline whose ScanNet-trained model is shown to degrade on Replica, supporting the generalization claim.","marker":"[40]"},{"why":"The supervised reconstruction baseline whose mesh plus Sequential RANSAC serves as a post-processing comparison.","marker":"[33]"},{"why":"Supplies the recurrent mean-shift update used to keep plane descriptors separable across the whole scene.","marker":"[18]"},{"why":"Provides the Bhattacharyya-distance merging criterion used to build the Gaussian Mixture Tree.","marker":"[14]"},{"why":"Supplies the ScanNet benchmark scenes on which the main plane segmentation results are reported.","marker":"[6]"},{"why":"Supplies the Replica dataset used to demonstrate cross-dataset generalization.","marker":"[32]"}],"fun_headline_variants":["Gaussian splats parse 3D planes from RGB, no labels or depth","No labels, no depth: Gaussian splats build 3D planes in 16 min","RGB-only Gaussian splats detect 3D planes without supervised training","Per-scene optimization beats supervised methods for 3D planes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the depth and normal fields rendered from the Gaussian splats are accurate enough to decide which 2D segments belong to the same plane, so any region where few Gaussians are placed, such as dark areas, can degrade the plane labels that the whole grouping depends on.","fun_headline_variants_meta":{"raw":{"variants":["Gaussian splats parse 3D planes from RGB, no labels or depth","No labels, no depth: Gaussian splats build 3D planes in 16 min","RGB-only Gaussian splats detect 3D planes without supervised training","Per-scene optimization beats supervised methods for 3D planes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001412,"raw_usage":{"total_tokens":5685,"prompt_tokens":909,"completion_tokens":4776,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":4694}},"tokens_in":525,"tokens_out":4776,"duration_ms":29965,"temperature":1.0,"reasoning_tokens":4694,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:00:33.467764+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PGS on a scene with a large dark wall beside a lighter wall at the same orientation and compare the recovered plane instances with the ground-truth plane boundaries; the paper's stated limitation predicts the dark wall will be under-reconstructed and split, which would directly test whether the descriptor grouping and region adjacency graph merging hold where rendered depth is weakest.","supporting_citations":[{"cited_title":"3d gaussian splatting for real-time radiance field rendering","cited_arxiv_id":null,"evidence_quote":"Supplies the 3D Gaussian splatting representation and differentiable rendering that PGS extends with normals and descriptors."},{"cited_title":"Omnidata: A scalable pipeline for making multi- task mid-level vision datasets from 3D scans","cited_arxiv_id":null,"evidence_quote":"Provides the off-the-shelf normal predictions used to supervise the rendered normal field."},{"cited_title":"Neural Mesh Fusion: Unsupervised 3D Planar Surface Understanding","cited_arxiv_id":"2402.16739","evidence_quote":"The closest optimization-based baseline; PGS is compared against it for plane quality and runtime."},{"cited_title":"PlanarRecon: Real-time 3D plane detection and reconstruction from posed monocular videos","cited_arxiv_id":null,"evidence_quote":"The supervised baseline whose ScanNet-trained model is shown to degrade on Replica, supporting the generalization claim."},{"cited_title":"NeuralRecon: Real-time coherent 3D recon- struction from monocular video","cited_arxiv_id":null,"evidence_quote":"The supervised reconstruction baseline whose mesh plus Sequential RANSAC serves as a post-processing comparison."},{"cited_title":"Recurrent pixel embed- ding for instance grouping","cited_arxiv_id":null,"evidence_quote":"Supplies the recurrent mean-shift update used to keep plane descriptors separable across the whole scene."},{"cited_title":"Methods for merging gaussian mixture components","cited_arxiv_id":null,"evidence_quote":"Provides the Bhattacharyya-distance merging criterion used to build the Gaussian Mixture Tree."},{"cited_title":"Scannet: Richly-annotated 3D reconstructions of indoor scenes","cited_arxiv_id":null,"evidence_quote":"Supplies the ScanNet benchmark scenes on which the main plane segmentation results are reported."}],"review_version":1}