{"id":"4c27740a-2c26-46db-b98b-fadf47ff6c57","arxiv_id":"2608.10057","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LEGO builds view-consistent, multi-level 3D semantic hierarchies from multi-view SAM masks by clustering their physical 3D scales, and grounds them with CLIP for open-vocabulary segmentation and LLM-driven spatial grounding.","lead":"LEGO is a method that lifts 2D masks from the Segment Anything Model into a consistent 3D hierarchy of semantic levels, then attaches CLIP language features and builds a scene graph for spatial questioning. It reports state-of-the-art results on several 3D segmentation and open-vocabulary grounding benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Scale-based level assignment (Eq. 1) can split same-rank objects of different physical sizes into different levels, so the claimed strict invariance to absolute size is not established.","rationale":"LEGO's empirical numbers are strong and the component design is sensible, but the paper's distinctive contribution—the 'structural level' representation that is strictly invariant to viewing distance and absolute size—rests entirely on the scale-histogram peak assignment of Sec. 3.1. That step contains no mechanism that recognizes semantic rank; it only sees 3D scale. The bouquet example the authors use to motivate their method is exactly the case where the mechanism can fail: a large and a small flower at the same semantic level produce distinct scale modes and therefore distinct level labels under Eq. (1). The later hierarchical-consistency axioms and decoupled feature training inherit those labels, so the error is not corrected downstream; it propagates into dense supervision and scene-graph construction. The concern is not that the system fails empirically—the reported gains on NVOS, LERF-OVS, and Mip-NeRF 360 may well hold because CLIP matching and local context carry much of the burden—but that the central claim, as stated, is unsupported and likely false in the motivating scenario. This is a conditional-acceptance issue: the paper should either weaken the invariance claim, demonstrate it on the motivating example with a precise peak-detection protocol, or provide a mechanism that maps same-rank instances to a shared level. The reader's verdict of CONDITIONAL is appropriate; my stress-test does not move it, but it sharpens the required condition.","tokens_in":19881,"tokens_out":7611,"duration_ms":77707,"concrete_test":"Run the Sec. 3.1 pipeline on a controlled capture (or the motivating bouquet) containing two flowers at the same semantic rank whose measured 3D diameters differ by at least 3x, together with co-visible flowerpot/bouquet/petal masks. Lift masks, compute s_i, build the scale histogram of N(m_i) ∪ {m_i}, detect peaks, and apply Eq. (1). Report the level indices assigned to the two flowers. Because the paper does not specify the histogram bin width or peak-detection parameters, repeat with bin widths spanning at least an order of magnitude; if no reasonable setting assigns both flowers the same level without also merging distinct structural levels (e.g., bouquet with flower), the strict-invariance claim in Sec. 1 is falsified. If code is not released, this test requires the authors to disclose the peak-detection hyperparameters and the MASt3R-SfM point-lifting used to compute s_i.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. 3.1 defines each mask's level as the nearest peak of the histogram of 3D scales of co-visible masks (Eq. 1). The Sec. 1 claim is that levels are 'strictly invariant to both viewing distances and absolute physical sizes,' e.g., all flowers in a bouquet belong to the 'bud' level. But the histogram is built from the same physical scales that the paper rejects in scale-based methods. If two same-rank instances have well-separated 3D diameters (a large flower vs. a small flower in the motivating bouquet), their scales form separate modes, and Eq. (1) assigns them different level indices. Nothing in the peak-detection procedure merges same-rank instances across scale gaps; no theorem, constraint, or experiment is given. Because the later supervision (Sec. 3.1 Recursive Inclusion, Sec. 3.2 decoupled L-level features) inherits these labels, the error propagates into dense supervision and scene-graph construction. The failure mode the paper attributes to scale-based methods in Fig. 2(b) re-occurs whenever within-rank scale spread creates separate histogram peaks, which is precisely the bouquet-flower scenario used to motivate the method. A related gap is that levels are local peak ranks, so the same level index may denote different semantic ranks in different scene regions; the claimed unified global hierarchy is therefore under-specified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LEGO, a method for open-vocabulary and hierarchical understanding of 3D Gaussian splatting scenes. It lifts multi-view SAM masks to 3D, estimates their physical scale, and assigns each mask a discrete level by nearest histogram peak of co-visible mask scales (Eq. 1). The level labels are used to build pixel-pair indicators with monotonicity, recursive inclusion, and structural inheritance axioms; these supervise a decoupled per-level Gaussian identity feature space. Subsequent hierarchical segmentation uses recursive HDBSCAN over the level features, CLIP grounding uses an optimal view selection, and a level-wise scene graph supports LLM-driven Chain-of-Retrieval. The paper reports state-of-the-art results on NVOS, SPIn-NeRF, LERF-OVS, Mip-NeRF 360, and a new CoR benchmark.","tokens_in":20178,"tokens_out":4551,"duration_ms":42355,"significance":"If validated, the method offers a practical recipe for automatic, hierarchical, part-level 3D scene decomposition from 2D foundation-model masks, and the downstream LLM-grounded graph reasoning is a promising direction. The paper is well-executed on the engineering side: the supplementary formalizes the indicator construction and losses, ablations support the individual loss components, and the qualitative CoR results are compelling. However, the central conceptual claim, that the method discovers levels that are strictly invariant to viewing distance and absolute physical size, is not established by the current formulation, since levels are inferred from physical scale histograms. This weakens the hierarchy-discovery claim and the interpretation of the benchmark improvements.","major_comments":[{"comment":"The claimed strict invariance of the assigned levels to absolute physical size is not supported by the method. In Eq. (1), the level index is assigned by the nearest peak of a histogram of physical scales of co-visible masks, so two same-rank entities with well-separated physical diameters, such as a large flower and a small flower in the motivating bouquet example, will form separate histogram modes and receive different level indices. No constraint, theorem, or experiment in the paper enforces that same-rank instances across scale gaps are merged into one level, and the manuscript itself in Sec. 1 calls physical scales an \"initial proxy,\" which is in tension with the \"strictly invariant\" phrasing used in the same section. Because the level labels propagate into the supervision in Sec. 3.1 and the feature learning in Sec. 3.2, this is a load-bearing gap; the authors should either prove the invariance or directly evaluate it, for example by measuring level consistency across instances of known semantic rank with varied physical sizes.","section":"Sec. 3.1, Eq. (1); Sec. 1, \"Our Insight\""},{"comment":"The level index is defined locally, not globally. Because peaks are detected for each target mask from its co-visible neighbor set N(mi), the resulting li is a local rank within a region, and the same level index can denote different semantic ranks in different scene regions. This is in tension with the paper's abstract and with the evaluation protocol in Supp. Sec. B, which uses \"the first three levels\" as if levels were globally comparable. Without a definition of global level semantics or an empirical demonstration that local peak ranks coincide across regions, the claimed unified hierarchy is under-specified, and the reported open-vocabulary results may reflect local ranking artifacts rather than a scene-wide semantic hierarchy.","section":"Sec. 3.1, \"Local Peak-based Mask Level Assignment\""},{"comment":"The hierarchy depth L is fixed to 8 and the method is expected to identify L prominent peaks, but the paper does not specify behavior when the local scale histogram has fewer than eight modes or when peak detection is ambiguous. Since Eq. (1) depends critically on the peak set, the free choice of L and the peak-detection parameters should be analyzed with a sensitivity study. As written, the adaptivity claim is not fully supported, and the reader cannot determine whether the reported gains are robust to these choices.","section":"Sec. 3.1 and Supp. Sec. B"},{"comment":"The open-vocabulary evaluation protocol retrieves all clusters within the selected level whose similarity exceeds 0.9*Smax, but because the selected level is the local level of the best-matching cluster, there is no guarantee that the retrieved clusters are semantically co-ranked with the target. This protocol, combined with the local level assignment of Sec. 3.1, makes the reported gains on LERF-OVS and Mip-NeRF 360 difficult to interpret as evidence for a unified semantic hierarchy. The authors should report results with a stricter protocol that retrieves only the single most similar cluster, and should ablate the 0.9 threshold to show that the improvement is not a threshold artifact.","section":"Supp. Sec. B, \"Open-Vocabulary Understanding\""}],"minor_comments":[{"comment":"The caption contains the typo \"Stable hierachy\" and would benefit from copy-editing.","section":"Fig. 2"},{"comment":"The term \"co-visible 3D regions\" is not precisely defined; please clarify how N(mi) is constructed and what overlap threshold is used.","section":"Sec. 3.1"},{"comment":"The new CoR benchmark is described with only a one-sentence summary in the main text; the annotation procedure, query construction, and evaluation protocol should be described in the main text or the supplementary should be referenced more explicitly.","section":"Table 5"},{"comment":"The notation m* is used before its definition in the following sentence; reorder the derivation for clarity.","section":"Eq. (6)"},{"comment":"N2F2 [2] appears in the supplementary comparison table but is not discussed in the related work; a brief description would help readers interpret the comparison.","section":"Table 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is competently engineered and the experimental apparatus is extensive, but the central claim of scale-invariant structural level discovery is not supported by the method as written. If the authors reframe the contribution as local, scale-based multi-level clustering with a careful analysis of when levels are comparable, the work could be publishable after substantial revision. I also note that the new CoR benchmark, as described, lacks the detail needed for reproduction, and the open-vocabulary evaluation protocol may be inflating the gains."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: LEGO is a solid systems paper, and the mechanism at its core—re-grading multi-view SAM masks into a 3D-consistent level structure by clustering co-visible masks on physical scale—is genuinely new. The numbers on the standard benchmarks are as good as anything I've seen, and the decoupled per-level feature design is clean. But the abstract's \"strictly invariant to viewing distances and absolute physical sizes\" claim outruns what the method actually does, and the hierarchy itself—the supposedly new capability—is only tested on a self-created 120-query benchmark.\n\nWhat's actually new: prior hierarchical LGS methods (LaGa, Occam's, Hi-LSplat) take SAM's three 2D granularities at face value, inheriting their view-dependence; scale-based grouping (GARField-style) relies on a global scale parameter. LEGO instead aggregates multi-view masks, lifts them to 3D, computes each mask's physical diameter, and assigns levels by local histogram peaks of co-visible masks' scales. That consensus mechanism is a real step forward, and Fig. 2 is the clearest comparison of these paradigms I've seen. The dense supervision via the pixel-pair indicator (monotonicity, recursive inclusion, structural inheritance) is principled, and the supplementary formalization is coherent. The optimal-view selection for CLIP grounding is a sensible fix for feature blurring. Experiments are broad—promptable, open-vocabulary, 3D-OVS—with SOTA margins. Citation pattern is fair: GAGS (their own prior work) is treated as a baseline.\n\nSoft spots, in proportion:\n\n1. The load-bearing claim doesn't follow from Eq. 1. Levels come from nearest-peak assignment to a histogram of 3D scales. When two same-rank instances have well-separated sizes—the large-flower/small-flower bouquet that motivates the paper—they form separate modes and get different levels. Nothing in the peak-detection step merges them, so \"strictly invariant to absolute physical size\" is not established. The authors do call scale an \"initial proxy\" in Sec. 3.1, but that's a weaker claim than the abstract's. Fixable: report level membership for same-rank objects with large size spread.\n\n2. The CoR benchmark is self-created (120 queries, four scenes, no human evaluation), and the gap over LaGa+CoR (51.6 vs 14.3 mIoU) is large enough that I'd want the queries and protocol released before trusting it.\n\n3. The peak-detection algorithm is underspecified (prominence/distance parameters change the hierarchy), and there's no code release.\n\nNone of this sinks the empirical contribution. The method works well on established benchmarks, and the level-consensus mechanism is worth building on. This deserves serious peer review. A good referee will push them to soften the invariance claim, specify the peak detector, and release the CoR evaluation. I'd send it out and expect a major revision, not a rejection.","headline":"Genuinely new multi-view mask re-grading mechanism and SOTA numbers on standard benchmarks, but the 'strictly invariant to absolute physical size' claim is not supported by the method, and the hierarchy itself is only evaluated on a self-created benchmark.","tokens_in":20687,"tokens_out":6511,"would_cite":true,"duration_ms":56183,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LEGO re-grades volatile multi-view SAM masks into a unified 3D-consistent semantic hierarchy whose levels are strictly invariant to viewing distance and absolute physical size, and uses that hierarchy for open-vocabulary segmentation and…","keywords":["3D scene understanding","semantic hierarchy","Gaussian splatting","open-vocabulary segmentation","SAM masks","scene graph","large language models","spatial reasoning"],"falsifier":"Build or render a scene containing two instances of the same semantic object at strongly different physical sizes, such as a large flower and a small flower in one bouquet. If LEGO assigns them to different levels because their scales fall near different histogram peaks, then the claimed invariance to absolute physical size fails; the assignment is directly observable by inspecting the per-mask level labels computed by Eq. (1) and the resulting 3D segmentation.","tokens_in":1802,"feed_emoji":"🧩","tokens_out":3445,"duration_ms":90635,"temperature":0.7,"pith_summary":"LEGO claims that the messy, view-dependent granularities produced by 2D segmentation models can be stitched into a single, 3D-consistent semantic hierarchy, one in which the same physical entity always receives the same structural level no matter how close or far the camera is or how large or small the object is in absolute terms. The key step is to lift multi-view masks into 3D, estimate each mask's physical span, and then let the local histogram of those spans reveal the latent levels, assigning every mask to the nearest histogram peak. Once levels are fixed, the method trains a Gaussian splat field with level-wise decoupled identity features, yielding hierarchical segments that can be grounded with CLIP and organized into a scene graph for LLM-driven spatial queries. If this works, automatic 3D scene understanding would move from flat object recognition to scale-free part-whole reasoning, which matters for robots and assistants that must parse cluttered scenes at multiple granularities.","feed_headline":"Multi-view masks merge into one 3D scene hierarchy","feed_subtitle":"Re-graded 3D levels stay stable across view and size, unlocking hierarchical segmentation and LLM spatial grounding.","key_machinery":"The load-bearing mechanism is the local peak-based mask level assignment: for each lifted SAM mask, its 3D spatial diameter $s_i = 2\\sqrt{\\sum_{d\\in\\{x,y,z\\}} \\mathrm{std}(\\mathcal{P}_{m_i,d})^2}$ is compared with the histogram of scales of co-visible neighboring masks, and the detected peaks define the levels that place the mask via Eq. (1). This converts continuous physical scale into a discrete, locally normalized structural rank. The second mechanism is the level-wise decoupled feature field: each Gaussian carries $L$ independent $d$-dimensional features, one per level, trained by the contrastive indicator loss, which prevents cross-level interference and enables recursive HDBSCAN clustering into a nested segmentation tree.","core_discovery":"The paper argues that the three fixed 2D granularities produced by SAM are not a defect but fragmented glimpses of a deeper latent hierarchy: a close-up view of a bouquet may see \"bud → petal\", while a distant view sees \"flowerpot → bouquet → bud\", and together these observations reconstruct the full lineage. LEGO formalizes the reconstruction by computing a 3D physical scale for each lifted mask and running peak detection on the scale histogram of co-visible masks; the peaks are taken as the structural levels and Eq. (1) assigns each mask to the nearest peak. The paper claims this \"3D level-based\" assignment is strictly invariant to viewing distance and absolute physical size, and that supervising a Gaussian field with a dense level-wise indicator, built on monotonicity, recursive inclusion, and structural inheritance, yields a decoupled feature space in which each level can be segmented and grounded without cross-level semantic entanglement.","pith_inferences":["Inference: if the level-consistency claim holds, the same machinery could be used to build a canonical part-whole taxonomy across a dataset of scenes, because levels are assigned by local histogram peaks rather than a global scale threshold, so object instances could be compared level-by-level without a hand-built ontology.","Inference: a stress test follows directly from the method's own initialization: since level assignment starts from physical scale, the invariance to size is only as strong as the histogram peaks' ability to separate sizes within a local neighborhood; constructing a bouquet with one flower an order of magnitude larger than its neighbors would test whether both flowers still receive the same level.","Inference: the scene-graph beam search is a natural scaffold for distance- or orientation-based queries; extending the graph edges with metric spatial relations would let the same retrieval chain answer \"left of\" or \"farther than\" without retraining the field.","Inference: because the pipeline inherits SAM's mask quality and CLIP's text-image alignment, its open-vocabulary ceiling is tied to those backbones, so swapping in stronger foundation models should improve grounding directly, suggesting the hierarchy itself is backbone-agnostic."],"forward_implications":["SAM's fixed three granularities can be replaced by an arbitrary-depth hierarchy learned from the scene itself, because the level count comes from histogram peaks rather than from SAM's architecture.","Objects that share a structural rank, such as all buds in a bouquet or all petals on a flower, remain at the same level regardless of camera distance or physical size, which reduces the granularity blurring seen in view-dependent distillation.","Level-wise decoupled features give cleaner boundaries for fine-grained parts, such as corn and onion segments in the ramen scene, than flat or entangled feature fields, as reported in the open-vocabulary benchmarks.","A level-wise language scene graph with hierarchical and adjacency edges allows an LLM to answer compositional queries through relation-constrained beam search, a capability that flat CLIP matching lacks.","Promptable segmentation improves on saturated benchmarks, with up to +1.6 mIoU on NVOS and strong mIoU on SPIn-NeRF, suggesting the hierarchy also helps under manual point or scribble prompts."],"supporting_citations":[{"why":"Supplies the multi-view 2D masks and their three built-in granularities (whole, part, subpart) that LEGO re-grades into 3D levels.","marker":"[23]"},{"why":"Provides the 3D Gaussian splatting representation that LEGO equips with level-wise identity features.","marker":"[20]"},{"why":"Supplies the scene geometry and pixel-to-point mapping used to lift 2D masks into 3D and estimate their physical scales.","marker":"[10]"},{"why":"Supplies the identity-embedding and contrastive-distillation formulation that LEGO generalizes to a decoupled multilevel feature space; also a strong promptable-segmentation baseline.","marker":"[4]"},{"why":"Provides the HDBSCAN clustering used to recursively partition parent segments into child segments during hierarchical scene segmentation.","marker":"[30]"},{"why":"Defines the LERF-OVS open-vocabulary benchmark and the CLIP-distillation paradigm LEGO builds on.","marker":"[21]"},{"why":"Supplies the granularity-aware distillation baseline and the Mip-NeRF 360 annotations used for open-vocabulary evaluation.","marker":"[37]"},{"why":"Supplies the language-Gaussian-splatting baseline and extended LERF annotations used for open-vocabulary comparisons.","marker":"[38]"}],"fun_headline_variants":["From flowerpots to petals: one 3D hierarchy from many views","Multi-view masks reveal a single 3D semantic ladder","Peak-detect scales to stitch views into scene levels","Unified 3D scene levels from fragmented 2D views","Cross-view consensus builds level-wise 3D reasoning"],"cache_read_input_tokens":22784,"weakest_assumption_plain":"The method assumes that an object's 3D physical span, measured by the spread of its lifted mask points, is a reliable proxy for its semantic structural rank, so that histogram peaks over co-visible masks recover the true level hierarchy regardless of intra-class size variation.","fun_headline_variants_meta":{"raw":{"variants":["From flowerpots to petals: one 3D hierarchy from many views","Multi-view masks reveal a single 3D semantic ladder","Peak-detect scales to stitch views into scene levels","Unified 3D scene levels from fragmented 2D views","Cross-view consensus builds level-wise 3D reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00038,"raw_usage":{"total_tokens":2003,"prompt_tokens":915,"completion_tokens":1088,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":1002}},"tokens_in":531,"tokens_out":1088,"duration_ms":9792,"temperature":1.0,"reasoning_tokens":1002,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:14:24.236568+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build or render a scene containing two instances of the same semantic object at strongly different physical sizes, such as a large flower and a small flower in one bouquet. If LEGO assigns them to different levels because their scales fall near different histogram peaks, then the claimed invariance to absolute physical size fails; the assignment is directly observable by inspecting the per-mask level labels computed by Eq. (1) and the resulting 3D segmentation.","supporting_citations":[{"cited_title":"In: Proceedings of the AAAI Conference on Artificial Intelligence","cited_arxiv_id":null,"evidence_quote":"Supplies the identity-embedding and contrastive-distillation formulation that LEGO generalizes to a decoupled multilevel feature space; also a strong promptable-segmentation baseline."}],"review_version":1}