{"id":"eeaa1870-1ee0-42e6-8782-b01f2133df29","arxiv_id":"2411.13251","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"BelHouse3D is a synthetic indoor point cloud benchmark with an occlusion-based OOD test set, showing that fully supervised segmentation models lose 32 to 49 percent mIoU under occlusion.","lead":"This paper introduces a synthetic 3D point cloud dataset of 424 indoor scenes based on 32 real Belgian houses, with a test set that artificially adds occlusion to mimic real-world conditions. It benchmarks several popular point cloud segmentation models and finds that occlusion notably hurts fully supervised models, while few-shot models degrade less.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OOD validity rests on unvalidated alignment between approximate synthetic layouts and original scanner viewpoints; the paper provides no quantitative check that the resulting occlusion masks correspond to real occlusions.","rationale":"The reader's weakest assumption is exactly the geometric fidelity of the synthetic scenes to the real-world references, specifically the approximate object placement in Sec. 3.2 and its effect on the occlusion filter in Sec. 3.3. I agree that this is the most load-bearing concern because the dataset's raison d'etre is to provide a realistic occlusion-based OOD test set; if the occlusion masks are generated from scene geometry that is only approximately aligned, the benchmark's quantitative claims about model degradation and few-shot robustness cannot be trusted as measurements of real-world occlusion. I considered the few-shot support-set protocol as an alternative concern: Sec. 4.2 appears to draw support samples from the same test set as the query, so FSL methods receive test-time labels from the OOD distribution while fully supervised methods do not, which would confound the robustness comparison. However, this issue, while real, affects one specific result, whereas the alignment problem undermines the validity of the entire OOD test set and thus the dataset itself. The synthetic alignment is also explicitly disclosed as approximate, yet no validation is provided, making it the least secure link in the central claim. The proposed test, per-object registration error plus per-class visible-surface comparison, would settle whether the OOD test set reproduces real occlusion patterns within a defined tolerance. Given the reader's verdict is already CONDITIONAL and lists validation of the synthetic occlusion protocol as a required condition, my concern does not change the verdict.","tokens_in":15055,"tokens_out":8312,"duration_ms":94139,"concrete_test":"Select a subset of the 32 houses; register each Blender scene to the corresponding real point cloud using building structures only (walls, floor, ceiling) and report the residual per-object pose error after registration. Then render the synthetic scenes from the recorded RealSense camera poses and compare the resulting visible-surface fraction per class against the real scans. If mean per-object translation error exceeds 10% of the object's bounding-box diagonal, or the per-class visible-surface fraction differs by more than 10 percentage points from the real scans, the OOD occlusion masks are not faithful and the benchmark's conclusions about occlusion robustness are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"BelHouse3D's central value is an OOD test set that reproduces real-world occlusion. The construction in Sec. 3.3 filters synthetic points using the original RealSense viewpoints, but Sec. 3.2 states that object placement in the Blender scenes 'does not strictly adhere to exact object positioning', and no quantitative comparison between the synthetic scenes and the real reconstructions is reported. If object-position errors are comparable to object size, the visibility masks derived from the real viewpoints will hide and reveal the wrong surfaces, so the occlusion patterns are not those of the real scenes. This makes the benchmark's headline measurements of model degradation untrustworthy as a characterization of real occlusion robustness. The failure mode is concrete: the paper admits the approximate placement and provides no alignment error or occlusion-statistics validation; therefore the OOD test set's realism is unestablished. This is the load-bearing assumption because every downstream conclusion, including the fully supervised mIoU drops in Table 2 and the FSL robustness comparison in Table 3, depends on the test set faithfully representing occlusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces BelHouse3D, a synthetic point cloud dataset for indoor semantic segmentation, built from real-world references of 32 Belgian houses. It contains clean, fully labeled point clouds for training and in-distribution testing, plus an out-of-distribution (OOD) test set created by filtering synthetic points through the original RealSense viewpoints to simulate occlusion. The authors benchmark five fully supervised point-based segmentation methods (PointNet, PointNet++, DGCNN, Stratified Transformer, Point Transformer V2) and four few-shot segmentation methods (ProtoNet, AttProto, MPTI, AttMPTI) on both IID and OOD test sets, reporting mIoU and OA. Their main empirical finding is that fully supervised models degrade substantially under occlusion (mIoU drops of 32–49% in Table 2), while few-shot methods appear comparatively more robust.","tokens_in":15197,"tokens_out":5057,"duration_ms":55130,"significance":"If the dataset is publicly released and the OOD construction is quantitatively validated, BelHouse3D would fill a clear gap: a dedicated 3D indoor point cloud benchmark for occlusion robustness. The paper provides initial baselines and a concrete evaluation protocol, which could be reused by the community. The finding that fully supervised methods lose large amounts of mIoU under occlusion, while few-shot methods show smaller or even positive changes, is interesting and falsifiable. However, the current manuscript does not release the dataset or code, and the core assumption that the viewpoint-filtered synthetic occlusions faithfully reproduce real occlusions is not verified, thus limiting the immediate significance and reliability of the reported numbers.","major_comments":[{"comment":"The OOD test set is the load-bearing contribution: every downstream conclusion about model degradation and few-shot robustness depends on its realism. The construction filters synthetic points using the original RealSense viewpoints (Sec. 3.3), but Sec. 3.2 states that object placement 'does not strictly adhere to exact object positioning.' The paper provides no quantitative evidence that the synthetic scene geometry aligns with the real reconstructions well enough for these visibility masks to be meaningful. Without an alignment error metric or a comparison of occlusion statistics (e.g., per-class visibility rates, occlusion pattern distributions) between the synthetic OOD set and the real scans, the claim that the occlusions 'closely mimic real-world scenarios' is unsubstantiated. Please add such validation or explicitly reframe the OOD set as a synthetic, controlled occlusion model whose realism is not yet established.","section":"Sec. 3.2 and Sec. 3.3"},{"comment":"This is a benchmark dataset paper, but no URL, download link, or data availability statement is provided for BelHouse3D, and no code release is mentioned. Without access to the dataset, readers cannot reproduce the tables, verify the OOD construction, or use the benchmark as a community resource. The manuscript should include a clear statement on where and under what terms the dataset and evaluation code will be released.","section":"Throughout (data availability)"},{"comment":"The paper reports a single run per method for each setting, with no error bars, standard deviations, or statistical tests. The central comparative claim in Sec. 4.2 that 'few-shot learning methods exhibit better robustness' rests on differences that are often small (e.g., Table 3, 1-way 1-shot: AttMPTI gains +8.28% on Novel, but ProtoNet gains only -0.09% on Novel and several other changes are within 1-2%). Without repeated runs or a measure of variance, these differences may not be significant. Please report mean and standard deviation over at least three seeds, or otherwise demonstrate that the observed robustness gap is not due to noise.","section":"Tables 2 and 3"},{"comment":"The description of the few-shot support/query construction is ambiguous. The text says 'N × 5 samples are selected from the entire sample list for each class and designated as the support set, with the remaining samples forming the query set,' but it is unclear what a 'sample' is (a point cloud block, a subcloud, or a fixed-size point set) and whether support and query points are drawn from the same point clouds or different ones. This ambiguity affects the reproducibility of Table 3 and should be clarified with precise definitions.","section":"Sec. 4.2 (FSL setup)"}],"minor_comments":[{"comment":"ScanNet200 is listed as synthetic (S) in the R/S column, but it is a real-world dataset. This mislabeling should be corrected to accurately situate BelHouse3D relative to prior work.","section":"Table 1"},{"comment":"The text refers to 'SceneNet [30]' but reference [30] is SceneNN (Hua et al.). SceneNet is a different dataset (McCormac et al.); please correct the citation or the dataset name.","section":"Sec. 2 (Related Work)"},{"comment":"The caption mentions 'bubble sizes represent the number of points in each class,' but the figure appears to be a bar graph. Please align the caption with the actual visualization or clarify what the bubbles refer to.","section":"Fig. 2 caption"},{"comment":"The process of 'sampling' points from Blender surfaces is not described in enough detail: no density, number of points per scene, or sampling strategy is specified. These details are important for reproducibility and for understanding the IID/OOD point distributions.","section":"Sec. 3.2"},{"comment":"The paper states that occluded test data are generated 'using the original viewpoints' but does not explain how visibility is computed (e.g., ray casting, depth buffers, or projection). Adding this implementation detail would strengthen reproducibility.","section":"Sec. 3.3"},{"comment":"The limitations section mentions the future expansion of classes and attributes but does not acknowledge the approximate object placement or the lack of validation of the OOD realism; consider adding these as limitations.","section":"Sec. 5 (Limitations)"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is promising, but the OOD realism concern is critical. The authors should be asked to either provide strong quantitative validation of the synthetic-to-real alignment or re-scope the paper as a controlled synthetic occlusion benchmark without claims of real-world fidelity. The data release is also mandatory for this type of contribution. If these issues are addressed and the tables gain error bars, I would be willing to support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"BelHouse3D is a genuine attempt to fill a real gap: no dedicated indoor 3D point cloud benchmark exists for occlusion-based OOD, and the authors have built one with a sensible construction. The idea of using the original RealSense viewpoints to filter synthetic scenes is clever and more realistic than random point dropping. The benchmark effort is solid — five fully supervised and four few-shot methods, and the reported mIoU drops are large and consistent. If released, this could become a useful testbed.\n\nBut the load-bearing claim — that the OOD test set reproduces real-world occlusion — is not actually validated. The paper admits in Section 3.2 that object placement in the Blender scenes 'does not strictly adhere to exact object positioning.' The OOD set is generated by filtering those approximate scenes through the original camera viewpoints. If object positions are off by anything comparable to object size, the visibility masks hide and reveal the wrong surfaces, so the occlusion patterns are not the real ones. No alignment error or occlusion statistics are reported. That makes the headline numbers in Tables 2 and 3 hard to interpret as evidence about real occlusion robustness. The stress-test concern holds up on reading.\n\nOther issues are smaller but material. The dataset and code are not released, so nothing can be reproduced today. Tables lack error bars, and a few swings (e.g., PointNet OA +4%) could easily be noise. Table 1 mislabels ScanNet200 as synthetic; it is a labeling of real ScanNet scans. That factual error in a survey table makes me less confident in the carefulness of the rest.\n\nCredit where due: the construction is transparent, the authors state their limitations (19 classes, XYZ only), and the few-shot robustness observation is interesting, even if it may be an artifact of the specific protocol. The related work is adequate and properly distinguishes Robo3D as outdoor LiDAR corruptions.\n\nThis paper is for researchers working on point cloud segmentation robustness in indoor scenes. It deserves a serious referee, but the referee should require release of the dataset and code, quantitative validation of the synthetic occlusion against real scans, error bars, and a corrected Table 1. I would not cite it in its current form.\n\nRecommendation: engage, but treat as a conditional accept pending those fixes.","headline":"A useful dataset idea with a clever occlusion construction, but the load-bearing claim of real-world alignment is unvalidated and the dataset is not released.","tokens_in":15750,"tokens_out":2631,"would_cite":false,"duration_ms":26208,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BelHouse3D, a synthetic indoor point cloud dataset built from 32 Belgian houses, shows that occlusion drops fully supervised semantic segmentation mIoU by 30-49% while few-shot methods remain robust.","keywords":["benchmarking dataset","OOD","semantic segmentation","few-shot","point cloud","indoor scene","occlusion robustness","synthetic dataset"],"falsifier":"Compare the occluded regions in BelHouse3D's OOD test set with the actual missing-point patterns in the real reconstructed scans of the same houses when viewed from the same viewpoints; if the shapes and frequencies of partial object occlusion do not match, the simulated OOD shift is not faithful. A more direct test is to train a model on the clean BelHouse3D set and evaluate it on real occluded scans, checking whether the mIoU drop falls in the paper's reported 30-49 percent range.","tokens_in":14821,"feed_emoji":"🏠","tokens_out":8239,"duration_ms":75850,"temperature":0.7,"pith_summary":"BelHouse3D is a synthetic point cloud dataset for indoor semantic segmentation, built from real scans of 32 Belgian houses, containing 424 scenes with 19 annotated classes. The paper's central contribution is an out-of-distribution (OOD) test set that simulates real-world occlusion by filtering synthetic points through the original recording viewpoints. Benchmarking five fully supervised point-based models, the paper reports mean intersection over union (mIoU) drops of 30-49 percent relative to the clean in-distribution test set, while overall accuracy stays comparatively stable. In the few-shot setting, the same occlusion causes only small drops, and some models even improve on novel classes. The intended consequence is a reusable benchmark for measuring and improving occlusion robustness in indoor 3D semantic segmentation.","feed_headline":"Occlusion cuts indoor 3D segmentation by up to 49%","feed_subtitle":"BelHouse3D's occlusion test set drops fully supervised mIoU by 32 to 49 percent; few-shot methods barely flinch.","key_machinery":"The central mechanism is the pipeline that turns real scans into paired clean and occluded synthetic point clouds. RGB-D frames captured with a handheld sensor are reconstructed into real point clouds; those reconstructions guide the layout of synthetic rooms and the placement of predefined object models in a 3D modeling program. To create the OOD test set, the synthetic scene is filtered through the original recording viewpoints, keeping only the points that would have been visible from those camera positions. This single filtering step is what converts an approximately arranged synthetic scene into occlusion patterns that mimic real-world visibility, and it is also the step that the paper's conclusions depend on.","core_discovery":"The paper claims that occlusion, a common and unavoidable condition of real-world indoor point clouds, constitutes a significant out-of-distribution shift that current fully supervised segmentation models are not robust to. On BelHouse3D, PointNet++ drops from 71.97 to 36.56 mIoU (-49%), DGCNN from 72.57 to 40.34 (-44%), Stratified Transformer from 79.12 to 44.54 (-44%), and Point TransformerV2 from 82.32 to 55.95 (-32%), with PointNet dropping from 38.38 to 26.97 (-30%). Since overall accuracy declines far less than mIoU, the paper argues occlusion disproportionately hurts smaller object classes rather than large building structures. The few-shot experiments show prototype-based and attention-based few-shot models are markedly more stable, with MPTI and AttMPTI improving on novel classes under occlusion in several configurations. The authors position BelHouse3D as the first dedicated benchmark for occlusion-based OOD generalization in indoor 3D point cloud segmentation.","pith_inferences":["A natural extension the authors do not run is to apply the same viewpoint-filtering recipe to real indoor datasets, which would test whether synthetic occlusion statistics match real ones without relying on approximate object placement.","If the few-shot robustness finding generalizes, it suggests occlusion resistance may come from transferable geometric prototypes rather than from large labeled training sets.","Because BelHouse3D currently stores only XYZ coordinates, the benchmark isolates geometry-based robustness; adding color or surface normals could change the measured degradation magnitudes.","The dataset could also support sim-to-real transfer studies by pretraining on clean synthetic scenes and fine-tuning on real occluded scans, a regime the paper does not report."],"forward_implications":["Fully supervised point-based segmentation models should be evaluated under occlusion-style OOD shifts, because IID test scores overstate real-world reliability.","The reported 30-49 percent mIoU drops provide a quantitative baseline that any proposed occlusion-robust method should beat on this benchmark.","Few-shot learning with prototype or attention mechanisms appears to be a more stable route to occlusion robustness than full supervision.","The gap between mIoU and overall accuracy under occlusion indicates that robustness efforts should concentrate on small object classes, not building structures."],"supporting_citations":[{"why":"Supplies the S3DIS real-world indoor point cloud benchmark that motivates the need for a synthetic alternative with precise labels.","marker":"[2]"},{"why":"Provides ScanNet, the large real-world indoor RGB-D dataset whose annotation cost motivates synthetic generation.","marker":"[13]"},{"why":"Introduces Robo3D, the closest existing 3D OOD benchmark for outdoor scenes, which the paper positions itself against.","marker":"[37]"},{"why":"Stratified Transformer is one of the fully supervised point-based models benchmarked on BelHouse3D.","marker":"[40]"},{"why":"PointNet is the foundational point-based segmentation model and the first baseline in the fully supervised benchmark.","marker":"[58]"},{"why":"PointNet++ is the benchmarked model with the largest reported mIoU drop under occlusion.","marker":"[59]"},{"why":"DGCNN is the graph-based benchmarked method whose 44 percent drop supports the claim that occlusion hurts strong models.","marker":"[73]"},{"why":"Point TransformerV2 is the best fully supervised baseline on BelHouse3D, dropping 32 percent under occlusion.","marker":"[74]"},{"why":"ScanNet++ supplies the recent high-fidelity indoor dataset used to argue that geometric fidelity matters for segmentation.","marker":"[82]"},{"why":"Provides the few-shot 3D segmentation framework (ProtoNet, AttProto, MPTI, AttMPTI) used in the few-shot benchmark.","marker":"[90]"}],"fun_headline_variants":["BelHouse3D: Occlusion drops 3D segmentation mIoU by up to 49%","Occlusion-blind models lose half their indoor 3D segmentation accuracy","Few-shot models resist occlusion better in new 3D benchmark","BelHouse3D benchmark: Occlusion cuts mIoU by 32-49% in indoor 3D","New benchmark exposes occlusion blind spot in 3D segmentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's conclusions assume that filtering the synthetic scenes through the original recording viewpoints reproduces the statistical structure of real occlusions, even though object placement in the synthetic scenes is only approximate and does not strictly adhere to exact positions.","fun_headline_variants_meta":{"raw":{"variants":["BelHouse3D: Occlusion drops 3D segmentation mIoU by up to 49%","Occlusion-blind models lose half their indoor 3D segmentation accuracy","Few-shot models resist occlusion better in new 3D benchmark","BelHouse3D benchmark: Occlusion cuts mIoU by 32-49% in indoor 3D","New benchmark exposes occlusion blind spot in 3D segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000767,"raw_usage":{"total_tokens":3453,"prompt_tokens":1053,"completion_tokens":2400,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":2293}},"tokens_in":669,"tokens_out":2400,"duration_ms":17837,"temperature":1.0,"reasoning_tokens":2293,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:38:36.068085+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the occluded regions in BelHouse3D's OOD test set with the actual missing-point patterns in the real reconstructed scans of the same houses when viewed from the same viewpoints; if the shapes and frequencies of partial object occlusion do not match, the simulated OOD shift is not faithful. A more direct test is to train a model on the clean BelHouse3D set and evaluate it on real occluded scans, checking whether the mIoU drop falls in the paper's reported 30-49 percent range.","supporting_citations":[{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"Stratified Transformer is one of the fully supervised point-based models benchmarked on BelHouse3D."},{"cited_title":"Advances in neural information processing systems 30 (2017) 18 U","cited_arxiv_id":null,"evidence_quote":"PointNet++ is the benchmarked model with the largest reported mIoU drop under occlusion."},{"cited_title":"Advances in Neural Information Processing Systems 35, 33330–33342 (2022)","cited_arxiv_id":null,"evidence_quote":"Point TransformerV2 is the best fully supervised baseline on BelHouse3D, dropping 32 percent under occlusion."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"Provides the few-shot 3D segmentation framework (ProtoNet, AttProto, MPTI, AttMPTI) used in the few-shot benchmark."}],"review_version":1}