{"id":"f94f566f-ac3b-4121-abe4-edd7368a982b","arxiv_id":"2411.13927","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new benchmark and model enable multi-object 3D reasoning segmentation, where a point-cloud scene and a question produce both explanations and masks for several objects at once.","lead":"This paper introduces ReasonSeg3D, a new benchmark that tests whether AI systems can reason about and segment multiple objects in 3D indoor scenes, and MORE3D, a model that produces both text explanations and 3D object masks. The work aims to help embodied AI agents and robots interpret natural-language questions about cluttered spaces.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ReasonSeg3D's ground truth depends on unverified GPT-4o question-answer generation; a systematic bias in object selection or spatial-relation wording would corrupt both training and evaluation, making the reported superiority an artifact.","rationale":"The reader identified the same load-bearing concern: the validity of the ReasonSeg3D benchmark as ground truth, given that the question-answer pairs are generated by GPT-4o with a modest human correction step. I agree that this is the most critical assumption underlying the paper's central claim. If GPT-4o systematically misdescribes spatial relations or selects unrepresentative objects, then both the training signal for MORE3D and the evaluation of all compared methods are biased, and the reported numerical superiority would not generalize. The paper's claim that only 3% of pairs require correction is presented without supporting evidence such as annotator instructions, inter-annotator agreement, or an error taxonomy, so the quality bar is not established. I also note secondary concerns, such as the omission of the directly relevant SegPoint baseline from Table 2 and the absence of any evaluation of the generated explanations, but these are less fundamental than the benchmark validity issue. The appropriate verdict remains CONDITIONAL, as the reader stated: the empirical claims are plausible but not fully verified until the dataset, code, and a rigorous quality audit are released. I do not see a demonstrated fatal error, so no change to the reader's verdict is warranted.","tokens_in":13114,"tokens_out":5580,"duration_ms":56926,"concrete_test":"Conduct an independent human audit on a random sample of 300 question-answer pairs from the ReasonSeg3D validation set, with two annotators working independently. Each annotator checks whether (a) every object named in the answer is correctly and completely covered by the attached ScanNetv2 ground-truth masks, (b) every stated spatial relation is geometrically true when checked against the point cloud, and (c) the question uniquely determines the intended object set. Compute per-item agreement between annotators and the fraction of items flagged as incorrect by both. If the independent correction rate is substantially higher than the reported 3%, or if inter-annotator agreement is low, the benchmark's ground truth is biased and the reported training and evaluation numbers are called into question. The audit protocol and error taxonomy should be published with the dataset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MORE3D achieves superior performance on ReasonSeg3D, and that ReasonSeg3D is a valid benchmark, rests on the assumption that the GPT-4o-generated questions and answers are correct and unbiased. In the Dataset Generation Pipeline, GPT-4o autonomously selects objects and writes answers containing 3D spatial relations, with human verification correcting only about 3% of question-answer pairs. The paper provides no details on the verification protocol: number of annotators, instructions, error taxonomy, or inter-annotator agreement. No independent error analysis is reported. Because the same benchmark is used both to train MORE3D (through the textual answer loss and the ground-truth-derived <SEG> index list) and to evaluate every compared method, any systematic bias in GPT-4o's object selection or spatial-relation phrasing would be learned by the model and then rewarded by the evaluation. For example, if GPT-4o tends to select only the most salient objects or to describe relations with a limited vocabulary such as 'next to,' the model could score high cIoU/gIoU on this benchmark while failing on genuinely diverse or relationally complex scenes. The conclusion that MORE3D has 'strong reasoning abilities' is additionally unsupported because no metric evaluates the quality or spatial correctness of the generated textual explanations. Thus the benchmark's validity is the most load-bearing assumption; if it fails, the claimed 30.19 cIoU and 32.01 gIoU superiority is not evidence of general 3D reasoning segmentation capability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new task, multi-object 3D reasoning segmentation, in which a model takes a 3D point cloud and a user question as input and produces both a textual answer with 3D spatial relations and segmentation masks for multiple objects. The authors introduce ReasonSeg3D, a benchmark built from ScanNetv2 with GPT-4o-generated question-answer pairs (with about 3% human-corrected), and MORE3D, a method that combines a 3D encoder, a LLaMA-7B multimodal LLM with LoRA, object-specific <SEG> token embeddings, and a 3D decoder. On the ReasonSeg3D validation set, MORE3D reports 30.19 cIoU and 32.01 gIoU, outperforming the listed 3D segmentation, general scene understanding, and reasoning segmentation baselines. The paper also includes ablations on prediction-branch design and loss functions.","tokens_in":13340,"tokens_out":3724,"duration_ms":36656,"significance":"If the benchmark is shown to be trustworthy, this is a worthwhile contribution: it is the first multi-object 3D reasoning segmentation dataset with spatial-relation explanations, and MORE3D provides a clean, reproducible framework that integrates textual explanation with mask prediction. The design choices are clearly motivated, and the ablations show that both the textual answer loss and the mask loss contribute meaningfully. The main significance rests on the validity of ReasonSeg3D as an evaluation platform, because the same GPT-4o-derived data is used for both training and evaluation. The paper promises release of data and code, which would enable follow-up work; however, the current manuscript does not yet provide enough evidence about benchmark quality to fully support the claim of a high-quality, unbiased evaluation platform.","major_comments":[{"comment":"The validity of ReasonSeg3D is load-bearing for the paper's central claims, but the manuscript provides no details on the human verification protocol, no inter-annotator agreement, no independent error analysis, and no study of systematic GPT-4o bias in object selection or spatial-relation phrasing. The text states that only about 3% of question-answer pairs required manual correction, but it does not report how many annotators were used, what instructions they received, what error taxonomy was applied, or how disagreements were resolved. Because the same benchmark is used both to train MORE3D (via the answer loss and the ground-truth-derived index list) and to evaluate every compared method, any systematic bias in GPT-4o's object choices or spatial vocabulary could inflate the reported gains. The authors should add an independent human evaluation on a held-out subset, report agreement statistics, and analyze the diversity of object categories and spatial-relation terms to demonstrate that the benchmark is not biased toward a narrow set of salient objects or phrases.","section":"Dataset Generation Pipeline"},{"comment":"SegPoint (He et al. 2024) is cited in Related Work as a recent 3D reasoning segmentation method and is directly relevant to the comparison, but it is absent from Table 2. The claim that MORE3D 'achieves superior performance across all evaluation metrics' is incomplete without this baseline. The authors should either add SegPoint to Table 2 (training and evaluating it under the same protocol) or provide a concrete justification for its exclusion. If SegPoint cannot be run because of code or data constraints, that limitation should be stated explicitly.","section":"Table 2"},{"comment":"The experimental section reports single-run cIoU and gIoU values with no error bars, repeated runs, or significance tests. The reported margin over Reason3D is about 5 cIoU points, and without variance estimates it is unclear whether this margin is reliable. Additionally, the definitions of cIoU and gIoU are not specified for the multi-object setting: it is not stated whether IoU is computed per object (averaged over objects then samples) or per sample (merging masks), nor how the cumulative union/intersection in cIoU is accumulated over multiple masks. The authors should report mean and standard deviation over at least three training runs and provide the exact aggregation formulas for both metrics.","section":"Evaluation Metrics and Experiments"}],"minor_comments":[{"comment":"The table header contains a typo: 'V enue' should be 'Venue'.","section":"Table 1"},{"comment":"There are inconsistent spacing issues in method names, such as 'LLaV A' and 'V isionLLM'; these should be normalized to 'LLaVA' and 'VisionLLM'.","section":"Related Work and throughout"},{"comment":"The sentence 'Samples that maximize diversity in both target objects and spatial-relation descriptions are then selected, thereby avoiding bias' is a claim without supporting evidence; the paper should describe the selection procedure concretely and show distribution statistics.","section":"Dataset Generation Pipeline"},{"comment":"Only two qualitative examples are shown in Figure 4, and there is no discussion of failure cases or limitations of the segmentation output; adding a few failure examples would strengthen the evaluation.","section":"Qualitative Results"},{"comment":"The paper states that all three loss terms are assigned equal weights, but the classification loss Lcls is not included in the ablation study (Table 4); an ablation including Lcls would clarify its contribution.","section":"Training Objectives and Ablations"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and useful task, and the method appears sound in its internal design. The main risk is the benchmark's validity: the reliance on GPT-4o with minimal reported human verification means the evaluation could be systematically biased, and the missing SegPoint comparison and lack of error bars further weaken the empirical claims. These issues are fixable within the scope of a revision, but they require substantial additional analysis rather than cosmetic changes. I would encourage the editor to require the benchmark validation and experimental rigor described in the major comments before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate benchmark-plus-method paper. The benchmark is genuinely new: ReasonSeg3D is the first 3D reasoning segmentation dataset with multiple objects, multiple categories, and spatial-relation explanations. The model is a serviceable 3D adaptation of the SEG-token idea, and the ablations show both answer and mask losses matter. The reported numbers support the claim on their own benchmark.\n\nThe soft spots are real but not fatal. The ground truth for question-answer pairs is GPT-4o generated with only ~3% human correction and no verification protocol described. That means systematic bias in object selection or spatial-relation wording could infect both training and evaluation. The paper should report inter-annotator agreement or at least an error analysis on a sample. It also doesn't evaluate the quality of the generated text at all, so \"strong reasoning abilities\" is not directly measured. Missing error bars and repeated runs means the 30.19 vs 25.12 cIoU gap is a point estimate; that's a minor issue but worth noting. SegPoint is cited but missing from Table 2, which is a gap for a directly relevant baseline. Data and code are promised but not released, so the numbers can't be independently checked.\n\nThe stress-test's concern about GPT-4o bias is plausible but not demonstrated. The paper discloses the generation process, and the benchmark does use external ScanNet masks as targets, so there's no circularity. The main risk is that the benchmark may be too easy or biased toward salient objects, which would inflate the method's performance relative to true reasoning ability. That is a standard concern for LLM-generated benchmarks and can be addressed with better analysis rather than a rejection.\n\nWho is this for: people working on 3D reasoning segmentation, embodied AI, and instruction-following in 3D. A serious referee should look at it, especially the dataset construction details. I'd recommend engagement with requests for the verification protocol, error bars, and the SegPoint baseline, but not a desk rejection.","headline":"A genuinely new multi-object 3D reasoning segmentation benchmark with a solid but incremental model; the reported gains are plausible, but the LLM-generated ground truth needs more scrutiny before the numbers can be fully trusted.","tokens_in":13976,"tokens_out":2038,"would_cite":true,"duration_ms":20241,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces a multi-object 3D reasoning segmentation task, builds the ReasonSeg3D benchmark with 20,113 question-answer pairs that carry 3D spatial relations, and proposes MORE3D, a network that extracts object-specific…","keywords":["3D reasoning segmentation","multi-object segmentation","point cloud understanding","spatial relations","multimodal large language model","ReasonSeg3D","MORE3D","ScanNet"],"falsifier":"Randomly sample ReasonSeg3D validation questions, have independent human annotators rewrite answers and select objects directly from the point cloud, then compare with the benchmark labels; if a meaningful fraction disagree on the objects or spatial relations, the reported 30.19 and 32.01 scores are measured against faulty labels. Alternatively, train MORE3D on a human-created subset of question-answer pairs and test on the same validation set; a large drop would show that the GPT-4o generation, not the model's reasoning, is doing the work.","tokens_in":1709,"feed_emoji":"🛋️","tokens_out":3038,"duration_ms":77821,"temperature":0.7,"pith_summary":"Most 3D scene understanding models either segment objects without reasoning or answer text questions without segmentation, and the few 3D reasoning-segmentation models handle only single objects or single categories. This paper tries to establish that a machine can take a free-form question about an indoor 3D scene, reason out which several objects the question implies, and output both a text explanation with spatial relations and segmentation masks for each object. To make that claim testable, it builds ReasonSeg3D, a benchmark of 1,513 ScanNet scenes with 20,113 question-answer pairs whose answers name multiple objects and their 3D spatial relations, and MORE3D, which attaches a separate segmentation token to each object mentioned in the answer. On its own validation split, MORE3D reaches 30.19 cumulative IoU and 32.01 generalized IoU, above the previous 3D reasoning-segmentation method at 25.12 and 25.90. If the benchmark is valid, this gives the field a way to measure and improve multi-object 3D reasoning rather than single-object referring.","feed_headline":"MORE3D segments multiple 3D objects from a single reasoning question","feed_subtitle":"New benchmark ReasonSeg3D pairs 20,113 questions with 3D masks; MORE3D answers and segments multiple objects in one pass.","key_machinery":"The central mechanism is object-specific embedding extraction from <SEG> tokens. The LLM outputs a textual answer in which each object name is followed by a <SEG> token; a multi-seg index list records the positions of these tokens, and the LLM hidden states at those positions become per-object embeddings. A 3D decoder computes dot products between each object embedding and the per-point features to produce masks, with separate branches for mask prediction and classification. The textual answer loss supervises the explanations, and the mask loss combines binary cross-entropy and Dice loss.","core_discovery":"The central claim is that multi-object 3D reasoning segmentation is a learnable task and that the proposed MORE3D solves it better than existing methods. The paper argues that injecting 3D spatial relations into the textual answer gives the model the signal it needs to identify multiple objects of different categories in a cluttered point cloud, and that extracting one LLM embedding per <SEG> token, then decoding each embedding against shared per-point features, is the mechanism that turns one question into several masks. The created ReasonSeg3D is claimed to be large-scale and high-quality because the question-answer pairs are generated by GPT-4o with access to scene images and ground-truth segmentation, then human-verified, with about 3% corrected.","pith_inferences":["A second independent human annotation pass on a sample of ReasonSeg3D would test whether GPT-4o's spatial-relation descriptions contain systematic errors; the paper reports no such study.","Because the multi-seg index list is taken from ground-truth answers during training, MORE3D is not trained to decide how many objects a question implies; a variant that predicts the object count would probe whether the reasoning truly generalizes.","The reported comparisons train all baselines on the same ReasonSeg3D split, so the gap over open-vocabulary methods may partly reflect their inability to emit multi-object <SEG> answers rather than a pure reasoning deficit.","Per-question-type score breakdowns would show whether the performance gain comes from hard spatial-relation questions or from easy single-object cases that dominate the aggregated cIoU and gIoU."],"forward_implications":["Future 3D reasoning-segmentation methods can be trained and compared on a common multi-object benchmark, since ReasonSeg3D provides 20,113 question-answer-mask triplets over 1,513 scenes.","Injecting 3D spatial relations into textual answers improves both explanation quality and mask accuracy; the ablation shows that either loss alone raises cIoU and gIoU, and both together give the largest gain.","Single-object 3D reasoning methods such as Reason3D transfer poorly to multi-object questions, indicating that multi-object reasoning requires multiple <SEG> tokens with object-specific embeddings.","Separating mask prediction and classification into independent decoder branches yields better cIoU and gIoU than a unified head.","Because the benchmark's masks come from ScanNetv2 instance annotations, ReasonSeg3D can be reused with existing 3D segmentation infrastructure."],"supporting_citations":[{"why":"Supplies the ScanNetv2 point clouds and ground-truth instance masks that ReasonSeg3D builds on.","marker":"(Dai et al. 2017)"},{"why":"Introduces reasoning segmentation with <SEG> tokens and the cIoU and gIoU evaluation metrics adopted here.","marker":"(Lai et al. 2024)"},{"why":"Reason3D is the closest 3D reasoning-segmentation baseline and the main comparison point for single-object reasoning.","marker":"(Huang et al. 2025)"},{"why":"SegPoint is the LLM-based 3D segmentation baseline that the multi-object benchmark must beat.","marker":"(He et al. 2024)"},{"why":"3D-LLM is a general 3D scene-understanding baseline re-implemented on the reasoning-segmentation task.","marker":"(Hong et al. 2023)"},{"why":"3D-VisTA is a pretrained 3D vision-text baseline re-implemented for comparison on ReasonSeg3D.","marker":"(Zhu et al. 2023)"},{"why":"LLaMA-7B is the multimodal LLM backbone that MORE3D fine-tunes with LoRA.","marker":"(Touvron et al. 2023)"},{"why":"LoRA is the parameter-efficient fine-tuning method used to train the LLM in MORE3D.","marker":"(Hu et al. 2021)"}],"fun_headline_variants":["MORE3D segments multiple 3D objects from one question","New benchmark ReasonSeg3D enables multi-object 3D reasoning","MORE3D uses spatial relations to reason and segment scenes","One reasoning question yields many 3D masks with MORE3D","MORE3D tackles complex multi-object 3D scenes with reasoning"],"cache_read_input_tokens":16000,"weakest_assumption_plain":"The load-bearing premise is that ReasonSeg3D's question-answer pairs are trustworthy ground truth: they are generated by GPT-4o and then human-verified, with only about 3% corrected, and the paper reports no independent check for systematic errors in the spatial-relation language or in which objects the questions select.","fun_headline_variants_meta":{"raw":{"variants":["MORE3D segments multiple 3D objects from one question","New benchmark ReasonSeg3D enables multi-object 3D reasoning","MORE3D uses spatial relations to reason and segment scenes","One reasoning question yields many 3D masks with MORE3D","MORE3D tackles complex multi-object 3D scenes with reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000415,"raw_usage":{"total_tokens":2146,"prompt_tokens":954,"completion_tokens":1192,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":1101}},"tokens_in":570,"tokens_out":1192,"duration_ms":10094,"temperature":1.0,"reasoning_tokens":1101,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:44:00.609131+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Randomly sample ReasonSeg3D validation questions, have independent human annotators rewrite answers and select objects directly from the point cloud, then compare with the benchmark labels; if a meaningful fraction disagree on the objects or spatial relations, the reported 30.19 and 32.01 scores are measured against faulty labels. Alternatively, train MORE3D on a human-created subset of question-answer pairs and test on the same validation set; a large drop would show that the GPT-4o generation, not the model's reasoning, is doing the work.","supporting_citations":[{"cited_title":"X.; Savva, M.; Halber, M.; Funkhouser, T.; and Nie ner, M","cited_arxiv_id":null,"evidence_quote":"Supplies the ScanNetv2 point clouds and ground-truth instance masks that ReasonSeg3D builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reason3D is the closest 3D reasoning-segmentation baseline and the main comparison point for single-object reasoning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"3D-VisTA is a pretrained 3D vision-text baseline re-implemented for comparison on ReasonSeg3D."}],"review_version":1}