{"id":"189f71ef-a1d4-4fba-97cf-b7867e5df265","arxiv_id":"2501.04995","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"IPDN improves 3D referring expression segmentation by fusing CLIP image features into point clouds and using task-driven prompts to focus the decoder.","lead":"This paper presents a neural network for 3D scene segmentation from natural language descriptions, combining multi-view image features with a prompt-aware decoder. It reports the best numbers so far on two standard benchmarks for this task.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 1.9-point mIoU gain over MDIN may be within run-to-run variance; no error bars or significance test are reported, and key details (PAD layer-1 initialization, spherical query radius) are underspecified.","rationale":"The reader's weakest_assumption focuses on camera calibration sensitivity, which is an external-validity concern: it would matter if IPDN were deployed on data without reliable poses or depth, but the benchmark itself (ScanRefer, Multi3DRefer) is built on ScanNet, which provides exactly these. Thus that assumption is satisfied for the claimed SOTA results and does not threaten the central claim as stated. The more load-bearing risk is internal: the empirical gap is small (1.9 mIoU), and the paper provides no evidence that this gap exceeds run-to-run noise. The absence of error bars is a well-known issue in this literature, and a 1-2 point mIoU difference can easily be non-significant. Additionally, the underspecified PAD initialization and spherical query radius are concrete reproducibility gaps; while the reader's rationale mentions these, they are not the weakest_assumption. I therefore partially agree with the reader: the camera-calibration concern is valid for generalization but not for the benchmark claim, while the statistical robustness and specification issues are more directly load-bearing. The recommended verdict stays CONDITIONAL because the paper needs additional runs and details before the SOTA claim can be fully accepted, but the concern does not warrant rejection outright.","tokens_in":14602,"tokens_out":4709,"duration_ms":47545,"concrete_test":"Using the released code (if accessible), train IPDN and MDIN on ScanRefer and Multi3DRefer with at least 5 different random seeds. Report mean and standard deviation of mIoU and run a paired significance test (e.g., bootstrap or paired t-test) between IPDN and MDIN. If the 1.9-point difference is within one standard deviation or not significant, the headline claim is unsupported. Also inspect the code to determine the initialization of P rob^0 for layer 1 of the PAD and the spherical query radius; if these are absent from the paper, document them and verify that the disclosed equations match the implementation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: IPDN outperforms the previous SOTA by 1.9 mIoU on ScanRefer and 4.2 mIoU on Multi3DRefer. The paper reports a single run for each configuration, with no error bars, no number of seeds, and no significance test. In 3D referring segmentation, reported mIoU differences of 1-2 points are often within run-to-run variance, especially with models trained on ~30k samples and evaluated on a small test split. Table 2 shows IPDN at 50.2 overall mIoU versus MDIN at 48.3; however, the ablation baseline without MSE and PAD already reaches 48.1, essentially matching MDIN. The claimed margin over the previous SOTA therefore depends almost entirely on the two new modules, each contributing about 1 point in the ablation (Table 4). Without repeated runs, the 1.9-point advantage is not demonstrably real. Independently, the method is underspecified: Eq. (8) filters queries using P rob^{l-1}, but P rob^0 is never defined for the first decoder layer; and Sec. 3.2 uses spherical querying without reporting the sphere radius. These omissions prevent faithful reproduction, even with code, and could hide implementation choices that materially affect the reported numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IPDN, a one-stage query-based network for 3D Referring Expression Segmentation (3D-RES) and Generalized 3D Referring Expression Segmentation (3D-GRES). It introduces two modules: MSE (Multi-view Semantic Embedding), which projects CLIP-extracted 2D image features into the point cloud via camera parameters and integrates them with 3D superpoint features under a spatial-aware attention mechanism, and PAD (Prompt-aware Decoder), which generates task-driven prompts from cross-attention scores between text and queries, filtering irrelevant queries using predicted probabilities. Experiments on ScanRefer and Multi3DRefer report improvements over the previous state of the art MDIN by 1.9 mIoU on 3D-RES and 4.2 mIoU on 3D-GRES, with ablations showing each module contributes roughly 1 point of mIoU.","tokens_in":14876,"tokens_out":3466,"duration_ms":35967,"significance":"If the reported gains are statistically robust, IPDN is a meaningful advance: it combines large-scale 2D vision-language pretraining with a lightweight task-driven prompt mechanism, and the long-tail analysis in Table 3 shows a substantially smaller performance drop on rare classes than prior 3D-only methods, which is a credible benefit of injecting CLIP features. The paper is clearly motivated, the component ablations are internally consistent, and the authors release code, which aids reproducibility. However, the current evidence is weakened by single-run evaluations without error bars or significance tests, and several implementation details needed for faithful reproduction or for assessing generality are missing. The significance is therefore contingent on addressing these empirical and specification gaps.","major_comments":[{"comment":"The central claim of state-of-the-art performance rests on single-run comparisons. IPDN reports 50.2 overall mIoU versus MDIN's 48.3 on ScanRefer, but the ablation without MSE and PAD already reaches 48.1, so the 1.9-point margin over the previous SOTA is essentially the sum of two roughly 1-point module gains. On a test set of this size, 1–2 mIoU differences are commonly within run-to-run variance, and no error bars, number of seeds, or significance tests are reported. Please provide mean and standard deviation over at least three runs for the main comparisons and the key ablations, or otherwise justify that the reported differences are not noise.","section":"Sec. 4.3, Tables 2 and 4"},{"comment":"The filtering operation in Eq. (8) uses P rob^{l−1}, but P rob^0 is never defined for the first decoder layer. Since PAD is one of the two main contributions, the paper must specify how the first-layer probabilities are initialized (for example, uniform values, a prediction from Q0, or skipping filtering at l=1), and confirm that the reported ablations use that exact choice.","section":"Sec. 3.3, Eq. (8)"},{"comment":"The spherical querying step in the MSE module is underspecified: the sphere radius is not reported, the aggregation over multiple views is described only as an average, and the number of input views N_I and how they are selected are not given in Section 4.1. These choices directly control how much 2D information enters the 3D features and could materially affect the results; they need to be stated explicitly, ideally with a sensitivity analysis.","section":"Sec. 3.2"},{"comment":"MSE assumes accurate camera intrinsics, extrinsics, and per-pixel depth for every input view, and the evaluation is limited to ScanNet-based benchmarks that provide this metadata. The paper does not test sensitivity to calibration error, missing views, or imperfect depth, and it does not discuss behavior on data without such metadata. Please add a robustness experiment or explicitly bound the generality claim so that the reported gains are not over-interpreted as holding in settings where the required camera information is unavailable.","section":"Sec. 3.2, Eq. (5), Sec. 4.2"}],"minor_comments":[{"comment":"The section heading reads \"Expriments\" and should be \"Experiments\"; Table 1's header duplicates \"Acc@0.25 Acc@0.5\" without a separating line, which makes the ZT/ST/MT columns hard to parse.","section":"Section 4 heading and Table 1"},{"comment":"The projection formula uses R(K^{-1}[u,v,1]^T D) + T; please clarify the coordinate convention (e.g., whether R is the rotation from camera to world or its inverse) so that the projection can be reproduced exactly.","section":"Eq. (5)"},{"comment":"The learning-rate schedule is called \"PolyRL\" and is likely a typo for \"PolyLR\"; please correct it and state the optimizer and weight decay.","section":"Sec. 4.1"},{"comment":"There are several small language issues: \"r is not necessarily better when it is larger\" should be \"a larger r is not always better\"; \"P robvalues\" needs a space; and the abstract's \"state-ofthe-art\" should be \"state-of-the-art.\"","section":"Sec. 4.4 and throughout"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a modest, well-constructed increment over the authors' own MDIN system. The headline gains are plausible but not yet proven, because every number comes from a single run with no error bars, and a couple of implementation details are missing.\n\nThe genuinely new pieces are the two modules. MSE projects CLIP patch features into the point cloud using camera parameters, then applies spatial-aware attention on seed queries to restore positional cues. PAD filters cross-attention relevance scores by the upper layer's predicted objectness probability before softmax, producing task-driven prompts that steer decoding. Both are simple and well motivated. The ablations show each module contributes roughly 1 mIoU on ScanRefer, consistently across Acc@0.25 and Acc@0.5. The frequency-based evaluation is a nice extra: IPDN beats MDIN by about 10 mIoU on low-frequency classes, which supports the claim that 2D pretraining helps long-tail recognition. Code is public.\n\nThe soft spots are real but fixable. The 1.9-point mIoU lead over MDIN is within the range where run-to-run variance matters, and no seeds or significance tests are reported. The ablation baseline without either module reaches 48.1, essentially matching MDIN at 48.3, so the entire claimed advantage rests on the two modules, each worth about a point. That is a thin margin to hang a SOTA claim on without repeated runs. The paper also underspecifies two details: Eq. (8) uses Prob^{l-1} but Prob^0 is never defined for the first decoder layer, and Sec. 3.2's spherical querying never gives the sphere radius. The MDIN sampling and feature-fusion modules are treated as black boxes, which makes exact reproduction harder even with code. The MSE module assumes accurate intrinsics, extrinsics, and depth; the paper only tests on ScanNet, so sensitivity to calibration error is unknown.\n\nFor someone working on 3D referring segmentation or grounding, this is worth reading and citing. It does not reorganize the field, but it is a coherent step forward. My recommendation: send it to peer review. Ask the authors for multiple seeds with error bars, define Prob^0 and the sphere radius, and ideally add a calibration-noise ablation for MSE. A serious referee can turn this into a solid paper; as is, I would be skeptical of the exact margins but not of the direction.","headline":"A sensible incremental combination of CLIP multi-view fusion and thresholded prompt decoding for 3D referring segmentation, with plausible single-run gains that need repeated-seed confirmation.","tokens_in":15418,"tokens_out":2503,"would_cite":true,"duration_ms":26319,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IPDN beats prior 3D-RES models by 1.9 and 4.2 mIoU points by injecting CLIP image features and task-driven prompts into the decoder.","keywords":["3D referring expression segmentation","point cloud segmentation","multi-view semantic embedding","prompt-aware decoder","task-driven prompts","CLIP","ScanRefer","Multi3DRefer"],"falsifier":"Perturb the camera intrinsics and extrinsics or add realistic depth noise to the ScanRefer validation views and re-measure IPDN's mIoU; if the gain over a point-cloud-only baseline vanishes or turns negative under calibration errors within sensor tolerance, the central claim of the MSE module is not robust.","tokens_in":14407,"feed_emoji":"🎯","tokens_out":6566,"duration_ms":53158,"temperature":0.7,"pith_summary":"The paper claims that the two known weaknesses of 3D referring expression segmentation—features computed from point clouds alone are noisy and under-aligned with language, and decoders treat every object query as equally relevant—can be fixed by two targeted modules. The proposed IPDN injects multi-view CLIP image features into the point cloud via geometric projection to repair lost or distorted visual information, and adds a prompt-aware decoder that derives task-driven guidance from cross-attention between text and queries. The authors report that on standard benchmarks this raises mean Intersection-over-Union by 1.9 points for 3D-RES and by 4.2 points for 3D-GRES, with the largest gains on rare object classes and on scenes cluttered with distractors. If those numbers hold, a practical takeaway is that using cheap 2D pretrained features plus a text-relevance signal is a reliable way to make 3D segmentation interpret language.","feed_headline":"Images and prompts lift 3D referring segmentation by 1.9 and 4.2 mIoU","feed_subtitle":"Multi-view CLIP features and text-aware prompts win on ScanRefer and Multi3DRefer.","key_machinery":"The two load-bearing mechanisms are the Multi-view Semantic Embedding (MSE) module and the Prompt-Aware Decoder (PAD). MSE takes CLIP patch features upsampled to pixel level, projects each pixel into 3D via the camera intrinsics, extrinsics, and depth (Eq. 5), and uses spherical querying to attach the projected 2D features to nearby points, then adds a spatial-aware masked self-attention over k-nearest neighbor queries so the fused features keep explicit 3D relations. PAD runs cross-attention with text as query and sparse queries as keys and values, sums the attention scores to get per-query relevance, filters out low-confidence queries by thresholding with the previous layer's predicted probability, softmaxes the rest, and concatenates the resulting task-driven prompt to the queries before each fusion layer. The prediction head then selects the highest-probability query (3D-RES) or merges all queries with probability above 0.5 (3D-GRES).","core_discovery":"The paper's central claim is that both feature ambiguity and intent ambiguity in 3D referring expression segmentation are reducible to two architectural interventions. By projecting CLIP's pixel-level image features into the point cloud with known camera parameters and averaging over overlapping views, the MSE module produces a fused visual representation that aligns better with text and preserves spatial structure; by computing cross-attention scores between text and sparse object queries, thresholding them with the model's own confidence, and attaching the weighted queries as prompts, the PAD module steers the decoder toward text-relevant queries. The paper's controlled ablation attributes roughly equal gains to the two modules (1.1 and 1.0 mIoU points) and reports state-of-the-art results on ScanRefer and Multi3DRefer.","pith_inferences":["A testable extension is to run IPDN on RGB-D scans whose camera poses or depth maps are synthetically perturbed; if the mIoU gain over point-cloud-only baselines shrinks under realistic calibration noise, the geometric projection in MSE is the fragile link.","The same recipe of CLIP feature injection plus confidence-gated prompt decoding could transfer to 2D referring segmentation or open-vocabulary 3D instance segmentation, where feature ambiguity and query ranking also matter.","Because the paper only evaluates on ScanNet-derived benchmarks, the performance on noisy outdoor or multi-room scans, or scenes with missing views, remains unknown; the claimed gains may not extrapolate until the sensitivity to view coverage is measured."],"forward_implications":["On the ScanRefer benchmark, IPDN reports 60.6 Acc@0.25, 54.9 Acc@0.5, and 50.2 mIoU, ahead of the previous best MDIN by 2.6, 1.8, and 1.9 points respectively.","On Multi3DRefer, IPDN reports 51.7 mIoU overall, 4.2 points above MDIN, with the biggest single-target gain at Acc@0.5 (8 points without distractors).","The model's mIoU drops only 6.3 points from high-frequency to low-frequency object classes, versus 17.5 for 3D-STMN and 14.9 for MDIN, suggesting the 2D-pretrained features help rare categories.","Ablations show the two modules are approximately additive: MSE contributes 1.1 mIoU and PAD contributes 1.0 mIoU over the baseline without either."],"supporting_citations":[{"why":"Supplies the CLIP visual encoder whose pre-aligned image-language features MSE projects into the point cloud.","marker":"(Radford et al. 2021)"},{"why":"Provides the ScanRefer dataset and benchmark on which IPDN reports state-of-the-art 3D-RES results.","marker":"(Chen, Chang, and Nießner 2020)"},{"why":"Provides the Multi3DRefer dataset and 3D-GRES benchmark where IPDN reports its larger 4.2 mIoU gain.","marker":"(Zhang, Gong, and Chang 2023)"},{"why":"MDIN is the previous state-of-the-art baseline and the source of the sampling module and feature fusion procedure IPDN adopts.","marker":"(Wu et al. 2024a)"},{"why":"3D-STMN is the one-stage superpoint-text matching baseline that IPDN compares against and improves upon.","marker":"(Wu et al. 2024b)"},{"why":"RoBERTa is the pretrained text encoder used to extract word-level embeddings for the referring expression.","marker":"(Liu et al. 2019)"},{"why":"Supplies the superpoint generation and query-based instance segmentation framework IPDN adapts for sparse query generation.","marker":"(Sun et al. 2023)"}],"fun_headline_variants":["Image-enhanced prompts lift 3D referring segmentation by 1.9 and 4.2 mIoU","Multi-view images and prompt-aware decoder win on 3D referring segmentation","IPDN: image-based prompts boost 3D-RES by up to 4.2 mIoU","Fusing images and prompts improves 3D referring segmentation by 4.2 mIoU","3D referring segmentation: image prompts add 1.9–4.2 mIoU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The MSE module assumes every training and test view comes with accurate camera intrinsics, extrinsics, and per-pixel depth, since any error in the projection of Eq. (5) degrades the image features injected into the point cloud.","fun_headline_variants_meta":{"raw":{"variants":["Image-enhanced prompts lift 3D referring segmentation by 1.9 and 4.2 mIoU","Multi-view images and prompt-aware decoder win on 3D referring segmentation","IPDN: image-based prompts boost 3D-RES by up to 4.2 mIoU","Fusing images and prompts improves 3D referring segmentation by 4.2 mIoU","3D referring segmentation: image prompts add 1.9–4.2 mIoU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00097,"raw_usage":{"total_tokens":4116,"prompt_tokens":929,"completion_tokens":3187,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":3065}},"tokens_in":545,"tokens_out":3187,"duration_ms":22969,"temperature":1.0,"reasoning_tokens":3065,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:20:25.118289+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Perturb the camera intrinsics and extrinsics or add realistic depth noise to the ScanRefer validation views and re-measure IPDN's mIoU; if the gain over a point-cloud-only baseline vanishes or turns negative under calibration errors within sensor tolerance, the central claim of the MSE module is not robust.","supporting_citations":[{"cited_title":"Z.; Chang, A","cited_arxiv_id":null,"evidence_quote":"Provides the ScanRefer dataset and benchmark on which IPDN reports state-of-the-art 3D-RES results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RoBERTa is the pretrained text encoder used to extract word-level embeddings for the referring expression."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the superpoint generation and query-based instance segmentation framework IPDN adapts for sparse query generation."}],"review_version":1}