{"id":"1304505d-74cb-4063-93a1-e8a1dea0ce8c","arxiv_id":"2501.09428","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AugRefer improves 3D visual grounding by augmenting training data with generated object-scene-caption pairs and by injecting global and pairwise spatial relations into the decoder attention.","lead":"This paper presents AugRefer, a method that creates extra training examples for 3D visual grounding by inserting objects into 3D scenes and generating descriptions with vision-language models. It also adds a decoder that uses spatial relationships in a scene to help locate the object named in a sentence, and it can be added to existing grounding models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's augmentation source pool is not restricted to the training split, so test-split object point clouds may leak into augmented training data; reported gains could reflect leakage rather than method.","rationale":"The reader flagged caption noise as the weakest assumption, and Table 5 does show degradation when the number of augmented pairs per scene rises from 3 to 5. That is a legitimate concern and supports a conditional verdict. However, a more decisive threat is the unconstrained augmentation source pool: Algorithm 1's S and 'another scene Sj' are not qualified by train/val/test split. Since ScanRefer/Nr3D/Sr3D are built from ScanNet scenes and the model trains on augmented pairs, any object sampled from an evaluation scene is leaked. The inserted object's category is even appended to prompts in the baselines, so the model could see category and geometry from a test object. This is not an accusation; it is a missing specification in a setting where the default reading (all ScanNet scenes) is harmful. Because no code is released, this cannot be verified from the manuscript. I also note the paper's 'SOTA on ScanRefer' wording is contradicted by Table 1 (CORE-3DVG 56.77 vs. EDA+AugRefer 55.68 on overall Acc@0.25), but that is a framing issue; the leakage question is the one that could overturn the empirical core. I therefore keep the reader's CONDITIONAL verdict until the split-safety check is run.","tokens_in":14869,"tokens_out":6608,"duration_ms":70430,"concrete_test":"Obtain the augmentation pipeline or config (code or author clarification) and determine the scene pool S used in Algorithm 1 for each dataset. Verify that S contains only scenes belonging to that dataset's training split, then rerun BUTD-DETR + AugRefer on ScanRefer with a strictly split-excluded version. If overall Acc@0.25 no longer shows the reported +3.05 gain, or the gap to Table 1 changes materially, the headline empirical claim is contaminated by test leakage. If the pipeline is split-safe, this concern is resolved and the reported gains stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 (Appendix A) samples the object to insert by 'randomly choosing another scene Sj' from '3D indoor dataset S' with no statement that S is restricted to the training split. Section 3.1 similarly says the external object is 'randomly selected from other scenes.' The experiments then add 22.5k generated pairs to training. If S is the full ScanNet used by ScanRefer/Nr3D/Sr3D, an object's point cloud can appear both in an augmented training scene and in an evaluation scene, giving the model direct access to test-set geometry at training time. The paper never specifies split-level separation for the augmentation source pool and releases no code, so this cannot be checked from the manuscript. If leakage is present, the +3.05/+9.81/+6.58 Acc@0.25 gains over BUTD-DETR and the +2.10/+4.41/+6.56 gains over EDA are not valid evidence for the method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AugRefer, a method for 3D visual grounding with two components: (1) a cross-modal augmentation pipeline that inserts external 3D objects into scenes, renders them at multiple angles and granularities, and uses BLIP2 and GPT-3.5 to generate new text-3D training pairs; and (2) a Language-Spatial Adaptive Decoder (LSAD) that injects global and pairwise spatial relations into cross-modal attention. The method is integrated into BUTD-DETR and EDA and evaluated on ScanRefer, Nr3D, and Sr3D, reporting consistent accuracy gains and SOTA or near-SOTA results, along with ablations in Tables 3-7.","tokens_in":15065,"tokens_out":4100,"duration_ms":41941,"significance":"If the reported results are valid, the cross-modal augmentation is a valuable plug-and-play contribution that can enrich text-3D training data for existing 3DVG methods, and the LSAD decoder provides a concrete way to exploit spatial relations in transformer decoders. The paper's extensive ablations (Tables 3-7) support both components individually and in combination, and the proposed augmentation is clearly described at a conceptual level. The main risk to significance is the unresolved possibility of train/test leakage in the augmentation source pool, which could invalidate the reported gains; this must be addressed before the results can be accepted as evidence for the method.","major_comments":[{"comment":"Algorithm 1 samples the object to insert by 'randomly choosing another scene Sj' from '3D indoor dataset S' (the ScanNet dataset), with no statement that Sj is restricted to the training split. The main text similarly says the external object is 'randomly selected from other scenes.' Since ScanRefer, Nr3D, and Sr3D are all built from ScanNet, this creates a concrete risk that a point cloud from a test scene is inserted into a training scene, giving the model direct access to test-set geometry during training. If this is the case, the reported gains (e.g., +9.81 on Nr3D, +6.58 on Sr3D) are not valid evidence for the method. The authors must specify explicitly that the augmentation source pool is restricted to the training split, and if it is not, they must rerun the experiments with a split-restricted pool and report the new numbers.","section":"Algorithm 1 (Appendix A) and Sec. 3.1"},{"comment":"The augmentation quantity n is chosen as 3 based on the overall Acc@0.25 values reported in Table 5, which appear to be test-set results on ScanRefer. The text says 'we set n to 3' directly from these results, but there is no statement that this decision was made on a held-out validation split. If Table 5 reports test-set performance, this is test-set tuning of a central hyperparameter, which inflates the reported effectiveness of the augmentation component. The authors should clarify the split used for model selection, or provide validation-based selection.","section":"Sec. 4.3, Table 5"},{"comment":"All reported numbers are single-run point estimates with no error bars, variance, significance tests, or multiple-seed averages. Since the claimed improvements over the baselines are often in the 2-4 percentage-point range (e.g., +2.10 on ScanRefer, +4.41 on Nr3D), run-to-run variance could be a substantial fraction of these differences. The authors should provide at least three independent runs with mean and standard deviation for the main comparisons, or otherwise justify that the gains are outside training noise.","section":"Sec. 4.1 and Tables 1-3"},{"comment":"The augmentation pipeline contains several hyperparameters that are never specified, most notably the value of M used to select the top CLIP images for captioning, and the exact number of rendered images per insertion level. Without these values, the augmentation pipeline is not fully reproducible. Please report all thresholds and counts used in the final experiments.","section":"Sec. 3.1 and Implementation Details"},{"comment":"The pairwise spatial relation vector rp_ij is defined only by reference to 'Chen et al. 2022' rather than being specified in the paper. To make the LSAD module self-contained and reproducible, the paper should give the exact formula for the five-dimensional relation vector (distance, sine and cosine of horizontal and vertical angles) in the main text or appendix.","section":"Appendix A, Pairwise Spatial Attention"}],"minor_comments":[{"comment":"The row labeled 'SAT (Roh et al. 2022)' appears to cite the wrong reference: SAT is Yang et al. 2021b, while LanguageRefer is Roh et al. 2022. Please correct the citation.","section":"Table 2"},{"comment":"The text says 'our cross-model augmentation' where 'cross-modal' is intended; there are also inconsistent capitalizations of 'SR3D' and 'NR3D' across the manuscript.","section":"Sec. 4.3"},{"comment":"The description of the multi-angle camera placement states cameras are positioned at 0, 45, and 90 degrees and rotated around the object, but Fig. 3(a) shows only 'Angle1, Angle2, Angle3'; clarifying the exact camera trajectories would help reproducibility.","section":"Fig. 3 and Sec. 3.1"},{"comment":"The reference 'Yang et al. 2024a' is listed as NeurIPS 2023 in the bibliography but '2024a' in the citations; please standardize the year and venue information.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The leakage concern in Algorithm 1 is the key issue: the manuscript's text and algorithm permit sampling augmentation-source objects from the full ScanNet dataset without enforcing train/test separation. This is a correctness risk, not just a presentation issue. Please require the authors to state the split restriction unambiguously and, if the original experiments did not use a split-restricted pool, to rerun and report corrected results. The absence of code makes this even harder to verify, so a clear written commitment or released code would be desirable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the thing to know: AugRefer's cross-modal augmentation is genuinely new for 3D visual grounding, and the LSAD decoder is a sensible plug-in. But the paper never states that the object source pool for augmentation is restricted to the training split, and that is a load-bearing ambiguity. Algorithm 1 samples 'another scene Sj' from '3D indoor dataset S' with no split qualifier. Since the benchmarks (ScanRefer, Nr3D, Sr3D) all use ScanNet scenes with fixed train/test splits, if S includes test scenes, then object point clouds from evaluation scenes are being pasted into training scenes. That would give the model direct access to test geometry, and the +3.05/+9.81/+6.58 gains over BUTD-DETR would not reflect the method. The paper does not rule this out, and no code is released to check.\n\nWhat is legitimately good: the combination of object insertion, multi-angle/multi-level rendering, and BLIP2/GPT-3.5 captioning produces text-3D pairs that improve two strong baselines across three benchmarks. The ablations in Tables 3-7 support both the augmentation and the spatial decoder. LSAD injecting global and pairwise spatial relations is a nice extension of prior spatial attention work.\n\nSoft spots beyond the leakage: the SOTA claim is overstated. Table 2 shows EDA+AugRefer at 46.49 on Nr3D, while CORE-3DVG is 49.57, so it is not SOTA there; only on Sr3D does it surpass. The paper should say 'top among open-source methods' accurately. Also, all results are single-run point estimates, no error bars, and the augmentation quantity n=3 is tuned on validation (Table 5). The caption pipeline has unreported thresholds like the top-M CLIP selection. The multi-level ablation is odd: each level alone hurts, only all three together help, which could indicate a threshold effect but deserves an explanation.\n\nWho this is for: researchers working on 3D visual grounding, particularly data augmentation and spatial relation modeling. They will find the augmentation pipeline a useful recipe, and the decoder is easy to adapt.\n\nOverall: deserves a serious referee, because the core idea is useful and the experiments are extensive, but the referee must demand clarification on the split-level separation of the augmentation source pool, plus code or data release and error bars. If the leakage is real, the empirical claims collapse; if it is not, the method is a solid incremental contribution.","headline":"Useful cross-modal augmentation for 3DVG, but a potential test-set leakage in the augmentation source pool must be resolved before the reported gains can be trusted.","tokens_in":15583,"tokens_out":3640,"would_cite":false,"duration_ms":34419,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AugRefer claims that generating synthetic text-3D pairs and injecting spatial relations into decoder attention lifts 3D visual grounding accuracy by up to 9.81 points on Nr3D and 6.58 points on Sr3D over the BUTD-DETR baseline.","keywords":["3D visual grounding","cross-modal augmentation","data augmentation","spatial relation reasoning","transformer decoder","point cloud","referring expression comprehension","foundation model captioning"],"falsifier":"A concrete check: take the augmented pairs and test whether a strong vision-language model can re-locate the inserted object from the generated caption in the rendered image; a low re-grounding hit rate would indicate the captions are not aligned with the inserted object and would undermine the data-side claim. A cheaper quantitative version is to train the baseline on original pairs plus the same number of caption-object pairings randomly shuffled; if accuracy matches the AugRefer gain, the benefit is quantity rather than semantic alignment.","tokens_in":14677,"feed_emoji":"🎯","tokens_out":7940,"duration_ms":72092,"temperature":0.7,"pith_summary":"AugRefer addresses two bottlenecks in 3D visual grounding: too few text-3D training pairs and underuse of spatial context. Its cross-modal augmentation inserts new objects into real 3D scenes, renders them at object, local, and scene levels, and uses foundation models to generate and refine descriptions, producing about 22.5k extra training pairs. Its Language-Spatial Adaptive Decoder (LSAD) injects global and pairwise spatial relations directly into transformer attention so the model can tell the target apart from distractors. Across ScanRefer, Nr3D, and Sr3D, AugRefer improves both BUTD-DETR and EDA, including a +9.81 Acc@0.25 gain on Nr3D and +6.58 on Sr3D over BUTD-DETR, reaching state-of-the-art results on ScanRefer and Sr3D. The authors' claim is that the two mechanisms are complementary and together translate into consistent gains for existing 3DVG models.","feed_headline":"AugRefer lifts 3D visual grounding by up to 9.81 points","feed_subtitle":"Adding synthetic text-3D pairs and spatial attention pushes two baselines to state-of-the-art on three benchmarks.","key_machinery":"The load-bearing mechanism is the Language-Spatial Adaptive Decoder (LSAD), a transformer decoder layer whose attention is augmented with spatial biases. Pairwise spatial relations between object proposals—Euclidean distance plus horizontal and vertical sine/cosine components of the connecting line—are MLP-mapped and added to the attention logits; global spatial relations are the normalized coordinates of the object center in the scene, injected the same way. The companion data mechanism is the cross-modal augmentation pipeline: stander objects are inserted on the ground plane with collision-free placement, rendered from three angles at three granularities (object, local, scene), and captioned by BLIP2 with prompts tailored to each level, then corrected and rephrased by GPT-3.5.","core_discovery":"The central claim is that 3D visual grounding can be advanced by combining two mechanisms: data-side, synthetic text-3D pairs generated by inserting objects into scenes and captioning the resulting renders with foundation models; and model-side, a decoder that explicitly injects spatial relations into attention. The paper demonstrates the claim by appending both components to two transformer-based baselines and showing accuracy improvements on every one of the three benchmarks. The decoder carries the model-side argument: LSAD applies cross-attention to language first, then pairwise spatial attention over object proposals, then global spatial attention over scene-wide position, with spatial features added to the attention logits. The augmentation carries the data-side argument: multi-angle, multi-level rendering plus BLIP2/GPT-3.5 captioning produces about 22.5k extra pairs, and the authors identify three generated pairs per scene and level as the best working point.","pith_inferences":["Inference: The augmentation recipe appears portable beyond grounding: unlabeled 3D scenes plus an object bank could generate paired data for navigation instruction, spatial QA, or scene-graph tasks, as long as a captioner can reliably describe the inserted object.","Inference: The ceiling on augmentation benefit is caption fidelity. A consistency filter—e.g., trying to re-ground each generated caption to the inserted object and dropping failures—would likely let models use more than three pairs per scene before noise dominates.","Inference: The spatial-bias-in-attention design suggests a generic recipe: encode relative geometry as MLP-mapped vectors and add to QK^T, which could be applied to other relation-sensitive tasks beyond 3D, such as video grounding or multi-object tracking.","Inference: Because the paper only tests two baselines (the strongest open one and an older one), the claim that the pairs are universally usable remains open; testing on the not-open-source strong model or future ones would be the natural next check."],"forward_implications":["Any transformer-based 3DVG model can take the generated text-3D pairs without changing its architecture; training time grows but inference time does not.","Explicit spatial attention should help most in 'multiple' splits, where distractors of the same category must be told apart by relation to anchors or scene position.","The three caption granularities are complementary: each alone helps less than the combination, so descriptions should mix object attributes, local anchors, and scene-level context.","Language-first ordering in the decoder matters: grounding with text before spatial aggregation works better than spatial-first alternatives.","Adding too many generated pairs per scene (five instead of three) hurts accuracy, so augmentation quantity should be balanced against caption noise."],"supporting_citations":[{"why":"Supplies the BLIP2 captioner that generates the multi-level descriptions of inserted objects.","marker":"Li et al. 2023"},{"why":"Supplies the GPT-3.5 model used to correct, summarize, and rephrase those captions.","marker":"Brown et al. 2020"},{"why":"Provides the BUTD-DETR transformer baseline whose decoder is replaced by LSAD in the main experiments.","marker":"Jain et al. 2022"},{"why":"Provides the EDA baseline, the strongest open-source model tested, on which AugRefer reports the largest overall gains.","marker":"Wu et al. 2023"},{"why":"Defines the ScanRefer benchmark used for the main quantitative comparisons.","marker":"Chen et al. 2020"},{"why":"Defines the ReferIt3D (Nr3D and Sr3D) benchmarks with ground-truth objects used for the second set of comparisons.","marker":"Achlioptas et al. 2020"},{"why":"Supplies the 3D Copy-Paste object insertion approach that motivates physically plausible object insertion, here extended to generating text-3D pairs.","marker":"Ge et al. 2024"},{"why":"Supplies the pairwise spatial encoding (distance plus sine/cosine directions) reused in LSAD and the alternative Vil3DRef decoder used in the ablation.","marker":"Chen et al. 2022"}],"fun_headline_variants":["AugRefer: synthetic pairs and spatial decoder lift 3D grounding","3D grounding improves when AugRefer adds synthetic text-3D pairs and spatial attention","AugRefer's dual boost: synthetic data and spatial relation decoder","Boosting 3D grounding with cross-modal augmentation and spatial reasoning","AugRefer: more text-3D data and spatial logic for accurate grounding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the captions produced by BLIP2 and GPT-3.5 are accurate and semantically aligned with the inserted object; if they are noisy, the extra pairs mislead training—and the paper's own ablation shows accuracy falling when the number of generated pairs per scene rises from three to five.","fun_headline_variants_meta":{"raw":{"variants":["AugRefer: synthetic pairs and spatial decoder lift 3D grounding","3D grounding improves when AugRefer adds synthetic text-3D pairs and spatial attention","AugRefer's dual boost: synthetic data and spatial relation decoder","Boosting 3D grounding with cross-modal augmentation and spatial reasoning","AugRefer: more text-3D data and spatial logic for accurate grounding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000929,"raw_usage":{"total_tokens":3970,"prompt_tokens":925,"completion_tokens":3045,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":2949}},"tokens_in":541,"tokens_out":3045,"duration_ms":28402,"temperature":1.0,"reasoning_tokens":2949,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:01:28.654112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: take the augmented pairs and test whether a strong vision-language model can re-locate the inserted object from the generated caption in the rendered image; a low re-grounding hit rate would indicate the captions are not aligned with the inserted object and would undermine the data-side claim. A cheaper quantitative version is to train the baseline on original pairs plus the same number of caption-object pairings randomly shuffled; if accuracy matches the AugRefer gain, the benefit is quantity rather than semantic alignment.","supporting_citations":[{"cited_title":"D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al","cited_arxiv_id":null,"evidence_quote":"Supplies the GPT-3.5 model used to correct, summarize, and rephrase those captions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the BUTD-DETR transformer baseline whose decoder is replaced by LSAD in the main experiments."},{"cited_title":"Z.; Chang, A","cited_arxiv_id":null,"evidence_quote":"Defines the ScanRefer benchmark used for the main quantitative comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the ReferIt3D (Nr3D and Sr3D) benchmarks with ground-truth objects used for the second set of comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pairwise spatial encoding (distance plus sine/cosine directions) reused in LSAD and the alternative Vil3DRef decoder used in the ablation."}],"review_version":1}