{"id":"87826c73-e298-4ec9-bee7-4cd302eed3a1","arxiv_id":"2411.19774","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"PerLA improves 3D question answering and dense captioning by combining Hilbert-curve partitioned local point-cloud details with global context through cross-attention and graph convolution.","lead":"PerLA is a 3D language assistant that splits a point cloud into local pieces using a Hilbert curve and merges those local details with a global view before feeding the result to a large language model. It reports state-of-the-art results on three 3D scene understanding benchmarks, ScanQA, ScanRefer, and Nr3D, with gains of about 1 to 4 points in CiDEr.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The k-NN neighbor selection in Eq. 3 relies on geometric labels from [27] that do not guarantee same-instance neighbors; if neighborhood purity is low, the reported gains are not attributable to the claimed local-to-global mechanism.","rationale":"The paper's central empirical claim is that PerLA outperforms prior 3DLAs because its perceptive scene encoder preserves local details and integrates them with global context without increasing visual tokens. The critical mechanism for that integration is the localized cross-attention in Eq. 3, which consumes the k nearest local super-points selected by the Hilbert/geometric-label search in §3.1. If that selection does not actually return same-object neighbors, then the cross-attention aggregates features from unrelated geometry, and the improvement attributed to 'perception' is not explained by the proposed design. The reader's weakest_assumption identified exactly this point, and I agree. The concern is concrete and testable: one can measure the instance purity of the selected neighborhoods and compare the approximate search against exact Euclidean k-NN. This is more load-bearing than the absence of code or error bars, because those affect reproducibility and statistical confidence, whereas a low-purity result would weaken the causal story behind the architecture even if the benchmark numbers are reproducible. I do not think this warrants rejection: the empirical gains could still hold, and the authors could add a failure analysis or alter the neighbor selection. CONDITIONAL acceptance with a request for purity statistics, code, and an exact-k-NN comparison would be appropriate.","tokens_in":24519,"tokens_out":8037,"duration_ms":75938,"concrete_test":"On ScanNet validation scenes, reproduce PerLA's neighborhood construction with L=6, M=1024, k=24, [27] geometric labels, and combined-index sorting. For each global super-point, compute the fraction of its 24 selected local super-points whose ground-truth ScanNet instance ID matches the global super-point's instance ID, and report the median over scenes. Compare with exact Euclidean k-NN and with random selection. If median purity is below roughly 0.5, or no better than the exact/random baselines, the same-instance guarantee is false and the aggregation is not selecting the intended neighbors. Additionally, retrain PerLA replacing the approximate search with exact Euclidean k-NN within the same geometric labels; if CiDEr on ScanQA validation shifts by more than about 1 point, the approximate search materially affects the reported results.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing assumption is in §3.1 (Eq. 3): for each global super-point p^g_i, the k=24 local super-points are selected by Hilbert-index proximity after imposing 'geometric labels' from [27]. The paper claims these labels 'guarantee that p^g_i and its nearest local super-points all originate from the same instance.' However, [27] produces over-segmented geometric super-points, not instance labels, and the propagation of these labels from the original point cloud to the FPS-downsampled global/local super-points is unspecified. A super-point can straddle an object boundary, or an object can be split across several labels; a global point near a label boundary can therefore have its top-24 serialized neighbors drawn from a different object. Because localized cross-attention (Eq. 3) is the only mechanism that injects local details into global representations, and because Tabs. 3-4 attribute the gains to this aggregation, a low same-instance purity of the selected neighborhoods would mean the reported improvements are not explained by the claimed perceptive mechanism, even if the benchmark numbers reproduce. No failure-case analysis or purity statistics are provided.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PerLA, a 3D language assistant for question answering and dense captioning. The method preserves local point-cloud detail by partitioning the scene with a Hilbert-curve serialization, encoding each partition with the same 3D encoder as the global cloud, and then aggregating local and global superpoint representations through a localized cross-attention module and a GCN message-passing layer. A consensus loss is added to the standard next-token prediction objective to regularize local representations. Experiments on ScanQA, ScanRefer, and Nr3D report consistent improvements over LL3DA and other baselines, with ablations showing that the proposed components each contribute positively. The central claim is that PerLA achieves state-of-the-art performance while keeping the number of visual tokens fed to the LLM fixed at 32.","tokens_in":24824,"tokens_out":3461,"duration_ms":30086,"significance":"If the reported gains are robust, the paper makes a useful contribution: it demonstrates that local detail can be injected into a scene-level 3D representation without increasing the LLM token budget, and the ablation study (Tabs. 3-5) supports the positive contribution of each proposed component. The method is also computationally lightweight compared to multi-view approaches such as 3D-LLM. However, the significance is moderated by two concerns that bear directly on the validity of the central claim: (i) the claimed 'same-instance' guarantee for the Hilbert-based k-NN neighborhoods is not substantiated and appears inconsistent with the nature of the superpoint labels used; and (ii) the headline improvement over the official LL3DA baseline is smaller than the discrepancy between that official baseline and the authors' own reproduction. These issues require additional analysis before the SOTA claim can be accepted.","major_comments":[{"comment":"The paper states that the geometric labels from [27] 'guarantee that p^g_i and its nearest local super-points all originate from the same instance.' This claim is not supported. Reference [27] produces over-segmented geometric superpoints, not instance labels, and the manuscript does not specify how these labels are assigned to the FPS-downsampled global and local superpoints, nor how 'same instance' is defined with respect to ScanNet instance annotations. Because the localized cross-attention in Eq. (3) is the only mechanism that injects local detail into the global representation, and because Tabs. 3-4 attribute the gains to this aggregation, the correctness of the proposed mechanism depends on the spatial purity of the selected neighborhoods. Please provide quantitative neighbor-purity statistics (e.g., the fraction of the k=24 neighborhoods whose members share a ScanNet instance label) on the ScanNet validation set, and a qualitative analysis of failure cases where the Hilbert-order or label boundaries cross object boundaries. If purity is low, the performance gains might instead be explained by generic spatial smoothing, which would require a revised interpretation of the method's contribution.","section":"Sec. 3.1, Eq. (3)"},{"comment":"The central SOTA claim is weakened by the discrepancy between the official LL3DA numbers and the authors' reproduction. In Tab. 1, official LL3DA scores 76.79 CiDEr on ScanQA validation, the authors' reproduction scores 74.37, and PerLA scores 78.13. The gain over the official baseline is +1.34, which is smaller than the 2.42-point gap between the official and reproduced LL3DA. This suggests that a non-negligible part of the observed improvement may come from differences in training data, schedule, or other setup choices rather than from the proposed perceptive encoder. Please report results of official LL3DA under the exact same data and training configuration as PerLA (or provide a detailed explanation of why the reproduction differs), and compare against that baseline. In addition, all reported numbers are from a single run; given that the headline margins are on the order of 1-4 CiDEr, please provide means and standard deviations over at least three seeds, or otherwise justify the stability of the results.","section":"Tabs. 1-2, Sec. 4.1"},{"comment":"The hyperparameters L (number of partitions), k (neighbors per global point), λ, μ, beam size, and the Hilbert grid resolution d are tuned on the validation set, and the final configuration is selected on the basis of those validation numbers. The differences between some configurations are small (e.g., Tab. 6 shows 6 vs. 8 partitions differing by 0.1 CiDEr on ScanQA), so the claimed optimality of the chosen settings is fragile. Please include a sensitivity analysis or error bars for the key hyperparameters (at least L and k), and clarify whether the same validation set was used for both model selection and the final reported numbers, in which case the reported performance may be optimistic.","section":"Sec. 4.2, Tabs. 5-6"}],"minor_comments":[{"comment":"The caption contains a typo: 'a more percepti solution' should be 'a more perceptive solution'.","section":"Fig. E caption (Supp. Mat.)"},{"comment":"The 'Hard Drive (GB)' entry for 3D-LLM is '74563.49', which appears to be a formatting error or an implausibly large value; please check the unit and the number.","section":"Supp. Tab. G"},{"comment":"The notation d_k is initially defined as a diagonal matrix representing row-wise sums of W^g, but in Eq. (5) it is used as a scalar denominator. Please clarify whether d_k is the degree of node k or a diagonal element of the degree matrix.","section":"Sec. 3.1, Eq. (5)"},{"comment":"The statement that the Hilbert-based k-NN search has 'O(1) complexity' is potentially misleading: the one-time serialization and sorting is O(N log N), and only the per-query lookup after that preprocessing is O(1). Please clarify that the O(1) complexity applies to query time after preprocessing.","section":"Sec. 3.1"},{"comment":"The rows labeled 'Global' and 'Local' are not fully defined in the main text; a one-sentence description of what these variants do (e.g., whether they use only the global branch or only the local branch for the final representation) would improve readability.","section":"Tab. 3"},{"comment":"For LL3DA (repr.), results are reported only on the validation split for ScanQA; please state why test-set numbers are omitted, or provide them if available.","section":"Sec. 4.1, Tab. 1"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the journal/conference and the empirical claims are plausible, but the 'same-instance guarantee' issue and the baseline-reproduction discrepancy are load-bearing. Both are addressable with additional experiments and a revised presentation, so major revision rather than rejection is warranted. I would also encourage the editor to ask for a released implementation or at least a clear reproducibility statement, given that the SOTA claim hinges on a comparison that the authors could not fully reproduce."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PerLA is a well-engineered 3D language assistant that gets consistent gains on ScanQA, ScanRefer, and Nr3D by adding local point-cloud detail through Hilbert-based partitioning and local-to-global aggregation, without increasing the token count to the LLM. The ablations are genuinely informative: each component (localized cross-attention, GCN, consensus loss, partition count) is tested, and the comparison against an LL3DA variant with double tokens (LL3DA†) makes the case that the gains come from the architecture rather than just more capacity.\n\nThe main novelty is the specific combination of known pieces. Hilbert serialization for point clouds is not new, and the introduction overstates the case by saying partitioning \"has not yet been applied to point clouds\" when their own references apply it. That sentence should be fixed. The bigger soft spot is the claim that geometric labels from Landrieu and Simonovsky's superpoint graphs \"guarantee\" that a global super-point and its k local neighbors come from the same instance. Those labels are over-segmented geometric partitions, not instance labels, and the propagation from original points to FPS-downsampled super-points is unspecified. The guarantee is therefore not justified. I don't think this sinks the paper—the method may still work with imperfect neighborhoods—but the authors should provide neighborhood-purity statistics or soften the claim. They also report no error bars or multiple seeds; the +1.34 CiDEr over the official LL3DA is smaller than the gap between the official number and their own LL3DA reproduction (76.79 vs 74.37), which suggests evaluation variance that should be quantified. Missing code also makes it hard to verify details like the Hilbert grid resolution.\n\nWho is this for? Anyone working on 3D vision-language models or point-cloud encoders for LLMs. It's an incremental but useful result that deserves proper peer review, with requests for code, variance reporting, and revision of the novelty and neighbor-guarantee claims. I'd send it to review, not desk-reject.","headline":"PerLA is a solid, incremental 3D language assistant with consistent benchmark gains; its novelty claim is overstated and the same-instance neighbor guarantee is unsupported, but the core method is sound.","tokens_in":25324,"tokens_out":2127,"would_cite":true,"duration_ms":19657,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Splitting a 3D scene into local parts before encoding lets a language assistant preserve fine details and beat state-of-the-art baselines on question answering and dense captioning.","keywords":["3D language assistant","point cloud perception","Hilbert curve","local-global aggregation","cross-attention","graph neural network","3D question answering","dense captioning"],"falsifier":"Measure, on ScanNet validation scenes, the instance-consistency precision of PerLA's approximate k-NN: for each global super-point, compute the fraction of its k local neighbors that share its ground-truth object instance label. If the fraction is high and correlates with downstream gains, the Hilbert and geometric neighbor assumption carries the result; if it is low or uncorrelated, the stated mechanism is not what explains the scores.","tokens_in":24297,"feed_emoji":"🤖","tokens_out":4829,"duration_ms":38518,"temperature":0.7,"pith_summary":"PerLA argues that 3D language assistants lose fine-grained scene information when they downsample or divide point clouds, and that the fix is to process high-resolution local parts alongside a low-resolution global view without sending extra tokens to the LLM. The paper claims that its perceptive scene encoder, built on Hilbert-curve serialization, approximate k-NN, localized cross-attention, and a GCN refinement step, captures details such as object color and small-object identity that global-only models miss. On ScanQA, ScanRefer, and Nr3D it reports state-of-the-art results, including gains of up to +1.34 CiDEr on ScanQA and +4.22 on ScanRefer over LL3DA. If correct, the work shows that local-global aggregation at the point level is a practical alternative to simply increasing token counts.","feed_headline":"Splitting 3D scenes into local parts sharpens language AI","feed_subtitle":"PerLA's Hilbert-curve encoder keeps fine detail with no extra LLM tokens, outperforming SOTA on ScanQA, ScanRefer, Nr3D.","key_machinery":"The Hilbert curve is the ordering that lets unordered point clouds be split into spatially coherent parts: the paper serializes the cloud with Hilbert indices, cuts it into L equal-size partitions, and uses the same serialized index (with geometric-label bits prepended) to run an O(1)-per-query approximate k-NN that finds local super-point neighbors. Localized cross-attention then updates each global super-point representation from its k local neighbors using relative-position Fourier embeddings, and one round of GCN message passing refines the result. The consensus loss Lcon=Lsmt+μLreg pushes neighboring aggregated representations to agree while keeping them close to the original global representations.","core_discovery":"PerLA's central claim is that a 3D language assistant can be made more perceptive by encoding local parts of the scene at higher resolution and merging them with a global context representation, using a Hilbert-curve-based pipeline. The pipeline partitions the point cloud into L equal-cardinality spatially compact parts, encodes each part and the whole cloud with the same pre-trained 3D encoder, then for each global super-point finds k local super-point neighbors by an approximate Hilbert-serialized k-NN search constrained by geometric labels. Localized cross-attention with relative position embeddings updates each global representation, a GCN message-passing step refines it, and a consensus loss (smoothness plus regularization) stabilizes training. The authors report that this yields gains over LL3DA of +1.34 CiDEr on ScanQA, +4.22 on ScanRefer, and +3.88 on Nr3D, while keeping the number of visual tokens fed to the LLM unchanged.","pith_inferences":["The locality-preserving Hilbert partition could be applied to other point-cloud backbones or to streaming and memory-limited settings where full-scene processing is infeasible, since the partitions are processed in parallel.","If the geometric-label constraint is the load-bearing part of neighbor selection, PerLA's gains may transfer to any setting with reliable superpoint or instance labels; conversely, in cluttered scenes where labels cross object boundaries the aggregation would be expected to degrade, a failure case the paper does not analyze.","A testable extension is measuring whether PerLA's detail perception scales with the number of partitions beyond eight, or combining it with token merging to push resolution higher without growing the token count.","The same local-global consensus idea could be adapted to 2D multi-crop vision-language models, where a global view plus high-resolution crops currently increases token counts."],"forward_implications":["On ScanQA, ScanRefer and Nr3D, PerLA outperforms prior 3D language assistants across nearly all reported metrics, with gains of +1.34 CiDEr on ScanQA, +4.22 on ScanRefer@0.5, and +3.88 on Nr3D@0.5.","The gains come from local-global aggregation at point level, not from adding visual tokens: an extended LL3DA with doubled tokens stays below PerLA on all three benchmarks.","Ablations show localized cross-attention beats mean and max pooling, GCN refinement consistently helps, and the consensus loss improves all metrics, so each component contributes to the reported result.","The same generalist pre-trained model fine-tunes successfully to both question answering and dense captioning, indicating transferability across 3D-language tasks.","Because the token count stays fixed, the approach adds perception without increasing LLM computational cost, addressing a key practical bottleneck in 3D language assistants."],"supporting_citations":[{"why":"Supplies the Hilbert space-filling curve used to serialize and partition the point cloud into spatially compact parts.","marker":"[50]"},{"why":"Supplies the geometric labels (superpoint segmentation) whose centers are prepended to serialized indices to constrain the approximate k-NN to same-instance neighbors.","marker":"[27]"},{"why":"Is the main baseline (LL3DA) whose encoder architecture and interaction design PerLA adopts, and the method PerLA is compared against on all benchmarks.","marker":"[9]"},{"why":"Provides the pre-training data and the prior 3D-LLM baseline used for generalist training and comparison.","marker":"[21]"},{"why":"Provides the ScanQA benchmark used to evaluate 3D question answering.","marker":"[2]"},{"why":"Provides the ScanRefer benchmark used to evaluate 3D dense captioning.","marker":"[6]"},{"why":"Provides the Nr3D benchmark used to evaluate 3D dense captioning with fine-grained referential descriptions.","marker":"[1]"},{"why":"Provides the set-abstraction layer used by the 3D encoder to downsample points into super-points.","marker":"[45]"},{"why":"Provides the Q-Former architecture used as the multimodal adapter that converts scene representations into tokens for the LLM.","marker":"[31]"}],"fun_headline_variants":["PerLA splits scenes to keep 3D details for language AI","Hilbert-curve encoding boosts 3D language assistants","Local details plus global context: PerLA wins benchmarks","PerLA: 3D language AI that sees both parts and whole","Point cloud split sharpens language understanding by 4+"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approximate k-NN search, which assumes that Hilbert-order proximity plus geometric labels from superpoint segmentation always yields neighbors on the same object, is the load-bearing premise; if that link breaks, the localized cross-attention aggregates unrelated points and the reported gains depend on the aggregation being correct.","fun_headline_variants_meta":{"raw":{"variants":["PerLA splits scenes to keep 3D details for language AI","Hilbert-curve encoding boosts 3D language assistants","Local details plus global context: PerLA wins benchmarks","PerLA: 3D language AI that sees both parts and whole","Point cloud split sharpens language understanding by 4+"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1225,"prompt_tokens":966,"completion_tokens":259,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":173}},"tokens_in":582,"tokens_out":259,"duration_ms":3415,"temperature":1.0,"reasoning_tokens":173,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:49:25.655192+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, on ScanNet validation scenes, the instance-consistency precision of PerLA's approximate k-NN: for each global super-point, compute the fraction of its k local neighbors that share its ground-truth object instance label. If the fraction is high and correlates with downstream gains, the Hilbert and geometric neighbor assumption carries the result; if it is low or uncorrelated, the stated mechanism is not what explains the scores.","supporting_citations":[{"cited_title":"Scanqa: 3d question answering for spatial scene understanding","cited_arxiv_id":null,"evidence_quote":"Provides the ScanQA benchmark used to evaluate 3D question answering."},{"cited_title":"Scanrefer: 3d object localization in rgb-d scans using natural language","cited_arxiv_id":null,"evidence_quote":"Provides the ScanRefer benchmark used to evaluate 3D dense captioning."},{"cited_title":"Hilbert’s space-filling curve","cited_arxiv_id":null,"evidence_quote":"Supplies the Hilbert space-filling curve used to serialize and partition the point cloud into spatially compact parts."},{"cited_title":"Large-scale point cloud semantic segmentation with superpoint graphs","cited_arxiv_id":null,"evidence_quote":"Supplies the geometric labels (superpoint segmentation) whose centers are prepended to serialized indices to constrain the approximate k-NN to same-instance neighbors."},{"cited_title":"Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning","cited_arxiv_id":null,"evidence_quote":"Is the main baseline (LL3DA) whose encoder architecture and interaction design PerLA adopts, and the method PerLA is compared against on all benchmarks."},{"cited_title":"3d-llm: Injecting the 3d world into large language models","cited_arxiv_id":null,"evidence_quote":"Provides the pre-training data and the prior 3D-LLM baseline used for generalist training and comparison."},{"cited_title":"Referit3d: Neural listeners for fine-grained 3d object identifica- tion in real-world scenes","cited_arxiv_id":null,"evidence_quote":"Provides the Nr3D benchmark used to evaluate 3D dense captioning with fine-grained referential descriptions."},{"cited_title":"Pointnet++: Deep hierarchical feature learn- ing on point sets in a metric space.NeurIPS, 30, 2017","cited_arxiv_id":null,"evidence_quote":"Provides the set-abstraction layer used by the 3D encoder to downsample points into super-points."},{"cited_title":"Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models","cited_arxiv_id":null,"evidence_quote":"Provides the Q-Former architecture used as the multimodal adapter that converts scene representations into tokens for the LLM."}],"review_version":1}