Pith. sign in

REVIEW 3 major objections 6 minor 70 references

PerLA: Perceptive 3D Language Assistant

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Splitting a 3D scene into local parts before encoding lets a language assistant preserve fine details and beat state-of-the-art baselines on question answering and dense captioning.

desk verdict PerLA is a solid, incremental 3D language assistant with consistent benchmark gains; its novelty claim is overstated and the same-instance neighbor guarantee is unsupported, but the core method is sound. read the letter →

arxiv 2411.19774 v2 pith:GPGMQBAA submitted 2024-11-29 cs.CV cs.CLcs.LG

Guofeng Mei , Wei Lin , Luigi Riz , Yujiao Wu , Fabio Poiesi , Yiming Wang This is my paper · ORCID
classification cs.CVcs.CLcs.LG
keywords 3DlanguageassistantpointcloudperceptionHilbertcurvelocal-globalaggregationcross-attentiongraphneuralnetworkquestionansweringdensecaptioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PerLA argues that 3D language assistants lose fine-grained scene information when they downsample or divide point clouds, and that the fix is to process high-resolution local parts alongside a low-resolution global view without sending extra tokens to the LLM. The paper claims that its perceptive scene encoder, built on Hilbert-curve serialization, approximate k-NN, localized cross-attention, and a GCN refinement step, captures details such as object color and small-object identity that global-only models miss. On ScanQA, ScanRefer, and Nr3D it reports state-of-the-art results, including gains of up to +1.34 CiDEr on ScanQA and +4.22 on ScanRefer over LL3DA. If correct, the work shows that local-global aggregation at the point level is a practical alternative to simply increasing token counts.

What carries the argument

The Hilbert curve is the ordering that lets unordered point clouds be split into spatially coherent parts: the paper serializes the cloud with Hilbert indices, cuts it into L equal-size partitions, and uses the same serialized index (with geometric-label bits prepended) to run an O(1)-per-query approximate k-NN that finds local super-point neighbors. Localized cross-attention then updates each global super-point representation from its k local neighbors using relative-position Fourier embeddings, and one round of GCN message passing refines the result. The consensus loss Lcon=Lsmt+μLreg pushes neighboring aggregated representations to agree while keeping them close to the original global representations.

What would settle it

Measure, on ScanNet validation scenes, the instance-consistency precision of PerLA's approximate k-NN: for each global super-point, compute the fraction of its k local neighbors that share its ground-truth object instance label. If the fraction is high and correlates with downstream gains, the Hilbert and geometric neighbor assumption carries the result; if it is low or uncorrelated, the stated mechanism is not what explains the scores.

Watch

Extended reading notes

Core claim

PerLA's central claim is that a 3D language assistant can be made more perceptive by encoding local parts of the scene at higher resolution and merging them with a global context representation, using a Hilbert-curve-based pipeline. The pipeline partitions the point cloud into L equal-cardinality spatially compact parts, encodes each part and the whole cloud with the same pre-trained 3D encoder, then for each global super-point finds k local super-point neighbors by an approximate Hilbert-serialized k-NN search constrained by geometric labels. Localized cross-attention with relative position embeddings updates each global representation, a GCN message-passing step refines it, and a consensus loss (smoothness plus regularization) stabilizes training. The authors report that this yields gains over LL3DA of +1.34 CiDEr on ScanQA, +4.22 on ScanRefer, and +3.88 on Nr3D, while keeping the number of visual tokens fed to the LLM unchanged.

Load-bearing premise

The approximate k-NN search, which assumes that Hilbert-order proximity plus geometric labels from superpoint segmentation always yields neighbors on the same object, is the load-bearing premise; if that link breaks, the localized cross-attention aggregates unrelated points and the reported gains depend on the aggregation being correct.

Editorial extensions

If this is right

  • On ScanQA, ScanRefer and Nr3D, PerLA outperforms prior 3D language assistants across nearly all reported metrics, with gains of +1.34 CiDEr on ScanQA, +4.22 on ScanRefer@0.5, and +3.88 on Nr3D@0.5.
  • The gains come from local-global aggregation at point level, not from adding visual tokens: an extended LL3DA with doubled tokens stays below PerLA on all three benchmarks.
  • Ablations show localized cross-attention beats mean and max pooling, GCN refinement consistently helps, and the consensus loss improves all metrics, so each component contributes to the reported result.
  • The same generalist pre-trained model fine-tunes successfully to both question answering and dense captioning, indicating transferability across 3D-language tasks.
  • Because the token count stays fixed, the approach adds perception without increasing LLM computational cost, addressing a key practical bottleneck in 3D language assistants.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The locality-preserving Hilbert partition could be applied to other point-cloud backbones or to streaming and memory-limited settings where full-scene processing is infeasible, since the partitions are processed in parallel.
  • If the geometric-label constraint is the load-bearing part of neighbor selection, PerLA's gains may transfer to any setting with reliable superpoint or instance labels; conversely, in cluttered scenes where labels cross object boundaries the aggregation would be expected to degrade, a failure case the paper does not analyze.
  • A testable extension is measuring whether PerLA's detail perception scales with the number of partitions beyond eight, or combining it with token merging to push resolution higher without growing the token count.
  • The same local-global consensus idea could be adapted to 2D multi-crop vision-language models, where a global view plus high-resolution crops currently increases token counts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces PerLA, a 3D language assistant for question answering and dense captioning. The method preserves local point-cloud detail by partitioning the scene with a Hilbert-curve serialization, encoding each partition with the same 3D encoder as the global cloud, and then aggregating local and global superpoint representations through a localized cross-attention module and a GCN message-passing layer. A consensus loss is added to the standard next-token prediction objective to regularize local representations. Experiments on ScanQA, ScanRefer, and Nr3D report consistent improvements over LL3DA and other baselines, with ablations showing that the proposed components each contribute positively. The central claim is that PerLA achieves state-of-the-art performance while keeping the number of visual tokens fed to the LLM fixed at 32.

Significance. If the reported gains are robust, the paper makes a useful contribution: it demonstrates that local detail can be injected into a scene-level 3D representation without increasing the LLM token budget, and the ablation study (Tabs. 3-5) supports the positive contribution of each proposed component. The method is also computationally lightweight compared to multi-view approaches such as 3D-LLM. However, the significance is moderated by two concerns that bear directly on the validity of the central claim: (i) the claimed 'same-instance' guarantee for the Hilbert-based k-NN neighborhoods is not substantiated and appears inconsistent with the nature of the superpoint labels used; and (ii) the headline improvement over the official LL3DA baseline is smaller than the discrepancy between that official baseline and the authors' own reproduction. These issues require additional analysis before the SOTA claim can be accepted.

major comments (3)
  1. [Sec. 3.1, Eq. (3)] The paper states that the geometric labels from [27] 'guarantee that p^g_i and its nearest local super-points all originate from the same instance.' This claim is not supported. Reference [27] produces over-segmented geometric superpoints, not instance labels, and the manuscript does not specify how these labels are assigned to the FPS-downsampled global and local superpoints, nor how 'same instance' is defined with respect to ScanNet instance annotations. Because the localized cross-attention in Eq. (3) is the only mechanism that injects local detail into the global representation, and because Tabs. 3-4 attribute the gains to this aggregation, the correctness of the proposed mechanism depends on the spatial purity of the selected neighborhoods. Please provide quantitative neighbor-purity statistics (e.g., the fraction of the k=24 neighborhoods whose members share a ScanNet instance label) on the ScanNet validation set, and a qualitative analysis of failure cases where the Hilbert-order or label boundaries cross object boundaries. If purity is low, the performance gains might instead be explained by generic spatial smoothing, which would require a revised interpretation of the method's contribution.
  2. [Tabs. 1-2, Sec. 4.1] The central SOTA claim is weakened by the discrepancy between the official LL3DA numbers and the authors' reproduction. In Tab. 1, official LL3DA scores 76.79 CiDEr on ScanQA validation, the authors' reproduction scores 74.37, and PerLA scores 78.13. The gain over the official baseline is +1.34, which is smaller than the 2.42-point gap between the official and reproduced LL3DA. This suggests that a non-negligible part of the observed improvement may come from differences in training data, schedule, or other setup choices rather than from the proposed perceptive encoder. Please report results of official LL3DA under the exact same data and training configuration as PerLA (or provide a detailed explanation of why the reproduction differs), and compare against that baseline. In addition, all reported numbers are from a single run; given that the headline margins are on the order of 1-4 CiDEr, please provide means and standard deviations over at least three seeds, or otherwise justify the stability of the results.
  3. [Sec. 4.2, Tabs. 5-6] The hyperparameters L (number of partitions), k (neighbors per global point), λ, μ, beam size, and the Hilbert grid resolution d are tuned on the validation set, and the final configuration is selected on the basis of those validation numbers. The differences between some configurations are small (e.g., Tab. 6 shows 6 vs. 8 partitions differing by 0.1 CiDEr on ScanQA), so the claimed optimality of the chosen settings is fragile. Please include a sensitivity analysis or error bars for the key hyperparameters (at least L and k), and clarify whether the same validation set was used for both model selection and the final reported numbers, in which case the reported performance may be optimistic.
minor comments (6)
  1. [Fig. E caption (Supp. Mat.)] The caption contains a typo: 'a more percepti solution' should be 'a more perceptive solution'.
  2. [Supp. Tab. G] The 'Hard Drive (GB)' entry for 3D-LLM is '74563.49', which appears to be a formatting error or an implausibly large value; please check the unit and the number.
  3. [Sec. 3.1, Eq. (5)] The notation d_k is initially defined as a diagonal matrix representing row-wise sums of W^g, but in Eq. (5) it is used as a scalar denominator. Please clarify whether d_k is the degree of node k or a diagonal element of the degree matrix.
  4. [Sec. 3.1] The statement that the Hilbert-based k-NN search has 'O(1) complexity' is potentially misleading: the one-time serialization and sorting is O(N log N), and only the per-query lookup after that preprocessing is O(1). Please clarify that the O(1) complexity applies to query time after preprocessing.
  5. [Tab. 3] The rows labeled 'Global' and 'Local' are not fully defined in the main text; a one-sentence description of what these variants do (e.g., whether they use only the global branch or only the local branch for the final representation) would improve readability.
  6. [Sec. 4.1, Tab. 1] For LL3DA (repr.), results are reported only on the validation split for ScanQA; please state why test-set numbers are omitted, or provide them if available.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claim is external benchmark performance, evaluated against fixed metrics and ablations, with no fitted constant relabeled as a prediction.

full rationale

The paper's central claims are empirical: PerLA outperforms prior 3D language assistants on ScanQA, ScanRefer, and Nr3D. These results are evaluated against externally published methods using fixed metrics (CiDEr, BLEU-4, METEOR, Rouge-L) on standard benchmark splits, so the reported numbers are not derived from the method's own assumptions by construction. Hyperparameters such as the number of partitions L=6, the neighbor count k=24, and the loss weights lambda=mu=0.1 are conventional validation-set choices; the paper does not take any fitted value and relabel it as a predicted quantity. The main components--Hilbert-based partitioning, localized cross-attention, and GCN message passing--are independently ablated (global-only, local-only, without GCN, mean/max pooling alternatives), so their contributions are not assumed into the final result. The only load-bearing external input is the geometric partitioning of [27] used to constrain the approximate k-NN search; the paper's claim that these labels guarantee same-instance neighborhoods is a correctness or robustness concern, not a circular one, because it concerns whether an external preprocessing tool has a property, rather than defining one quantity in terms of another. There are no load-bearing self-citations by the authors, and no equation-level reduction of a prediction to an input. Therefore, no significant circularity is present.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on domain assumptions about the pre-trained 3D encoder, Hilbert-based neighbor approximation, geometric label quality, and the usefulness of graph smoothing. The paper reports several hand-chosen hyperparameters, including L=6, k=24, lambda=0.1, mu=0.1, beam size 4, and an unreported Hilbert resolution d. No new physical or conceptual entities are introduced.

free parameters (6)
  • L (number of local partitions) = 6
    Ablated over 4, 6, 8 in Table 6; 6 chosen on ScanQA validation. Affects the effective local resolution.
  • k (number of local neighbors per global point) = 24 (4L)
    Set in Implementation details as k=4L; no separate ablation is reported.
  • lambda (consensus loss weight) = 0.1
    Chosen in Implementation details; no sensitivity analysis is reported.
  • mu (regularization weight in consensus loss) = 0.1
    Chosen in Implementation details; no sensitivity analysis is reported.
  • beam size = 4
    Used for inference beam search in Eq. 7.
  • Hilbert curve grid resolution d = not reported
    Algorithm 1 in the supplementary requires a resolution d, but the value used in experiments is never given, making partition boundaries implementation-dependent.
assumptions (5)
  • domain assumption The pre-trained 3D scene encoder from LL3DA produces usable point-level features when applied to both global and local partitions.
    Section 3.1 and Supp. B.1 adopt the LL3DA encoder architecture unchanged; no analysis of whether its features remain meaningful on smaller local patches that differ from the training distribution.
  • ad hoc to paper Hilbert-curve serialization preserves enough spatial locality for an approximate k-NN search that reliably finds same-object neighbors.
    Section 3.1 'Hilbert-based nearest-neighbor search'; the O(1) neighbor lookup replaces true geometric k-NN, and its errors are not measured.
  • ad hoc to paper Superpoint geometric labels from [27] coincide with object and instance boundaries, so the paper's statement that the labels 'guarantee' same-instance neighbors holds.
    Section 3.1; geometric superpoints are oversegmentations, not instance masks, so the guarantee is overstated and unverified on the used data.
  • domain assumption Graph-convolution message passing and the smoothness regularizer improve, rather than distort, the fused representations.
    Sections 3.1 and 3.2; the gains are empirical and no formal guarantee on feature quality is provided.
  • domain assumption The 3D-LLM pretraining data plus ScanQA, ScanRefer, and Nr3D fine-tuning sets are sufficient for the reported generalization.
    Section 4 and Supp. D; no evaluation on additional, unseen 3D datasets is included.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PerLA: Perceptive 3D Language Assistant." pith.science (2026). https://pith.science/paper/GPGMQBAA

@misc{pith2026241119774,
  author       = {Pith},
  title        = {Pith review of: PerLA: Perceptive 3D Language Assistant},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GPGMQBAA}},
  note         = {Machine review of arXiv:2411.19774}
}
read the original abstract

Enabling Large Language Models (LLMs) to understand the 3D physical world is an emerging yet challenging research direction. Current strategies for processing point clouds typically downsample the scene or divide it into smaller parts for separate analysis. However, both approaches risk losing key local details or global contextual information. In this paper, we introduce PerLA, a 3D language assistant designed to be more perceptive to both details and context, making visual representations more informative for the LLM. PerLA captures high-resolution (local) details in parallel from different point cloud areas and integrates them with (global) context obtained from a lower-resolution whole point cloud. We present a novel algorithm that preserves point cloud locality through the Hilbert curve and effectively aggregates local-to-global information via cross-attention and a graph neural network. Lastly, we introduce a novel loss for local representation consensus to promote training stability. PerLA outperforms state-of-the-art 3D language assistants, with gains of up to +1.34 CiDEr on ScanQA for question answering, and +4.22 on ScanRefer and +3.88 on Nr3D for dense captioning. https://gfmei.github.io/PerLA/

Figures

Figures reproduced from arXiv: 2411.19774 by the authors.

Figure 1
Figure 1. PerLA is a 3D language assistant that integrates local details with global context to learn informative representations of 3D [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of PerLA. (Left): The overall pipeline of PerLA, which begins by extracting interaction-aware 3D scene represen [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The qualitative comparison between our method, PerLA, and LL3DA [ [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative comparisons on the dense captioning task across the Nr3D [ [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 51 canonical work pages

  1. [27]

    Large-scale point cloud semantic segmentation with superpoint graphs

    Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. In CVPR, pages 4558–4567, 2018. 4, 14

    work page 2018

  2. [1]

    Referit3d: Neural listeners for fine-grained 3d object identifica- tion in real-world scenes

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mo- hamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identifica- tion in real-world scenes. In ECCV, pages 422–440. Springer, 2020. 2, 5, 6, 7, 8, 14, 15, 16, 17, 18

    work page 2020

  3. [2]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In CVPR, 2022. 2, 5, 6, 14, 15, 16, 17, 18

    work page 2022

  4. [3]

    Meteor: An auto- matic metric for mt evaluation with improved correla- tion with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An auto- matic metric for mt evaluation with improved correla- tion with human judgments. In ACL Workshop, pages 65–72, 2005. 5, 16

    work page 2005

  5. [4]

    3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds

    Daigang Cai, Lichen Zhao, Jing Zhang, Lu Sheng, and Dong Xu. 3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds. In CVPR, pages 16464–16473, 2022. 2, 6, 7

    work page 2022

  6. [5]

    Crossvit: Cross-attention multi-scale vision transformer for image classification

    Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In ICCV, pages 357–366, 2021. 2

    work page 2021

  7. [6]

    Scanrefer: 3d object localization in rgb-d scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In ECCV, 2020. 2, 5, 6, 7, 8, 14, 15, 16, 17, 18

    work page 2020

  8. [7]

    D 3 net: A unified speaker-listener ar- chitecture for 3d dense captioning and visual ground- ing

    Dave Zhenyu Chen, Qirui Wu, Matthias Nießner, and Angel X Chang. D 3 net: A unified speaker-listener ar- chitecture for 3d dense captioning and visual ground- ing. In ECCV, pages 487–505. Springer, 2022. 7

    work page 2022

Show all 70 references
  1. [8]

    End-to-end 3d dense cap- tioning with vote2cap-detr

    Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Gang Yu, and Tao Chen. End-to-end 3d dense cap- tioning with vote2cap-detr. In CVPR, pages 11124– 11133, 2023. 1, 2, 3, 7, 12

    2023

  2. [9]

    Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In CVPR, 2024. 1, 2, 3, 5, 6, 7, 12, 14, 16, 17, 18

    2024

  3. [10]

    V ote2cap-detr++: Decoupling localization and describing for end-to-end 3d dense captioning

    Sijin Chen, Hongyuan Zhu, Mingsheng Li, Xin Chen, Peng Guo, Yinjie Lei, YU Gang, Taihao Li, and Tao Chen. V ote2cap-detr++: Decoupling localization and describing for end-to-end 3d dense captioning. IEEE TPAMI, 2024. 2

    2024

  4. [11]

    Grounded 3d-llm with referent tokens

    Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Ruiyuan Lyu, Runsen Xu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens. arXiv preprint arXiv:2405.10370, 2024. 1

    2024 arXiv

  5. [12]

    Scan2cap: Context-aware dense cap- tioning in rgb-d scans

    Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2cap: Context-aware dense cap- tioning in rgb-d scans. In CVPR, pages 3193–3203,

  6. [13]

    Unit3d: A unified trans- former for 3d dense captioning and visual grounding

    Zhenyu Chen, Ronghang Hu, Xinlei Chen, Matthias Nießner, and Angel X Chang. Unit3d: A unified trans- former for 3d dense captioning and visual grounding. In ICCV, pages 18109–18119, 2023. 2, 6, 7

    2023

  7. [14]

    Scannet: Richly-annotated 3d reconstructions of in- door scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of in- door scenes. In CVPR, 2017. 5, 14, 15

    2017

  8. [15]

    Objaverse-xl: A universe of 10m+ 3d objects

    Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram V oleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. NeurIPS, 36, 2024. 2

    2024

  9. [16]

    Multi-clip: Contrastive vision-language pre-training for ques- tion answering tasks in 3d scenes

    Alexandros Delitzas, Maria Parelli, Nikolas Hars, Georgios Vlassis, Sotirios Anagnostidis, Gregor Bachmann, and Thomas Hofmann. Multi-clip: Contrastive vision-language pre-training for ques- tion answering tasks in 3d scenes. arXiv preprint arXiv:2306.02329, 2023. 6

    2023 arXiv

  10. [17]

    Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd. arXiv preprint arXiv:2404.06512 ,

    arXiv

  11. [18]

    Multiscale vision transformers

    Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Fe- ichtenhofer. Multiscale vision transformers. In ICCV, pages 6824–6835, 2021. 2

    2021

  12. [19]

    Scene-llm: Extending language model for 3d visual understanding and reasoning

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wen- han Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401, 2024. 3

    2024 arXiv

  13. [20]

    xt: Nested tokenization for larger context in large images

    Ritwik Gupta, Shufan Li, Tyler Zhu, Jitendra Malik, Trevor Darrell, and Karttikeya Mangalam. xt: Nested tokenization for larger context in large images. arXiv preprint arXiv:2403.01915, 2024. 2

    2024 arXiv

  14. [21]

    3d-llm: Injecting the 3d world into large language models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. NeurIPS, 2023. 1, 2, 3, 5, 6, 14, 16, 17

    2023

  15. [22]

    Chat-scene: Bridging 3d scene and large language models with ob- ject identifiers

    Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with ob- ject identifiers. NeurIPS, 2024. 2

    2024

  16. [23]

    An embodied generalist agent in 3d world

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In ICML,

  17. [24]

    Segment3d: Learning fine- grained class-agnostic 3d segmentation without man- ual labels

    Rui Huang, Songyou Peng, Ayca Takmaz, Federico Tombari, Marc Pollefeys, Shiji Song, Gao Huang, and Francis Engelmann. Segment3d: Learning fine- grained class-agnostic 3d segmentation without man- ual labels. In ECCV, pages 278–295. Springer, 2025. 3

    2025

  18. [25]

    More: Multi-order re- lation mining for dense captioning in 3d scenes

    Yang Jiao, Shaoxiang Chen, Zequn Jie, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. More: Multi-order re- lation mining for dense captioning in 3d scenes. In ECCV, pages 528–545. Springer, 2022. 6, 7

    2022

  19. [26]

    Context-aware alignment and mutual masking for 3d-language pre-training

    Zhao Jin, Munawar Hayat, Yuwei Yang, Yulan Guo, and Yinjie Lei. Context-aware alignment and mutual masking for 3d-language pre-training. InCVPR, pages 10984–10994, 2023. 2, 6, 7

    2023

  20. [28]

    What matters when building vision- language models? Advances in Neural Information Processing Systems, 37:87874–87907, 2024

    Hugo Laurenc ¸on, L´eo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision- language models? Advances in Neural Information Processing Systems, 37:87874–87907, 2024. 15

    2024

  21. [29]

    Mpvit: Multi-path vision transformer for dense prediction

    Youngwan Lee, Jonghee Kim, Jeffrey Willette, and Sung Ju Hwang. Mpvit: Multi-path vision transformer for dense prediction. In CVPR, pages 7287–7296,

  22. [30]

    Llava-onevision: Easy vi- sual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Zi- wei Liu, and Chunyuan Li. Llava-onevision: Easy vi- sual task transfer. arXiv preprint arXiv:2408.03326 ,

    arXiv

  23. [31]

    Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. In ICML, pages 19730–19742. PMLR,

  24. [32]

    Mini-gemini: Mining the poten- tial of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the poten- tial of multi-modality vision language models. arXiv preprint arXiv:2403.18814, 2024. 2

    2024 arXiv

  25. [33]

    Pointmamba: A simple state space model for point cloud analysis

    Dingkang Liang, Xin Zhou, Wei Xu, Xingkui Zhu, Zhikang Zou, Xiaoqing Ye, Xiao Tan, and Xiang Bai. Pointmamba: A simple state space model for point cloud analysis. In NeurIPS, 2024. 3

    2024

  26. [34]

    Rouge: A package for automatic eval- uation of summaries

    Chin-Yew Lin. Rouge: A package for automatic eval- uation of summaries. In Text summarization branches out, pages 74–81, 2004. 5, 16

    2004

  27. [35]

    Llava- next: Improved reasoning, ocr, and world knowledge,

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuan- han Zhang, Sheng Shen, and Yong Jae Lee. Llava- next: Improved reasoning, ocr, and world knowledge,

  28. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS,

  29. [37]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, pages 11976–11986,

  30. [38]

    Fixing weight decay regularization in adam

    Ilya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 5, 2017. 5

    2017 arXiv

  31. [39]

    Sqa3d: Situated question answering in 3d scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yi- tao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022. 2

    2022 arXiv

  32. [40]

    Complete 3d relationships extraction modality alignment network for 3d dense captioning

    Aihua Mao, Zhi Yang, Wanxin Chen, Ran Yi, and Yong-jin Liu. Complete 3d relationships extraction modality alignment network for 3d dense captioning. IEEE TVCG, 2023. 7

    2023

  33. [41]

    Fast march- ing farthest point sampling

    Carsten Moenning and Neil A Dodgson. Fast march- ing farthest point sampling. Technical report, Univer- sity of Cambridge, ECCV Laboratory, 2003. 3

    2003

  34. [42]

    Bongki Moon, Hosagrahar V Jagadish, Christos Faloutsos, and Joel H. Saltz. Analysis of the cluster- ing properties of the hilbert space-filling curve. IEEE TKDE, 13(1):124–141, 2001. 13

    2001

  35. [43]

    Bleu: a method for automatic evalu- ation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evalu- ation of machine translation. In ACL, pages 311–318,

  36. [44]

    Clip-guided vision-language pre-training for question answering in 3d scenes

    Maria Parelli, Alexandros Delitzas, Nikolas Hars, Georgios Vlassis, Sotirios Anagnostidis, Gregor Bachmann, and Thomas Hofmann. Clip-guided vision-language pre-training for question answering in 3d scenes. In CVPR, pages 5607–5612, 2023. 2, 6

    2023

  37. [45]

    Pointnet++: Deep hierarchical feature learn- ing on point sets in a metric space.NeurIPS, 30, 2017

    Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learn- ing on point sets in a metric space.NeurIPS, 30, 2017. 12

    2017

  38. [46]

    Shapellm: Universal 3d object understanding for embodied interaction

    Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. Shapellm: Universal 3d object understanding for embodied interaction. In ECCV, pages 214–238,

  39. [47]

    Semantic segmentation for real point cloud scenes via bilateral augmentation and adaptive fusion

    Shi Qiu, Saeed Anwar, and Nick Barnes. Semantic segmentation for real point cloud scenes via bilateral augmentation and adaptive fusion. In CVPR, pages 1757–1767, 2021. 1

    2021

  40. [48]

    Radford, J

    A. Radford, J. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 2

    2021

  41. [49]

    Hiera: A hierarchical vision trans- former without the bells-and-whistles

    Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggar- wal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, et al. Hiera: A hierarchical vision trans- former without the bells-and-whistles. InICML, pages 29441–29454. PMLR, 2023. 2

    2023

  42. [50]

    Hilbert’s space-filling curve

    Hans Sagan. Hilbert’s space-filling curve. In Space- filling curves, pages 9–30. Springer, 1994. 2, 3, 4

    1994

  43. [51]

    When do we not need larger vision models? In ECCV, pages 444–462

    Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do we not need larger vision models? In ECCV, pages 444–462. Springer, 2025. 2

    2025

  44. [52]

    Fourier features let networks learn high frequency functions in low dimensional domains

    Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Sing- hal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS, 33: 7537–7547, 2020. 4

    2020

  45. [53]

    Minigpt-3d: Efficiently aligning 3d point clouds with large language models using 2d priors

    Yuan Tang, Xu Han, Xianzhi Li, Qiao Yu, Yixue Hao, Long Hu, and Min Chen. Minigpt-3d: Efficiently aligning 3d point clouds with large language models using 2d priors. ACM’MM, 2024. 2

    2024

  46. [54]

    Llama: Open and effi- cient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Ham- bro, Faisal Azhar, et al. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1, 17

    2023 arXiv

  47. [55]

    Cider: Consensus-based image descrip- tion evaluation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image descrip- tion evaluation. In CVPR, pages 4566–4575, 2015. 5, 16

    2015

  48. [56]

    Spatiality-guided transformer for 3d dense captioning on point clouds

    Heng Wang, Chaoyi Zhang, Jianhui Yu, and Wei- dong Cai. Spatiality-guided transformer for 3d dense captioning on point clouds. arXiv preprint arXiv:2204.10688, 2022. 2, 7

    2022 arXiv

  49. [57]

    Octformer: Octree-based trans- formers for 3d point clouds

    Peng-Shuai Wang. Octformer: Octree-based trans- formers for 3d point clouds. ACM TOG, 42(4):1–11,

  50. [58]

    Divergence-augmented policy opti- mization

    Qing Wang et al. Divergence-augmented policy opti- mization. NeurIPS, 32, 2019. 15

    2019

  51. [59]

    Chat-3d: Data-efficiently tun- ing large language model for universal dialogue of 3d scenes

    Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data-efficiently tun- ing large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308.08769, 2023. 2

    2023 arXiv

  52. [60]

    Point transformer v3: Simpler faster stronger

    Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler faster stronger. In CVPR, pages 4840–4851, 2024. 1, 3, 4

    2024

  53. [61]

    Pointllm: Empower- ing large language models to understand point clouds

    Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empower- ing large language models to understand point clouds. arXiv preprint arXiv:2308.16911, 2023. 2

    2023 arXiv

  54. [62]

    Llava-uhd: an lmm perceiving any aspect ratio and high-resolution im- ages

    Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zan- lin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution im- ages. arXiv preprint arXiv:2403.11703, 2024. 3

    2024 arXiv

  55. [63]

    Focal attention for long-range interactions in vision transformers

    Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal attention for long-range interactions in vision transformers. NeurIPS, 34:30008–30022, 2021. 2

    2021

  56. [64]

    Fouhey, and Joyce Chai

    Jianing Yang, Xuweiyi Chen, Nikhil Madaan, Mad- havan Iyengar, Shengyi Qian, David F. Fouhey, and Joyce Chai. 3d-grand: A million-scale dataset for 3d-llms with better grounding and less hallucination. arXiv preprint 2406.05132, 2024. 2

    2024 arXiv

  57. [65]

    3d question answering

    Shuquan Ye, Dongdong Chen, Songfang Han, and Jing Liao. 3d question answering. arXiv preprint arXiv:2112.08359, 2021. 2

    2021 arXiv

  58. [66]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. 1, 5, 17

    2022 arXiv

  59. [67]

    Toward explainable 3d grounded visual question answering: A new benchmark and strong baseline

    Lichen Zhao, Daigang Cai, Jing Zhang, Lu Sheng, Dong Xu, Rui Zheng, Yinjie Zhao, Lipeng Wang, and Xibo Fan. Toward explainable 3d grounded visual question answering: A new benchmark and strong baseline. IEEE TCSVT, 33(6):2935–2949, 2022. 2

    2022

  60. [68]

    Contextual modeling for 3d dense captioning on point clouds

    Yufeng Zhong, Long Xu, Jiebo Luo, and Lin Ma. Contextual modeling for 3d dense captioning on point clouds. arXiv preprint arXiv:2210.03925, 2022. 7

    2022 arXiv

  61. [69]

    Llava-3d: A simple yet effec- tive pathway to empowering lmms with 3d-awareness

    Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effec- tive pathway to empowering lmms with 3d-awareness. arXiv preprint arXiv:2409.18125, 2024. 2

    2024 arXiv

  62. [70]

    de- scribe

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. InICCV, pages 2911–2921, 2023. 6, 7 PerLA : Perceptive 3D language assistant Supplementary Material A. Introduction In this supplemen...

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.