Pith. sign in

REVIEW 4 major objections 4 minor 63 references

3D Scene Graph Guided Vision-Language Pre-training

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Scene-graph-guided pre-training with multi-level contrastive learning and masked modality modeling produces transferable 3D-language features that improve 3D visual grounding, dense captioning, and question answering after fine-tuning.

desk verdict Competent scene graph-guided 3D VLP paper with solid ablations, but the same-dataset pre-training protocol undercuts the 'universal representation' claim; worth refereeing with a demand for a held-out pre-training test. read the letter →

arxiv 2411.18666 v1 pith:2V7X3W7U submitted 2024-11-27 cs.CV

Hao Liu , Yanni Ma , Yan Liu , Haihong Xiao , Ying He This is my paper · ORCID
classification cs.CV
keywords 3Dvision-languagepre-trainingscenegraphmulti-levelcontrastivelearningmaskedmodalitymodelingvisualgroundingdensecaptioningquestionansweringpointclouds
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision–language models for 3D scenes have so far been built one task at a time, each with its own modules and losses. This paper argues that a single pre-training scheme can serve 3D visual grounding, dense captioning, and question answering together, if the pre-training exploits the structural parallel between a scene graph and a sentence: just as a sentence decomposes into subject, predicate, and object, a scene graph decomposes into two object nodes and a relation edge. The proposed objectives—a three-level contrastive alignment (words to objects, sentences to referred objects, whole scenes to descriptions) and a masked-modality task that reconstructs masked words and predicts the semantic category of masked objects from position clues—are meant to pull 3D and language features into one shared space. After fine-tuning, the model reports the best or comparable numbers among the methods it is compared to on ScanRefer, Scan2Cap, and ScanQA, suggesting that task-specific designs may be unnecessary.

What carries the argument

The load-bearing object is the 3D scene graph: a graph in which nodes are object proposals from a 3D detector (VoteNet) and edges encode spatial relationships between neighboring proposals. A three-layer graph convolutional network (EdgeConv) updates node and edge features by message passing, so each node accumulates contextual information from its neighbors; this updated graph is the substrate for all three contrastive alignments. The second mechanism is masked modality modeling: masked language modeling reconstructs hidden words through cross-attention with object features, while masked object modeling feeds visible objects plus mask tokens with positional embeddings (the 27-dimensional box geometry) through cross-attention with text, and predicts the semantic category of each masked object rather than reconstructing its point cloud. Together these objectives replace task-specific losses; only detection and a language-to-object classification loss are added during pre-training.

What would settle it

Pre-train the same model and objectives on a separate 3D-text corpus that does not overlap with the downstream benchmarks (for example, the ScanScribe collection used by 3D-VisTA) and then fine-tune on ScanRefer; if the gains over training from scratch largely disappear, the claim of transferable universal representations fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that pre-training a 3D vision–language model with scene graph-guided multi-level contrastive learning (SG MCL) and masked modality modeling (MMM) produces transferable features that a light task head can adapt to 3D visual grounding, 3D dense captioning, and 3D question answering. SG MCL operates at three granularities: it aligns each object proposal with the word naming its category, aligns the updated node corresponding to the referential object with the whole sentence, and aligns an aggregated scene representation with the scene description. MMM masks a fraction of words and objects; a cross-attention module reconstructs the missing words and, instead of regenerating masked point clouds, predicts each masked object's semantic category from its positional embedding and the visible context. On the ScanRefer-based benchmarks the authors report 51.87% Acc@0.25 and 39.91% Acc@0.5 for visual grounding, 24.80% EM@1 and 59.24% EM@10 for question answering, and 55.32 C@0.5 for dense captioning with 2D+3D inputs, the best reported numbers among the compared baselines under the same detector settings.

Load-bearing premise

The argument that pre-training learns universal 3D-language representations rests on measuring gains from pre-training on the ScanRefer training split and then fine-tuning on the same ScanRefer-derived benchmarks, so the reported improvements could reflect extra in-distribution training rather than true transfer.

Editorial extensions

If this is right

  • A single pre-trained backbone plus a small task head can replace the specialized modules and auxiliary losses that current 3D visual grounding, dense captioning, and question answering models rely on.
  • Predicting semantic categories of masked objects from box positions is a viable alternative to point-cloud reconstruction in 3D-language pre-training, which simplifies the pre-training objective.
  • Scene-graph structure provides fine-grained contrastive supervision at word, object, and scene levels, going beyond global image-text alignment and improving downstream accuracy when fine-tuned on the same benchmarks.
  • Pre-training on one benchmark's training split can still produce measurable gains on that benchmark's validation sets, suggesting that structural alignment can compensate partly for limited 3D-text data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because pre-training and fine-tuning both draw on ScanRefer-derived data, the transferability claim would be strengthened by pre-training on an independent 3D-text corpus and then fine-tuning on ScanRefer; the current design cannot fully rule out in-distribution overfitting.
  • The failure cases on complex spatial relations suggest a natural extension: adding an edge-level (relation prediction) objective to the pre-training loss might resolve the relational errors the authors document.
  • The counting-question failures indicate that the contrastive alignments capture object identity and relations but not cardinality; a count-aware auxiliary task would be a direct test of whether the representation can be pushed further.
  • Because the scene graph is built from detector proposals, its quality is bounded by the detector; pre-training with a stronger proposal generator could amplify the observed gains.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a 3D vision-language pre-training framework that exploits the structural alignment between 3D scene graphs and natural language. Two pre-training objectives are introduced: scene graph-guided multi-level contrastive learning (SG MCL), which aligns features at word-object, sentence-referred-object, and scene levels, and masked modality modeling (MMM), which reconstructs masked words and object semantic categories. The model is pre-trained on the ScanRefer training split and then fine-tuned for 3D visual grounding (ScanRefer), dense captioning (Scan2Cap), and question answering (ScanQA), reporting state-of-the-art or competitive results (e.g., 51.87% Acc@0.25 on ScanRefer, 24.80% EM@1 on ScanQA). An ablation study shows incremental gains from the proposed objectives.

Significance. If the transferability claim were fully supported, the paper would offer a simple, unified alternative to task-specific 3D VL architectures, with the appealing idea of grounding pre-training in scene-graph structure. The paper includes useful ablations (number and type of scene graph layers), qualitative results, failure-case analysis, and reports training time. Its contributions are empirical rather than theoretical, and the central claim is the effectiveness of the proposed pre-training objectives for downstream tasks. However, the experimental protocol currently does not substantiate the broader claim of learning universal, transferable 3D-language representations, because pre-training and all downstream tasks use the same ScanRefer-derived data distribution. Reproducibility is also hindered by omitted hyperparameters and an undescribed parsing step.

major comments (4)
  1. [Section 4.2, Table 1; Section 5] The pre-training stage is performed on the ScanRefer training split (Table 1, 'Dataset' column), and all downstream tasks (ScanRefer VG, Scan2Cap DC, ScanQA QA) are constructed from ScanRefer descriptions. Consequently, the gains in Table 5 (e.g., Acc@0.25 49.85 to 51.87) may reflect additional training on the target distribution rather than generic pre-training. To support the claim of universal transferable representations, the authors should pre-train on a held-out corpus (e.g., ScanScribe) and fine-tune on ScanRefer, or at least provide a compute-matched baseline that trains from scratch for the same total number of epochs with the same auxiliary losses. The statement in Section 5 that cross-sensor generalization remains challenging further underscores this limitation.
  2. [Section 3.4, Eq. (1)] The overall pre-training loss Lpre = a·LSGMCL + b·LMMM + c·LDET + d·Llang is introduced without reporting the values of the weights a, b, c, and d. In addition, the text following Eq. (1) states that τ is the temperature parameter, but Eq. (1) contains no τ; temperature appears only in Eqs. (4) and (5). These omissions prevent reproduction and make it impossible to verify that the proposed objectives, rather than the loss weighting, drive the reported improvements. Please report all hyperparameters and correct the equation/text mismatch.
  3. [Section 3.2, Level 1 (word-object alignment)] The word-object alignment requires parsing object names from the input text ('we first parse the object names such as chair and cabinet'). The manuscript does not describe this parsing mechanism, its implementation, or whether it relies on ground-truth object labels or an external NLP tool. This step is load-bearing for the SG MCL loss: without a defined parser the method cannot be reproduced, and if the parser is an oracle the practical value for unseen descriptions is unclear. Please specify the parsing procedure and its supervision.
  4. [Section 4.3, Table 5] The ablation study adds SG MCL as a single block, so the individual contributions of the three contrastive levels (word-object, sentence-referred object, and scene-level) are not assessed. The central claim that multi-level alignment is beneficial would be considerably strengthened by ablating each level independently. Please include per-level ablations.
minor comments (4)
  1. [Appendix A.1] In the supplementary material, 'The model is fune-tuned using cross entropy loss' contains a typo; it should be 'fine-tuned'.
  2. [Tables 3 and 4] The names '3D-VLP' and '3DVLP' are both used and refer to different references ([26] and [57]); please distinguish them clearly in the text and tables to avoid confusion.
  3. [Section 4.1] The description of ScanQA states the number of questions and unique answers but does not state the number of scenes or the evaluation split; please clarify these details.
  4. [Section 3.2, Eq. (3)] The neighborhood sets R_i and R_j are not defined unambiguously; please clarify whether they include incoming, outgoing, or all connected nodes for both node i and node j.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the pre-training gains are empirical and not forced by construction; the only self-citation is a non-load-bearing related-work reference, and the same-dataset protocol is a generalizability caveat, not a circular step.

full rationale

This paper makes an empirical training-method claim rather than a formal derivation. The pre-training objectives are supervised losses on annotated ScanRefer data: word-object alignment (Eq. 1), sentence-referred object alignment (Eq. 4), scene-level alignment (Eq. 5), masked language modeling (Eq. 7), and masked object modeling (Eq. 9). None of these is defined in terms of the downstream metrics they are claimed to improve, and the ablation study (Table 5) compares a from-scratch baseline against incremental pre-training objectives, so the reported gains are experimental outcomes rather than identities. No fitted parameter is later renamed as a prediction: the loss weights a, b, c, d are unspecified and the temperature tau is mentioned but absent from Eq. (1), which are reproducibility omissions, not circularity. The paper does contain one self-citation — reference [31] by co-authors Yanni Ma and Hao Liu — but it appears only in the related-work sentence 'a handful of studies [17] have begun to explore 3D scene graph [31,46]' and is not load-bearing for the paper's central claim. The same-dataset pre-training on the ScanRefer training split with downstream tasks derived from ScanRefer is a legitimate threat to the strength of the 'universal representations' claim, and the conclusion concedes that 'it remains challenging to pre-train our model on 3D-text pairs collected from different types of sensors.' That is a protocol/generalizability limitation, not circular reasoning, because the pre-training objectives are not logically forced to produce the reported downstream scores. The overall circularity score is therefore low, reflecting only the minor, non-load-bearing self-citation.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method relies on established components (VoteNet, GloVE, GRU, EdgeConv, cross-attention) and does not introduce new physical or conceptual entities. The main free parameters are hand-chosen hyperparameters, several of which are not reported. The domain assumptions relate to the alignment between scene graphs and language, the reliability of object proposals, and the validity of same-dataset transfer evaluation.

free parameters (5)
  • Loss weights a, b, c, d in L_pre = Not reported
    L_pre = a*L_SG_MCL + b*L_MMM + c*L_DET + d*L_lang is defined in Section 3.4, but the values are never given. They directly control the contribution of each objective and are hand-tuned, so the exact method is not reproducible.
  • Temperature tau in contrastive losses = Not specified
    Eqs. (4) and (5) use a temperature tau, but its value is not reported in the implementation details. It affects the sharpness of the contrastive alignment.
  • Word and object mask ratios = 0.2 and 0.75
    Section 4.1 sets the mask ratio for words to 0.2 and for object proposals to 0.75. These are hand-chosen and directly affect the masked modality modeling objective.
  • Number of scene graph layers = 3
    Section 4.3 ablates 1 to 4 layers and selects 3 as a trade-off between performance and cost. The ablation shows performance still improves at 4 layers (Acc@0.25 52.01 vs 51.87), so the choice is manually tuned.
  • Number of cross-attention layers H = 2
    Supplementary A.2 states two cross-attention layers are used in the cross-modality fusion. This is a hand-chosen architecture hyperparameter.
assumptions (4)
  • domain assumption 3D scene graphs and natural language descriptions are structurally aligned so that word-object, sentence-referred object, and scene-level correspondences can be used as contrastive supervision.
    Section 1 and Fig. 1 argue this alignment; the entire multi-level contrastive learning design in Section 3.2 rests on it.
  • domain assumption Object proposals from VoteNet are reliable enough to serve as scene graph nodes for fine-grained alignment.
    Section 3.2 Level 1 uses object proposals as nodes and aligns them with word features, assuming proposal categories and locations are sufficiently accurate for the contrastive learning to be effective.
  • domain assumption Pre-training on the ScanRefer train split followed by fine-tuning on ScanRefer is a valid measure of transferable pre-training.
    Table 1 lists ScanRefer as the (pre-)training dataset for the method, and the downstream tasks are ScanRefer, Scan2Cap, and ScanQA, which are all built on ScanRefer scenes. The universal-representation claim depends on this assumption.
  • standard math GloVE embeddings and a GRU encoder provide adequate text representations for 3D vision-language alignment.
    Section 3.1 text encoding uses GloVE and GRU, inheriting standard NLP representations without additional justification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 3D Scene Graph Guided Vision-Language Pre-training." pith.science (2026). https://pith.science/paper/2V7X3W7U

@misc{pith2026241118666,
  author       = {Pith},
  title        = {Pith review of: 3D Scene Graph Guided Vision-Language Pre-training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2V7X3W7U}},
  note         = {Machine review of arXiv:2411.18666}
}
read the original abstract

3D vision-language (VL) reasoning has gained significant attention due to its potential to bridge the 3D physical world with natural language descriptions. Existing approaches typically follow task-specific, highly specialized paradigms. Therefore, these methods focus on a limited range of reasoning sub-tasks and rely heavily on the hand-crafted modules and auxiliary losses. This highlights the need for a simpler, unified and general-purpose model. In this paper, we leverage the inherent connection between 3D scene graphs and natural language, proposing a 3D scene graph-guided vision-language pre-training (VLP) framework. Our approach utilizes modality encoders, graph convolutional layers and cross-attention layers to learn universal representations that adapt to a variety of 3D VL reasoning tasks, thereby eliminating the need for task-specific designs. The pre-training objectives include: 1) Scene graph-guided contrastive learning, which leverages the strong correlation between 3D scene graphs and natural language to align 3D objects with textual features at various fine-grained levels; and 2) Masked modality learning, which uses cross-modality information to reconstruct masked words and 3D objects. Instead of directly reconstructing the 3D point clouds of masked objects, we use position clues to predict their semantic categories. Extensive experiments demonstrate that our pre-training model, when fine-tuned on several downstream tasks, achieves performance comparable to or better than existing methods in tasks such as 3D visual grounding, 3D dense captioning, and 3D question answering.

Figures

Figures reproduced from arXiv: 2411.18666 by the authors.

Figure 1
Figure 1. An illustration of the natural alignment between 3D [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of our model. Given a 3D point cloud-text pair, we first use a scene encoding module to extract 3D object proposals [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Scene graph-guided multi-level contrastive learning [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative results on downstream tasks: (a) 3D visual grounding, (b) 3D dense captioning and (c) 3D question answering. The [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results on downstream 3D visual grounding. The [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Qualitative results on downstream 3D dense captioning. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Qualitative results on downstream 3D question answering. The [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Failure cases on downstream tasks: (a) 3D visual grounding and (b) 3D question answering. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 47 canonical work pages

  1. [1]

    Referit3D: Neural listeners for fine-grained 3D object identification in real-world scenes

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3D: Neural listeners for fine-grained 3D object identification in real-world scenes. In Proceedings of the European Conference on Computer Vi- sion (ECCV), pages 422–440. Springer, 2020. 1

    work page 2020

  2. [2]

    Scanqa: 3D question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3D question answering for spatial scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19129–19139, 2022. 1, 2, 3, 5, 7

    work page 2022

  3. [3]

    Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, 2005. 6

    work page 2005

  4. [4]

    3DJCG: A unified framework for joint dense captioning and visual grounding on 3D point clouds

    Daigang Cai, Lichen Zhao, Jing Zhang, Lu Sheng, and Dong Xu. 3DJCG: A unified framework for joint dense captioning and visual grounding on 3D point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16464–16473, 2022. 6, 7

    work page 2022

  5. [5]

    Scanrefer: 3D object localization in RGB-D scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3D object localization in RGB-D scans using natural language. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pages 202–221. Springer,

    work page

  6. [6]

    UniT3D: A unified trans- former for 3D dense captioning and visual grounding

    Dave Zhenyu Chen, Ronghang Hu, Xinlei Chen, Matthias Nießner, and Angel X Chang. UniT3D: A unified trans- former for 3D dense captioning and visual grounding. arXiv preprint arXiv:2212.00836, 2022. 2, 6, 7

    arXiv 2022

  7. [7]

    D 3net: A unified speaker-listener architec- ture for 3D dense captioning and visual grounding

    Dave Zhenyu Chen, Qirui Wu, Matthias Nießner, and An- gel X Chang. D 3net: A unified speaker-listener architec- ture for 3D dense captioning and visual grounding. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pages 487–505. Springer, 2022. 6, 7

    work page 2022

  8. [8]

    Language conditioned spatial relation reasoning for 3D object grounding

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3D object grounding. arXiv preprint arXiv:2211.09646, 2022. 6

    arXiv 2022

Show all 63 references
  1. [9]

    End-to-end 3D dense captioning with vote2cap-detr

    Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Tao Chen, and Gang YU. End-to-end 3D dense captioning with vote2cap-detr. arXiv preprint arXiv:2301.02508, 2023. 3

    2023 arXiv

  2. [10]

    Simclr: A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. Simclr: A simple framework for contrastive learning of visual representations. In Proceedings of the In- ternational Conference on Machine Learning (ICML), pages 1597–1607, 2023. 1

    2023

  3. [11]

    Scan2cap: Context-aware dense captioning in RGB- D scans

    Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2cap: Context-aware dense captioning in RGB- D scans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3193–3203, 2021. 1, 2, 3, 5, 7, 12

    2021

  4. [12]

    Empirical evaluation of gated recurrent neural networks on sequence modeling

    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. 3, 12

    2014 arXiv

  5. [13]

    Scannet: Richly-annotated 3D reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3D reconstructions of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5828–5839, 2017. 5

    2017

  6. [14]

    Multi-clip: Contrastive vision-language pre-training for question answering tasks in 3D scenes.arXiv preprint arXiv:2306.02329, 2023

    Alexandros Delitzas, Maria Parelli, Nikolas Hars, Geor- gios Vlassis, Sotirios Anagnostidis, Gregor Bachmann, and Thomas Hofmann. Multi-clip: Contrastive vision-language pre-training for question answering tasks in 3D scenes.arXiv preprint arXiv:2306.02329, 2023. 3

    2023 arXiv

  7. [15]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 5

    2018 arXiv

  8. [16]

    Multi-modal align- ment using representation codebook

    Jiali Duan, Liqun Chen, Son Tran, Jinyu Yang, Yi Xu, Belinda Zeng, and Trishul Chilimbi. Multi-modal align- ment using representation codebook. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15651–15660, 2022. 2

    2022

  9. [17]

    Free-form description guided 3D visual graph network for object grounding in point cloud

    Mingtao Feng, Zhen Li, Qi Li, Liang Zhang, XiangDong Zhang, Guangming Zhu, Hui Zhang, Yaonan Wang, and Aj- mal Mian. Free-form description guided 3D visual graph network for object grounding in point cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vi...

    2021

  10. [18]

    Viewrefer: Grasp the multi-view knowledge for 3D visual grounding with gpt and prototype guidance

    Ziyu Guo, Yiwen Tang, Renrui Zhang, Dong Wang, Zhi- gang Wang, Bin Zhao, and Xuelong Li. Viewrefer: Grasp the multi-view knowledge for 3D visual grounding with gpt and prototype guidance. arXiv preprint arXiv:2303.16894 ,

    arXiv

  11. [19]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000–16009, 2022. 1

    2022

  12. [20]

    Text-guided graph neural networks for refer- ring 3D instance segmentation

    Pin-Hao Huang, Han-Hung Lee, Hwann-Tzong Chen, and Tyng-Luh Liu. Text-guided graph neural networks for refer- ring 3D instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , pages 1610– 1618, 2021. 6

    2021

  13. [21]

    Multi- view transformer for 3D visual grounding

    Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi- view transformer for 3D visual grounding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15524–15533, 2022. 1, 6

    2022

  14. [22]

    Clip2point: Transfer clip to point cloud classifica- tion with image-depth pre-training

    Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo. Clip2point: Transfer clip to point cloud classifica- tion with image-depth pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pag...

    2023

  15. [23]

    Bottom up top down detection transform- ers for language grounding in images and point clouds

    Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Kate- rina Fragkiadaki. Bottom up top down detection transform- ers for language grounding in images and point clouds. In Proceedings of the European Conference on Computer Vi- sion (ECCV), pages 417–433. Springer, 2022. 7

    2022

  16. [24]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom 9 Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In Proceedings of the International Conference on Machine Lea...

    2021

  17. [25]

    More: Multi-order relation mining for dense captioning in 3D scenes

    Yang Jiao, Shaoxiang Chen, Zequn Jie, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. More: Multi-order relation mining for dense captioning in 3D scenes. In Proceedings of the Eu- ropean Conference on Computer Vision (ECCV), pages 528–

  18. [26]

    Context-aware alignment and mutual masking for 3D-language pre-training

    Zhao Jin, Munawar Hayat, Yuwei Yang, Yulan Guo, and Yin- jie Lei. Context-aware alignment and mutual masking for 3D-language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10984–10994, 2023. 2, 6, 7

    2023

  19. [27]

    Lang3DSG: Language- based contrastive pre-training for 3D scene graph prediction

    Sebastian Koch, Pedro Hermosilla, Narunas Vaskevicius, Mirco Colosi, and Timo Ropinski. Lang3DSG: Language- based contrastive pre-training for 3D scene graph prediction. arXiv preprint arXiv:2310.16494, 2023. 3

    2023 arXiv

  20. [28]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, 2004. 6

    2004

  21. [29]

    3D-SPS: Single- stage 3D visual grounding via referred point progressive se- lection

    Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu. 3D-SPS: Single- stage 3D visual grounding via referred point progressive se- lection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pag...

    2022

  22. [30]

    Sgformer: Semantic graph transformer for point cloud-based 3D scene graph generation

    Changsheng Lv, Mengshi Qi, Xia Li, Zhengyuan Yang, and Huadong Ma. Sgformer: Semantic graph transformer for point cloud-based 3D scene graph generation. In Pro- ceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 4035–4043, 2024. 3

    2024

  23. [31]

    Heterogeneous graph learning for scene graph prediction in 3d point clouds

    Yanni Ma, Hao Liu, Yun Pei, and Yulan Guo. Heterogeneous graph learning for scene graph prediction in 3d point clouds. In Proceedings of the European Conference on Computer Vi- sion (ECCV), pages 274–291. Springer, 2025. 1

    2025

  24. [32]

    Complete 3D relationships extraction modality align- ment network for 3D dense captioning

    Aihua Mao, Zhi Yang, Wanxin Chen, Ran Yi, and Yong-jin Liu. Complete 3D relationships extraction modality align- ment network for 3D dense captioning. IEEE Transactions on Visualization and Computer Graphics (TVCG), 2023. 7

    2023

  25. [33]

    An end-to- end transformer model for 3d object detection

    Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to- end transformer model for 3d object detection. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision (ICCV), pages 2906–2917, 2021. 7

    2021

  26. [34]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pages 311– 318, 2002. 6

    2002

  27. [35]

    Clip-guided vision-language pre-training for question answering in 3D scenes

    Maria Parelli, Alexandros Delitzas, Nikolas Hars, Geor- gios Vlassis, Sotirios Anagnostidis, Gregor Bachmann, and Thomas Hofmann. Clip-guided vision-language pre-training for question answering in 3D scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2023

  28. [36]

    Pytorch: An im- perative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Pro- ceedings of the Advances in Neural Information ...

    2019

  29. [37]

    Glove: Global vectors for word representation

    Jeffrey Pennington, Richard Socher, and Christopher D Man- ning. Glove: Global vectors for word representation. In Pro- ceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, 2014. 3, 12

    2014

  30. [38]

    PointNet: Deep learning on point sets for 3D classification and segmentation

    Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 652–660, 2017. 3

    2017

  31. [39]

    PointNet++: Deep hierarchical feature learning on point sets in a metric space

    Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. PointNet++: Deep hierarchical feature learning on point sets in a metric space. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2017. 3, 12

    2017

  32. [40]

    Qi, Or Litany, Kaiming He, and Leonidas J

    Charles R. Qi, Or Litany, Kaiming He, and Leonidas J. Guibas. Deep hough voting for 3D object detection in point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9277–9286, 2019. 3, 5, 6, 7, 12

    2019

  33. [41]

    Improving language understanding by gen- erative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by gen- erative pre-training. 2018. 2

    2018

  34. [42]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In Proceedings of the International Conference ...

    2021

  35. [43]

    Mask3D: Mask trans- former for 3D semantic instance segmentation

    Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3D: Mask trans- former for 3D semantic instance segmentation. In Proceed- ings of the IEEE International Conference on Robotics and Automation (ICRA), pages 8216–8223. IEEE, 2023. 7

    2023

  36. [44]

    Vl-bert: Pre-training of generic visual- linguistic representations

    Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual- linguistic representations. arXiv preprint arXiv:1908.08530,

    1908 arXiv

  37. [45]

    Cider: Consensus-based image description evalua- tion

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4566–4575,

  38. [46]

    Learning 3D semantic scene graphs from 3D indoor reconstructions

    Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3D semantic scene graphs from 3D indoor reconstructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3961–3970, 2020. 1, 3

    2020

  39. [47]

    Spatiality-guided transformer for 3D dense captioning on point clouds

    Heng Wang, Chaoyi Zhang, Jianhui Yu, and Weidong Cai. Spatiality-guided transformer for 3D dense captioning on point clouds. arXiv preprint arXiv:2204.10688 , 2022. 3, 7 10

    2022 arXiv

  40. [48]

    Vl-sat: visual-linguistic semantics assisted training for 3D semantic scene graph prediction in point cloud

    Ziqin Wang, Bowen Cheng, Lichen Zhao, Dong Xu, Yang Tang, and Lu Sheng. Vl-sat: visual-linguistic semantics assisted training for 3D semantic scene graph prediction in point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pag...

    2023

  41. [49]

    Eda: Explicit text-decoupling and dense alignment for 3D visual and language learning

    Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang. Eda: Explicit text-decoupling and dense alignment for 3D visual and language learning. arXiv preprint arXiv:2209.14941, 2022. 2, 7

    2022 arXiv

  42. [50]

    Vision-language pre-training with triple contrastive learning

    Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-language pre-training with triple contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , ...

    2022

  43. [51]

    Sat: 2D semantics assisted training for 3D visual grounding

    Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo. Sat: 2D semantics assisted training for 3D visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1856–1866,

  44. [52]

    Deep modular co-attention networks for visual question an- swering

    Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question an- swering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6281–6290, 2019. 7, 12

    2019

  45. [53]

    Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual refer- ring

    Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual refer- ring. In Proceedings of the IEEE/CVF International Con...

    2021

  46. [54]

    X-trans2cap: Cross-modal knowledge transfer using transformer for 3D dense caption- ing

    Zhihao Yuan, Xu Yan, Yinghong Liao, Yao Guo, Guanbin Li, Shuguang Cui, and Zhen Li. X-trans2cap: Cross-modal knowledge transfer using transformer for 3D dense caption- ing. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 856...

    2022

  47. [55]

    Exploiting edge-oriented reasoning for 3D point-based scene graph analysis

    Chaoyi Zhang, Jianhui Yu, Yang Song, and Weidong Cai. Exploiting edge-oriented reasoning for 3D point-based scene graph analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9705–9715, 2021. 8, 12

    2021

  48. [56]

    Pointclip: Point cloud understanding by clip

    Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xu- peng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8552–8562, 2022. 2

    2022

  49. [57]

    Vision-language pre-training with object con- trastive learning for 3D scene understanding

    Taolin Zhang, Sunan He, Tao Dai, Zhi Wang, Bin Chen, and Shu-Tao Xia. Vision-language pre-training with object con- trastive learning for 3D scene understanding. InProceedings of the AAAI Conference on Artificial Intelligence (AAAI) , pages 7296–7304, 2024. 7

    2024

  50. [58]

    3DVG- Transformer: Relation modeling for visual grounding on point clouds

    Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3DVG- Transformer: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2928–2937,

  51. [59]

    Towards explainable 3D grounded visual question answer- ing: A new benchmark and strong baseline

    Lichen Zhao, Daigang Cai, Jing Zhang, Lu Sheng, Dong Xu, Rui Zheng, Yinjie Zhao, Lipeng Wang, and Xibo Fan. Towards explainable 3D grounded visual question answer- ing: A new benchmark and strong baseline. IEEE Transac- tions on Circuits and Systems for Video Technology (TCSVT),

  52. [60]

    Contextual modeling for 3D dense captioning on point clouds

    Yufeng Zhong, Long Xu, Jiebo Luo, and Lin Ma. Contextual modeling for 3D dense captioning on point clouds. arXiv preprint arXiv:2210.03925, 2022. 3, 7

    2022 arXiv

  53. [61]

    Point- clip v2: Prompting clip and gpt for powerful 3D open-world learning

    Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, and Peng Gao. Point- clip v2: Prompting clip and gpt for powerful 3D open-world learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2639–2650,

  54. [62]

    3d-vista: Pre-trained transformer for 3D vision and text alignment

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3D vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2911–2921, 2023. 6, 7 11 A.1. Downstream tas...

    2023

  55. [545]

    Springer, 2022. 3, 7

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.