Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyond

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 3D vision-language model on Gaussian splats claims detector-free scene reasoning and a five-fold gain over point-cloud baselines in out-of-domain settings.

desk verdict A solid systems contribution showing that Gaussian-splat 3D VLMs with a dual sparsifier work on in-domain benchmarks, but the fivefold OOD claim rests on one uncontrolled comparison and should not be the headline. read the letter →

arxiv 2507.00886 v1 pith:3M5TYOBG submitted 2025-07-01 cs.CV cs.RO

Anna-Maria Halacheva , Jan-Nico Zaech , Xi Wang , Danda Pani Paudel , Luc Van Gool This is my paper · ORCID
classification cs.CVcs.RO
keywords 3Dvision-languagemodelGaussiansplattingscene-centricreasoningdualsparsificationembodiedAIobjectdetector-freeSigLIP2sceneunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GaussianVLM is a 3D vision-language model that operates directly on Gaussian splat scenes instead of point clouds, attaching a language embedding to each Gaussian and then distilling those dense embeddings into a small set of task-aware tokens. The paper claims this is the first Gaussian-splatting-based VLM, and that the detector-free, scene-centric design outperforms prior object-centric 3D VLMs across embodied reasoning, planning, dialogue, and object captioning benchmarks. Its headline result is a five-fold improvement over the point-cloud baseline LL3DA on a new out-of-domain object-counting benchmark built from RGB-derived ScanNet++ reconstructions. The work matters because it suggests that photorealistic RGB-reconstructed scenes, combined with early language–geometry alignment, can support open-vocabulary embodied reasoning without a predefined object taxonomy.

What carries the argument

The load-bearing components are language-aligned per-Gaussian features and a dual sparsifier. SceneSplat, a 3D transformer operating on Gaussian splats, outputs a SigLIP2 language feature for each Gaussian, so the scene is natively represented in the same embedding space as text. The dual sparsifier then re-tokenizes the dense scene: a task-guided pathway applies depth-wise cross-attention, using attention-pooled task-prompt queries, to reduce the per-decoder-layer tokens first to 512 and then to 128 task-selected scene tokens, while a location-guided pathway extracts four ROI tokens from Gaussians within a 15 cm radius of any prompt-specified location. Learnable Fourier position encodings are injected in the final sparsifier layer, and a contrastive pre-training stage aligns sparsifier output with SigLIP2 label embeddings for spatial grounding. This machinery is what converts a dense, language-augmented Gaussian representation into a compact input an LLM can consume.

What would settle it

Repeat the ScanNet++ object-counting evaluation with the same GaussianVLM but with per-Gaussian features randomly permuted across Gaussians (preserving the feature distribution while destroying spatial–semantic correspondence); if accuracy stays near 24.1%, the dual sparsifier and LLM are not actually using the language alignment, and if accuracy drops to near LL3DA's 4.2%, the per-Gaussian features are the load-bearing component.

Watch

Extended reading notes

Core claim

The central claim is that a 3D VLM can be built by treating the entire scene, in the form of Gaussian splats, as the visual token source: the SceneSplat backbone predicts a SigLIP2 language feature for every Gaussian, and a dual sparsifier compresses these tens of thousands of features into 128 task-selected scene tokens plus four location-guided region-of-interest tokens, which are then fed to a frozen LLM tuned with LoRA. On the LEO protocol, GaussianVLM reaches 49.4% exact-match accuracy on SQA3D versus LEO's 47.0%; on the LL3DA protocol it raises embodied dialogue CIDEr from 145.9 to 270.1 and embodied planning CIDEr from 65.1 to 220.4, and lifts Nr3D object-caption METEOR from 5.8 to 20.8. On the new out-of-domain ScanNet++ counting benchmark, it reaches 24.1% accuracy against LL3DA's 4.2%, the 'five-fold' improvement of the abstract. The claimed mechanism is that language aligned at the per-Gaussian level, before any tokenization, preserves the open vocabulary and scene context that detector-based pipelines discard.

Load-bearing premise

The method inherits SceneSplat's per-Gaussian SigLIP2 features as semantic ground truth; if those features are misaligned or noisy on RGB-derived Gaussian splats, the sparsifier and LLM have no correct signal to recover, and the reported gains collapse.

Editorial extensions

If this is right

  • Object detectors become optional for 3D vision-language tasks; a scene-centric model can match or beat detector-dependent baselines on object-level benchmarks like Nr3D and ScanQA.
  • RGB-only capture pipelines can produce scene representations strong enough for embodied reasoning, since GaussianVLM's gains are largest on out-of-domain reconstructions made from RGB images.
  • Task-aware re-tokenization of dense 3D features is a viable alternative to fixed region tokenization, and a simple uniform downsampling is enough before cross-attention sparsification.
  • Embedding language features at the per-element level gives a single architecture that handles both scene-level tasks (dialogue, planning, captioning) and object-level tasks (referring and counting) without switching representations.
  • The dual sparsifier reduces a 40k-token scene to 128 + 4 tokens, making language-augmented 3D scenes practical for a frozen LLM with only 19M learnable parameters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the per-Gaussian language features are doing the heavy lifting, then improving the SceneSplat backbone's dense alignment should transfer directly to all downstream tasks; the paper does not isolate feature quality from sparsifier design.
  • The dual sparsifier is representation-agnostic: the same task-guided and location-guided distillation could be applied to per-point language features on point clouds, meshes, or neural radiance fields, which would test whether the gains come from Gaussian splats or from language-aligned dense features.
  • A direct ablation that permutes per-Gaussian features before sparsification would reveal how much of the five-fold OOD gain depends on spatial semantic alignment versus the sparsifier-LM readout.
  • The object-counting benchmark only covers one task type, so the strong OOD number should be read as a proof of concept until broader OOD benchmarks appear.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes GaussianVLM, a 3D vision-language model that operates directly on Gaussian splat scenes. The 3D backbone (SceneSplat) predicts a SigLIP2 language feature for each Gaussian, and a dual sparsifier (task-guided and location-guided) distills the dense per-Gaussian features into 128 task-selected scene tokens plus four ROI tokens, which are fed to a frozen LLM with LoRA. The manuscript reports state-of-the-art results on LL3DA and LEO scene-centric benchmarks, strong object-centric results on ScanRefer, ScanQA, and Nr3D, and a claimed fivefold accuracy improvement over LL3DA on a custom out-of-domain object-counting benchmark on ScanNet++ scenes.

Significance. If the central claims are upheld, the work is significant: it removes the object-detector dependency of prior 3D VLMs, introduces a language-aligned, task-aware tokenization for Gaussian splats, and provides in-domain evidence that the approach outperforms strong point-cloud baselines on embodied reasoning and planning tasks. The in-domain results are plausible, the ablations in Table IV isolate the contribution of the sparsifier components, and the paper is open about many implementation details. However, the headline out-of-domain claim is currently not established because the OOD comparison is confounded by simultaneous changes in architecture and input representation, and the paper's own limitation statement weakens the claim. The significance would be high if the OOD claim is properly controlled, but as written the generalization evidence is insufficient to support the abstract's strong wording.

major comments (3)
  1. [§IV-F, Table III, Abstract] The central out-of-domain claim ('improves performance of prior 3D VLM five folds, in out-of-the-domain settings') rests on an uncontrolled comparison. GaussianVLM receives GaussianWorld ScanNet++ Gaussian splats reconstructed from RGB images, while LL3DA receives COLMAP point-cloud reconstructions of the same scenes. The two systems thus differ simultaneously in architecture and in input representation, so the reported 474% accuracy improvement cannot be attributed to the proposed method or to the Gaussian-splat representation. A matched-input control is required, for example LL3DA evaluated on point clouds extracted from the same GaussianWorld reconstructions, or GaussianVLM evaluated on splats fitted to the COLMAP geometry if feasible. In addition, the OOD benchmark is a single automatically generated counting task with 1,000 QA pairs, a custom numeric-token accuracy metric, and no error bars or seed variance; the baseline operates near the floor (4.2% accuracy), making relative improvements unstable. The manuscript's own §IX limitation states that the OOD evaluation covers only one task, which contradicts the strong generalization wording in the abstract and introduction. Please either provide a properly controlled, multi-task OOD evaluation with confidence intervals, or temper the claims accordingly.
  2. [§III-A, §IV-F] The method's entire pipeline depends on the assumption that SceneSplat produces semantically aligned SigLIP2 features for each Gaussian, but no independent evidence is provided for the quality of these features on RGB-derived Gaussian splats, particularly on the out-of-domain ScanNet++ scenes. If the per-Gaussian features are noisy or misaligned on these inputs, the dual sparsifier has no reliable signal to distill, and the downstream gains would not be interpretable. I request a feature-level diagnostic, such as a retrieval or linear-probe classification test on the OOD scenes, or a sensitivity analysis that degrades or replaces the backbone features to show how downstream performance tracks feature quality. Without such a check, the language-alignment claim is an untested assumption rather than a demonstrated property of the method.
  3. [§IV-A, Table II] For object captioning, the paper states that it uses GPT-4o to generate 40 paraphrased variants per prompt during training. This is a training-data augmentation that is applied to GaussianVLM but not, as far as the description indicates, to the reproduced LL3DA baseline. Since METEOR and ROUGE are sensitive to answer phrasing and vocabulary, this augmentation could inflate the object-centric gains (e.g., Nr3D METEOR improving from 5.8 to 20.8) independently of the 3D representation or the proposed sparsifier. To make the object-centric comparison fair, either apply the same prompt paraphrasing to the baseline, ablate the augmentation for GaussianVLM, or report both numbers.
minor comments (6)
  1. [General] There are several typographical errors that should be corrected, including 'utlizes' in the Introduction, 'For the more' in the Introduction, 'and and' in Section II-C, and 'correcly' in the Appendix figure captions.
  2. [Table IV] In the object-centric ablation, row (2) 'No Scene Tokens' reports ScanQA Exact Match of 15.4, which is higher than the full model's 14.4. The claim that removing either module results in a 'substantial performance decrease' is not supported for this metric and should be qualified, for example by noting that the drop occurs on the aggregate metrics even if one metric improves.
  3. [Abstract, §IV-F] The abstract promises 'open access to all assets,' but no code repository, dataset link, or model release identifier appears in the manuscript. A link should be provided for reproducibility, especially for the custom object-counting benchmark.
  4. [All tables, especially Table III] No error bars or confidence intervals are reported anywhere. For the new OOD benchmark, where the comparison is close to the floor and the dataset is automatically generated, reporting variance across multiple evaluation seeds or per-question bootstrap intervals is essential.
  5. [§IV-F, Table III] The 'Accuracy' metric is described only in the Appendix ('custom accuracy metric that extracts numeric tokens... and compares them after normalization'). Since it is the headline metric in Table III, it should be defined in the main text or by a clear pointer to the Appendix.
  6. [§III-B, Eq. (2)] In the contrastive loss, the denominator notation 'sum_{j=1N}' should be written as 'sum_{j=1}^N' for clarity, and the spacing around the fraction in the negative log term should be fixed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the core results are externally benchmarked; the fivefold OOD claim rests on an uncontrolled comparison, which is an evaluation-validity problem, not a by-construction reduction.

full rationale

The paper's derivation chain is not circular in the relevant sense. GaussianVLM is trained with a standard autoregressive language-modeling loss (Eq. 1) and a contrastive objective (Eq. 2) on labeled object-caption data, and all headline numbers are obtained by evaluating the trained system on held-out external benchmarks (ScanRefer, ScanQA, Nr3D, SQA3D, 3D-LLM, and the newly constructed ScanNet++ counting set); no metric used for the fivefold claim is fitted to that benchmark and then re-reported as a prediction. The close dependence on SceneSplat [27] as a frozen pretrained 3D backbone is a use of an external model, not a circular justification: the paper invokes no SceneSplat theorem that forces its conclusions, and the downstream comparisons are made against LL3DA, LEO, and other baselines rather than against SceneSplat's own outputs. Hyperparameters such as 128 scene tokens, 4 ROI tokens, and the 15 cm ROI radius are inherited from prior ablations; architecture constants chosen from previous work do not amount to predicting the evaluated quantities. The appendix even limits its own OOD benchmark ('covers only one type of task in the out-of-distribution (OOD) evaluation setting'), and the IV-F experiment confounds architecture with input representation, because GaussianVLM sees GaussianWorld splats while LL3DA sees COLMAP point clouds; that is an external-validity or correctness concern, not a circularity. The unverified assumption that SceneSplat's per-Gaussian SigLIP2 features are semantically aligned is also an empirical premise, not a definitional loop.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim leans on the quality of pretrained SceneSplat features and on hand-chosen token budgets. These are documented in the paper but not independently justified beyond ablations. No new physical entities or new mathematical postulates are introduced.

free parameters (5)
  • ROI radius = 15 cm
    Radius for location-guided sparsifier; authors ablate 15 cm vs 30 cm and select 15 cm because it improves object-centric tasks (Tab VII).
  • Task-selected scene token count = 128
    Fixed number of global scene tokens after task-guided cross-attention; taken from LSceneLLM findings (Sec VII).
  • ROI token count = 4
    Number of region tokens produced by attention pooling; set to match LL3DA's location-aware encoding ablation (Sec VII).
  • Intermediate downsampled token count = 512
    Uniform downsampling target per decoder layer before cross-attention; chosen for efficiency and justified by ablation (Sec III-B, IV-G).
  • Sampled Gaussian count per scene = 40k
    Randomly sampled Gaussians used as input; follows prior work (Sec IV-D), but the sampled subset is not shown to preserve small objects.
assumptions (4)
  • domain assumption SceneSplat's per-Gaussian predictions are semantically aligned with the SigLIP2 language space.
    Section III-A adopts SceneSplat as the vision module and relies on its per-Gaussian language features without independent validation in this paper.
  • domain assumption 40k randomly sampled Gaussians preserve enough scene content for QA and captioning.
    Section IV-D samples 40k Gaussians per scene; random sampling can drop small objects, yet the method's performance depends on it.
  • domain assumption The LLM can condition on a small set of tokens without losing task-critical spatial information.
    The dual sparsifier compresses 40k tokens to 128 scene tokens plus 4 ROI tokens; no formal guarantee or reconstruction check is provided (Sec III-B).
  • domain assumption GaussianWorld ScanNet++ scenes and COLMAP point clouds are comparable test beds for OOD generalization.
    Table III compares GaussianVLM on Gaussian splats to LL3DA on COLMAP point clouds; the representations differ in source and density, so the comparison conflates method and input.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyond." pith.science (2026). https://pith.science/paper/3M5TYOBG

@misc{pith2026250700886,
  author       = {Pith},
  title        = {Pith review of: GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyond},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3M5TYOBG}},
  note         = {Machine review of arXiv:2507.00886}
}
read the original abstract

As multimodal language models advance, their application to 3D scene understanding is a fast-growing frontier, driving the development of 3D Vision-Language Models (VLMs). Current methods show strong dependence on object detectors, introducing processing bottlenecks and limitations in taxonomic flexibility. To address these limitations, we propose a scene-centric 3D VLM for 3D Gaussian splat scenes that employs language- and task-aware scene representations. Our approach directly embeds rich linguistic features into the 3D scene representation by associating language with each Gaussian primitive, achieving early modality alignment. To process the resulting dense representations, we introduce a dual sparsifier that distills them into compact, task-relevant tokens via task-guided and location-guided pathways, producing sparse, task-aware global and local scene tokens. Notably, we present the first Gaussian splatting-based VLM, leveraging photorealistic 3D representations derived from standard RGB images, demonstrating strong generalization: it improves performance of prior 3D VLM five folds, in out-of-the-domain settings.

Figures

Figures reproduced from arXiv: 2507.00886 by the authors.

Figure 1
Figure 1. The proposed GaussianVLM performs comprehensive scene understanding in natural language for 3D scenes [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The GaussianVLM architecture processes a user task prompt (query and optional location) and a 3D scene (Gaussian Splat representation). A 3D vision module (SceneSplat Transformer) predicts per-Gaussian language features. These dense features are then sparsified by a dual sparsifier module. The decoder’s hidden states also inform the task-guided sparsifier. The dual sparsifier comprises: 1) a location-guided pathway … view at source ↗
Figure 3
Figure 3. Qualitative results on scene-centric tasks. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Qualitative results on object-centric tasks. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Distribution of the questions on object counts, answered [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Distribution of the questions on object counts, answered [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Distribution of object count questions (correcly answered by GaussianVLM, vs all questions) according to [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Distribution of object count questions (correcly answered by LL3DA, vs all questions) according to [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Distribution of object count questions according to [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Distribution of object count questions (correcly answered by GaussianVLM, vs all questions) according to [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Distribution of object count questions (correcly answered by LL3DA, vs all questions) according to [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Distribution of object count questions based on ground truth object counts labels (log scale). We show the initial [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Distribution of object count questions based on object type label. We show the initial distribution upon generating [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. 3D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    3D-R1 uses a synthetic chain-of-thought cold start plus GRPO reinforcement learning with perception, semantic, and format rewards, and reports best published results across 3D dense captioning, QA, grounding, dialogue...

  2. Nav-R1: Reasoning and Navigation in Embodied Scenes

    cs.RO 2025-09 reject novelty 4.0 of 10

    Nav-R1 uses a 110K synthetic CoT dataset, GRPO with three rewards, and a fast-in-slow system to set new SOTA on R2R-CE, RxR-CE, and HM3D-OVON.

Reference graph

Works this paper leans on

56 extracted references · 41 canonical work pages · cited by 2 Pith papers

  1. [1]

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas J. Guibas. ReferIt3D: Neural listeners for fine-grained 3d object identification in real-world scenes. InECCV, 2020

    work page 2020

  2. [2]

    Spice: Semantic propositional image caption evaluation

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. InECCV, 2016

    work page 2016

  3. [3]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In CVPR, 2022

    work page 2022

  4. [4]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 2005

    work page 2005

  5. [5]

    π0: A vision-language-action flow model for general robot control.arXiv:2410.24164, 2024

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, and et al. π0: A vision-language-action flow model for general robot control.arXiv:2410.24164, 2024

    arXiv 2024

  6. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, and et al. Language models are few-shot learners. InNeurIPS, volume 33, 2020

    work page 2020

  7. [7]

    Language models are few-shot learners.NeurIPS, 33, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, et al. Language models are few-shot learners.NeurIPS, 33, 2020

    work page 2020

  8. [8]

    Scanrefer: 3d object localization in rgb-d scans using natural language.ECCV, 2020

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language.ECCV, 2020

    work page 2020

Show all 56 references
  1. [9]

    Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, et al. Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. InCVPR, 2024

    2024

  2. [10]

    End-to-end 3d dense captioning with vote2cap-detr

    Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, et al. End-to-end 3d dense captioning with vote2cap-detr. InCVPR, 2023

    2023

  3. [11]

    Grounded 3d-llm with referent tokens.arXiv:2405.10370, 2024

    Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Ruiyuan Lyu, et al. Grounded 3d-llm with referent tokens.arXiv:2405.10370, 2024

    2024 arXiv

  4. [12]

    Scan2cap: Context-aware dense captioning in rgb-d scans

    Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2cap: Context-aware dense captioning in rgb-d scans. InCVPR, 2021

    2021

  5. [13]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.See https://vicuna

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3), 2023

    2023

  6. [14]

    Rouge: A package for automatic evaluation of summaries

    Lin Chin-Yew. Rouge: A package for automatic evaluation of summaries. InProceedings of the Workshop on Text Summarization Branches Out, 2004, 2004

    2004

  7. [15]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. InCVPR, 2017

    2017

  8. [16]

    Embodied question answering

    Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. InCVPR, 2018

    2018

  9. [17]

    SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes

    Alexandros Delitzas, Ayca Takmaz, Federico Tombari, Robert Sumner, Marc Pollefeys, et al. SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes. InCVPR, 2024

    2024

  10. [18]

    Scene- llm: Extending language model for 3d visual reasoning

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene- llm: Extending language model for 3d visual reasoning. InIEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2025

    2025

  11. [19]

    Perla: Perceptive 3d language assistant

    Mei Guofeng, Lin Wei, Riz Luigi, Wu Yujiao, Poiesi Fabio, and Wang Yiming. Perla: Perceptive 3d language assistant. InCVPR, 2025

    2025

  12. [20]

    3d-llm: Injecting the 3d world into large language models.NeurIPS, 2023

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models.NeurIPS, 2023

    2023

  13. [21]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2), 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2), 2022

    2022

  14. [22]

    Chat-scene: Bridging 3d scene and large language models with object identifiers

    Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. InNeurIPS, 2024

    2024

  15. [23]

    An embodied generalist agent in 3d world

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, et al. An embodied generalist agent in 3d world. InICML, 2024

    2024

  16. [24]

    Less is more: Clipbert for video-and-language learning via sparse sampling

    Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, et al. Less is more: Clipbert for video-and-language learning via sparse sampling. InCVPR, 2021

    2021

  17. [25]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InICML, 2023

    2023

  18. [26]

    M3dbench: Towards omni 3d assistant with interleaved multi-modal instructions

    Mingsheng Li, Xin Chen, Chi Zhang, Sijin Chen, Hongyuan Zhu, et al. M3dbench: Towards omni 3d assistant with interleaved multi-modal instructions. InECCV, 2025

    2025

  19. [27]

    Scenesplat: Gaussian splatting-based scene understanding with vision- language pretraining.arXiv:2503.18052, 2025

    Yue Li, Qi Ma, Runyi Yang, Huapeng Li, Mengjiao Ma, Bin Ren, Nikola Popovic, Nicu Sebe, Ender Konukoglu, Theo Gevers, et al. Scenesplat: Gaussian splatting-based scene understanding with vision- language pretraining.arXiv:2503.18052, 2025

    2025 arXiv

  20. [28]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,NeurIPS, 2023

    2023

  21. [29]

    Robomp 2: A robotic multimodal perception-planning framework with mutlimodal large language models

    Qi Lv, Hao Li, Xiang Deng, Rui Shao, et al. Robomp 2: A robotic multimodal perception-planning framework with mutlimodal large language models. InICML, 2024

    2024

  22. [30]

    Sqa3d: Situated question answering in 3d scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, et al. Sqa3d: Situated question answering in 3d scenes. InICLR, 2023

    2023

  23. [31]

    Situational awareness matters in 3d vision language reasoning

    Yunze Man, Liang-Yan Gui, and Yu-Xiong Wang. Situational awareness matters in 3d vision language reasoning. InCVPR, 2024

    2024

  24. [32]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002

    2002

  25. [33]

    Openscene: 3d scene understanding with open vocabularies

    Songyou Peng, Kyle Genova, Chiyu "Max" Jiang, Andrea Tagliasacchi, Marc Pollefeys, and Thomas Funkhouser. Openscene: 3d scene understanding with open vocabularies. InCVPR, 2023

    2023

  26. [34]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, et al. Learning transferable visual models from natural language supervision. InICML, 2021

    2021

  27. [35]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of Machine Learning Research, 21(140), 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of Machine Learning Research, 21(140), 2020

    2020

  28. [36]

    Sayplan: Grounding large language models using 3d scene graphs for scalable task planning

    Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, et al. Sayplan: Grounding large language models using 3d scene graphs for scalable task planning. InCoRL, 2023

    2023

  29. [37]

    Sentence-BERT: Sentence embed- dings using Siamese BERT-networks

    Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embed- dings using Siamese BERT-networks. InEmpirical Methods in Natural Language Processing. ACL, 2019

    2019

  30. [38]

    Structure-from- motion revisited

    Johannes L Schonberger and Jan-Michael Frahm. Structure-from- motion revisited. InCVPR, 2016

    2016

  31. [39]

    Pixelwise view selection for unstructured multi-view stereo

    Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view selection for unstructured multi-view stereo. InECCV, 2016

    2016

  32. [40]

    Splat-mover: Multi-stage, open-vocabulary robotic manipulation via editable gaussian splatting

    Ola Shorinwa, Johnathan Tucker, Aliyah Smith, Aiden Swann, et al. Splat-mover: Multi-stage, open-vocabulary robotic manipulation via editable gaussian splatting. InCoRL, 2024

    2024

  33. [41]

    Habitat 2.0: Training home assistants to rearrange their habitat

    Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, et al. Habitat 2.0: Training home assistants to rearrange their habitat. In NeurIPS, 2021

    2021

  34. [42]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288, 2023

    2023 arXiv

  35. [43]

    Siglip 2: Multilingual vision- language encoders with improved semantic understanding, localization, and dense features.arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, et al. Siglip 2: Multilingual vision- language encoders with improved semantic understanding, localization, and dense features.arXiv:2502.14786, 2025

    2025 arXiv

  36. [44]

    Cider: Consensus-based image description evaluation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. InCVPR, 2015

    2015

  37. [45]

    Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes, 2023

    Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes, 2023

    2023

  38. [46]

    Embodied question answering in photorealistic environments with point cloud perception

    Erik Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruv Batra. Embodied question answering in photorealistic environments with point cloud perception. InCVPR, 2019

    2019

  39. [47]

    Tidybot: Personalized robot assistance with large language models

    Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, et al. Tidybot: Personalized robot assistance with large language models. InIROS, 2023

    2023

  40. [48]

    3d vision and language pretraining with large-scale synthetic data.arXiv:2407.06084, 2024

    Dejie Yang, Zhu Xu, Wentao Mo, Qingchao Chen, Siyuan Huang, and Yang Liu. 3d vision and language pretraining with large-scale synthetic data.arXiv:2407.06084, 2024

    2024 arXiv

  41. [49]

    Scannet++: A high-fidelity dataset of 3d indoor scenes

    Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. InICCV, 2023

    2023

  42. [50]

    Deep modular co-attention networks for visual question answering

    Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. InCVPR, 2019

    2019

  43. [51]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InICCV, 2023

    2023

  44. [52]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, et al. Opt: Open pre-trained transformer language models. arXiv:2205.01068, 2022

    2022 arXiv

  45. [53]

    Spartun3d: Situated spatial understanding of 3d world in large language models.arXiv:2410.03878, 2024

    Yue Zhang, Zhiyang Xu, Ying Shen, Parisa Kordjamshidi, and Lifu Huang. Spartun3d: Situated spatial understanding of 3d world in large language models.arXiv:2410.03878, 2024

    2024 arXiv

  46. [54]

    Lscenellm: Enhancing large 3d scene understanding using adaptive visual preferences.arXiv:2412.01292, 2024

    Hongyan Zhi, Peihao Chen, Junyan Li, Shuailei Ma, Xinyu Sun, et al. Lscenellm: Enhancing large 3d scene understanding using adaptive visual preferences.arXiv:2412.01292, 2024

    2024 arXiv

  47. [55]

    3d-vista: Pre-trained transformer for 3d vision and text alignment

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. InCVPR, 2023

    2023

  48. [56]

    How many

    Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, et al. Unifying 3d vision-language understanding via promptable queries. InECCV, 2025. APPENDIX This supplementary material provides additional results, implementation details, and data information supporting the m...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.