Pith. sign in

REVIEW 4 major objections 6 minor 5 cited by

VLM² shows a video-only vision-language model can outperform prior spatial-reasoning models by pairing view-consistent 3D grounding with a persistent dual-memory module.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 20:10 UTC pith:DFX6G257

load-bearing objection A well-ablated video spatial-reasoning system with a plausible architecture and large reported gains, but the SOTA claim rests on a scene-disjointness check the paper never reports. the 4 major comments →

arxiv 2511.20644 v2 pith:DFX6G257 submitted 2025-11-25 cs.CV

Vision-Language Memory for Spatial Reasoning

Zuntao Liu , Yi Du , Taimeng Fu , Shaoshu Su , Cherie Ho , Chen Wang This is my paper
classification cs.CV
keywords spatial reasoningvision-language modelvideo understanding3D-aware representationmemorydual-memoryVSI-BenchVSTI-Bench
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to establish that a vision-language model fed only monocular video can build a view-consistent 3D-aware representation and keep it in a persistent dual-memory store, and that this combination achieves state-of-the-art results among video-only models on four spatial reasoning benchmarks. If correct, it would mean explicit 3D sensors—depth maps, point clouds, camera matrices—are not strictly necessary for strong spatial question answering; the needed geometry can be predicted from 2D frames and aligned with semantics. A sympathetic reader would care because spatial reasoning is a bottleneck for robots and embodied agents, and the proposed recipe is video-only, bounded in compute, and shows the largest gains on long videos, where forgetting is most acute.

Core claim

VLM² claims that two design choices—(1) explicitly aligning semantic tokens with geometric tokens and predicted 3D coordinates before fusion, using an adaptive gate to filter noisy point predictions and viewpoint tokens to disambiguate geometry, and (2) maintaining a dual memory of a sliding-window working memory and a fixed-capacity episodic memory updated by cosine-similarity replacement—let a 7B video-tuned LLM achieve top accuracy among video-based models on VSI-Bench (68.8 average), VSTI-Bench (65.3), ScanQA (CIDEr 105.5), and SQA3D (EM 60.4). The authors attribute the gains to resolving semantic-geometric misalignment and preventing overwriting of past observations, with the largest im

What carries the argument

The load-bearing mechanism is a three-stage representation builder plus a dual-memory bank. Adaptive 3D position injection adds per-patch predicted 3D coordinates (from a frozen 3D foundation model, π₃) to visual tokens, scaled by a learned gate α_t ∈ [0,1] that decides which points are trustworthy. Viewpoint-aware geometry alignment enriches geometry tokens with patch- and frame-level view tokens so identical geometry at different camera poses is not conflated. A cross-attention step then fuses the position-aware visual tokens with viewpoint-aware geometry tokens to produce a 3D-aware token sequence per frame. On top, a sliding-window working memory is attended to for immediate context, and

Load-bearing premise

The whole 3D grounding rests on the frozen π₃ model's predicted point maps and view tokens being accurate enough in the benchmark scenes; if those geometric priors are noisy or systematically wrong, the adaptive gate and cross-attention can suppress but not fix the error.

What would settle it

Run VLM² on a held-out set of videos while adding calibrated Gaussian noise to the π₃ point maps before injection; if absolute-distance and relative-direction accuracy stays flat as noise grows, the 3D injection is not the mechanism behind the gains.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the claim holds, video-only spatial reasoning can match or exceed methods that require explicit 3D inputs, lowering sensor and data requirements for embodied systems.
  • The fixed-capacity episodic memory and sliding-window working memory keep computational cost bounded, suggesting long-video spatial reasoning does not require growing context windows.
  • The largest gains on videos longer than two minutes imply the dual-memory design mitigates forgetting, a direct benefit for robotics tasks with extended observation.
  • Removing the adaptive gating from 3D position injection hurts performance (from 61.6 to 58.9 in the ablation), indicating that selective use of predicted geometry, not raw injection, is what carries part of the benefit.
  • Cross-attention fusion of semantic and geometric features substantially outperforms simple concatenation or addition, showing that the alignment mechanism, not just the presence of geometry, matters.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the dual-memory pattern—a short-window buffer plus a fixed-capacity long-term bank with similarity-based eviction—could transfer to other streaming perception tasks such as 3D reconstruction or lifelong navigation, where the same overwriting problem appears.
  • Editorial extension: the method's ceiling is likely tied to the frozen 3D foundation model's accuracy; a stronger or trainable geometric backbone might push the same architecture further, and the paper does not test this dependency directly.
  • Editorial extension: a testable hypothesis is that the cosine-similarity replacement rule preserves view diversity rather than raw capacity; one could vary the bank size while fixing the diversity of stored viewpoints to see which factor drives the long-video gains.
  • Editorial extension: the paper's counting improvements (e.g., +28.2 on appearance order) suggest the memory may encode object identity across views, but the current experiments do not isolate whether the model tracks instances or just aggregates appearance statistics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes VLM^2, a video-only vision-language model for spatial reasoning. It constructs a view-consistent 3D-aware representation from 2D video by adaptively injecting predicted 3D point-map coordinates into visual tokens (Eq. 1), aligning geometry tokens with view tokens (Eqs. 2–3), and fusing the two token streams via cross-attention (Eq. 4). A dual-memory module combines a sliding-window working memory and a fixed-capacity episodic memory with gated fusion and similarity-based updates (Algorithm 1). The model is evaluated on VSI-Bench, VSTI-Bench, ScanQA, and SQA3D, reporting state-of-the-art results among video-only models and large gains over the strongest video-based baseline VLM-3R (e.g., +7.9 on VSI-Bench, +6.5 on VSTI-Bench). Ablations in Tables 5–7 show that each component contributes gains and that memory lengths matter.

Significance. If the reported numbers are validated, VLM^2 would be a meaningful advance: it shows that a bounded, persistent memory combined with explicit viewpoint-aware 3D alignment improves spatial and spatial-temporal reasoning in video-only settings. The component ablations (Tables 5–7) are controlled and mostly support the stated mechanisms; the cross-backbone comparison in Table 6 is also informative. However, the central SOTA claim rests on the assumption that the evaluation measures generalization to unseen scenes, and the paper does not establish scene-level disjointness between training and evaluation. The absence of error bars and the under-specification of the adaptive gate further limit the present claims. No code or data are released, which constrains reproducibility.

major comments (4)
  1. [§A.1 and §4.1] The training set described in §A.1 includes “spatial reasoning datasets from VLM-3R” whose task taxonomy matches VSI-Bench exactly (object count, absolute distance, object size, room size, relative distance, relative direction, appearance order) and spatial-temporal QA pairs matching VSTI-Bench. All four evaluation benchmarks (VSI-Bench, VSTI-Bench, ScanQA, SQA3D) are built on ScanNet/ScanNet++/ARKitScenes. The paper never states that the scenes used for training are disjoint from the evaluation scenes. Without an explicit scene-level split, the reported gains (e.g., Tables 1–2) could be inflated by scene-level memorization rather than by the proposed architectural mechanisms. This concern applies directly to the headline SOTA claim. The authors must report the exact training/evaluation scene IDs and verify disjointness, or provide a held-out-scene evaluation.
  2. [Tables 1–4] All results are reported as single-point estimates with no error bars, no multiple seeds, and no significance tests. Several claimed gains are small relative to likely evaluation noise (e.g., VSI-Bench Obj. Size +1.6 in Table 1; VSTI-Bench OO-RelP +1.2 in Table 2). The claim of “consistently outperforming” all baselines is too strong without variance information. The authors should report standard deviations over at least 3 runs (or bootstrap confidence intervals on the benchmark) and indicate which differences are statistically meaningful.
  3. [§3.2, Eq. (1)] The adaptive 3D position gate α_t is central to the method, but its definition is missing. The text says only that α_t is a “learnable gate” with α_t ∈ [0,1]^{h×w×1}. It is not specified whether α_t is a per-frame learned tensor, a function of the visual tokens F_t, the point map C_t, or something else. Without this, the adaptive mechanism cannot be reproduced or compared fairly. Contrast with the memory gate γ_t, which is properly specified via MLP in Eq. (6). The authors must give the exact parameterization and input to α_t.
  4. [§3.2 and Table 6] The entire 3D grounding is built from the frozen π3 model’s predicted point maps and view tokens (Sec. 3.2). Table 6 compares π3 against VGGT and CUT3R, but it does not isolate sensitivity to the accuracy of the geometric prior. The adaptive gate can suppress but cannot correct systematic geometric errors. To support the claim that the gate robustly handles noisy predictions, the authors should add a perturbation analysis (e.g., corrupting point maps by varying noise levels) or report per-scene accuracy against π3 reconstruction error. This is a correctness-risk concern, secondary to the train/eval disjointness issue but still important for the robustness of the central claim.
minor comments (6)
  1. [§3.2] Typo: “geometrc tokens” should be “geometric tokens” in the sentence after Eq. (3).
  2. [Tables 1–2] Table headers contain typos: “Qusetion” and “Appr. Order” in Table 1; “A vg.” spacing and abbreviations in Table 2 are inconsistent. Please standardize.
  3. [Tables 3 and 10] The “Avg.” column for VLM2-7B in Table 3 (69.4 on VSI-Bench) differs from the overall average in Table 1 (68.8). The long-horizon table likely uses a different video subset or weighting, but this is not explained. Please clarify how the Short/Mid/Long partition relates to the full benchmark and how the average is computed.
  4. [§4.3] The text says the 3D-aware representation yields “an average of 8.6% accuracy gain” over the baseline (55.2 → 63.8). This is an 8.6-point absolute gain, not 8.6% relative. Please correct the wording to avoid confusion.
  5. [Table 9] The model name is inconsistently written as “VLM 2-7B” in Table 9 and “VLM2-7B” elsewhere. Please unify notation.
  6. [§A.1] The description of training data composition would benefit from exact numbers of videos and QA pairs per source, and from explicit statement of whether the source data are the same benchmark splits used for evaluation. This is related to the first major comment but should also be part of the reproducibility statement.

Circularity Check

0 steps flagged

No significant circularity: the paper's results are empirical benchmark outcomes, not identities derived from their inputs.

full rationale

The paper's central claims are empirical benchmark scores (VSI-Bench avg 68.8, VSTI-Bench avg 65.3, ScanQA CIDEr 105.5, SQA3D EM 60.4) obtained by training a video LLM with frozen π3 geometric priors, adaptive 3D position injection, viewpoint-aware geometry alignment, and a dual-memory module. None of these components is defined in terms of the benchmark targets. The learnable gate α_t in Eq. (1), the viewpoint alignment in Eqs. (2)-(3), the semantic-geometric fusion in Eq. (4), and the memory operations in Eqs. (5)-(8) are trained on QA datasets; the evaluation metrics are not used as training targets or fitted constants, and no equation in Section 3 reduces a reported quantity to its own input. The comparison follows the standard held-out evaluation protocol described in Section A.3, and training for one epoch on the same objective as VLM-3R is a normal transfer setup, not a circular derivation. The possible scene-level overlap between the mixed training collection in A.1 and the ScanNet-based evaluation sets is a data-contamination / external-validity concern, not a definitional circularity; it cannot be exhibited as an equation-level reduction. Citations to VLM-3R, π3, CUT3R, and VSI-Bench are to external prior work, and no load-bearing claim is justified solely by a self-citation chain. Therefore, under the required standard, no circular step can be identified and the appropriate score is 0.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 0 invented entities

The central claim rests on a small number of learned gates and projection layers (expected for a trained system) and on three domain assumptions: pi3 geometry reliability, the episodic-memory replacement rule as a salience proxy, and a clean train/eval split. No new physical or conceptual entities are introduced.

free parameters (7)
  • adaptive 3D position gate alpha_t = learned
    Eq (1): learnable per-token gate deciding which 3D position cues are injected into visual tokens.
  • memory fusion gate gamma_t = learned
    Eq (6): learnable gate blending working-memory and episodic-memory outputs.
  • viewpoint projection layers psi_v, psi_g = learned
    Eqs (2)-(3): linear projections mapping view tokens into geometry-token space.
  • 3D position encoder MLP phi = learned
    Section 3.2: maps 3D coordinates to position embeddings before gated injection.
  • working memory length Lw = 8
    Table 7: selected by ablation on VSI-Bench; (Lw=8, Le=32) gives the best average.
  • episodic memory capacity Le = 32
    Table 7: selected by ablation; larger capacities help up to 32.
  • number of sampled frames N = 32
    Sections A.2/A.3: uniformly sample 32 frames per video, matching the configuration of Video-3D LLM.
axioms (4)
  • domain assumption pi3's predicted point maps and view tokens provide sufficiently accurate 3D geometry for grounding.
    Section 3.2: the whole 3D-aware representation is built from pi3 outputs; if these are inaccurate, adaptive gating can only partially compensate.
  • domain assumption Cosine similarity in the learned feature space is a good proxy for task-relevant redundancy when replacing episodic-memory entries.
    Algorithm 1 lines 16-24: the most similar entry is deleted; no experiment isolates this rule against alternative update strategies.
  • domain assumption Training on VLM-3R's QA pairs plus ScanQA/SQA3D training splits transfers to the evaluated val/test splits without leakage.
    A.1 and Section 4: the mixed dataset is used for one-epoch fine-tuning; the paper does not state exact split boundaries, so leakage cannot be ruled out.
  • domain assumption Frozen SigLIP and Qwen2 features carry enough semantic and language priors for spatial QA.
    Section 3: the vision encoder and LLM are kept frozen; the method relies on their pretrained representations.

pith-pipeline@v1.3.0-alltime-deepseek · 23347 in / 11948 out tokens · 117785 ms · 2026-08-03T20:10:27.282429+00:00 · methodology

0 comments
read the original abstract

Spatial reasoning is a critical capability for intelligent robots, yet current vision-language models (VLMs) still fall short of human-level performance in video-based spatial reasoning. This gap mainly stems from two challenges: a semantic-geometric misalignment that prevents consistent 3D understanding, and the absence of persistent memory to retain 3D representation and understanding across frames. To address these limitations, we present VLM$^2$, a Vision-Language Model with persistent Memory for spatial reasoning with a view-consistent, 3D-aware representation purely from 2D videos. Specifically, we incorporate a dual-memory module consisting of a working memory that operates as a sliding window to focus on immediate context, and an episodic memory that consolidates and stores critical information across frames. This design enables bounded and efficient spatial reasoning under a fixed computational cost. Extensive experiments on multiple benchmarks show that VLM$^2$ achieves state-of-the-art performance among video-based models, significantly advancing the frontier of visual-spatial intelligence.

Figures

Figures reproduced from arXiv: 2511.20644 by Chen Wang, Cherie Ho, Shaoshu Su, Taimeng Fu, Yi Du, Zuntao Liu.

Figure 1
Figure 1. Figure 1: VLM2 is a Vision-Language Model with Memory for long-horizon spatial reasoning that constructs view-consistent 3D-aware representations from 2D video and maintains persistent memory over time. Such capabilities are critical for questions like “How many chairs are in this room?”, which require both consis￾tent cross-view alignment and long-horizon memory. curate count over time). However, existing methods o… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the VLM2 Architecture. Our model constructs a view-consistent 3D-aware representation via adaptive 3D position injection, viewpoint-aware geometry alignment and semantic-geometric fusion. A dual-memory module with a sliding-window working memory and a fixed-capacity episodic memory maintains these representations over time, supporting long-horizon spatial reasoning. truth depth maps and camera … view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative examples on VSI-Bench [60]. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative examples on VSI-Bench [60]. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative examples on VSI-Bench [60]. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative examples on VSTI-Bench [16]. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative examples on VSTI-Bench [16]. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Vesta: A Generalist Embodied Reasoning Model

    cs.RO 2026-06 unverdicted novelty 6.0

    Vesta is a unified embodied generalist model that outperforms specialist baselines by over 20% on average and improves real-world robotic task success by over 35%.

  2. Probing Collision Grounding in Vision-Language Models for Safe Human-Robot Collaboration

    cs.CV 2026-05 unverdicted novelty 6.0

    TouchSafeBench evaluates VLMs on collision grounding, finding best Macro-F1 below 50% and that explicit depth does not yield reliable robot-body contact inference.

  3. Q-GeoMem: Question-Guided Geometric Memory for Video Spatial Reasoning

    cs.CV 2026-05 unverdicted novelty 6.0

    Q-GeoMem uses question-guided scoring to maintain a Fine-Grained Context Bank and Semantic-Geometric Evidence Bank, achieving SOTA on VSI-Bench and VSTI-Bench.

  4. SPATIOROUTE: Dynamic Prompt Routing for Zero-Shot Spatial Reasoning

    cs.CV 2026-05 unverdicted novelty 6.0

    SpatioRoute introduces dynamic prompt routing that improves zero-shot spatial VQA accuracy by up to 5% on the SQA3D benchmark across VLMs without 3D inputs or fine-tuning.

  5. Q-GeoMem: Question-Guided Geometric Memory for Video Spatial Reasoning

    cs.CV 2026-05 unverdicted novelty 5.0

    Question-guided dual geometric memories with relevance-novelty utility reportedly reach state-of-the-art video spatial reasoning on two in-domain and five out-of-distribution benchmarks.

Reference graph

Works this paper leans on

77 extracted references · 25 linked inside Pith · cited by 4 Pith papers

  1. [1]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. Inproceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19129– 19139, 2022. 2, 7, 13, 14

    2022

  2. [2]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 1, 2, 6, 7, 14, 15

    Pith/arXiv arXiv 2025

  3. [3]

    Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data.arXiv preprint arXiv:2111.08897, 2021

    Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data.arXiv preprint arXiv:2111.08897, 2021. 5

    Pith/arXiv arXiv 2021

  4. [4]

    Spatialbot: Pre- cise spatial understanding with vision language models

    Wenxiao Cai, Iaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Pre- cise spatial understanding with vision language models. In 2025 IEEE International Conference on Robotics and Au- tomation (ICRA), pages 9490–9498. IEEE, 2025. 3

    2025

  5. [5]

    Tackling view-dependent semantics in 3d language gaussian splatting.arXiv preprint arXiv:2505.24746, 2025

    Jiazhong Cen, Xudong Zhou, Jiemin Fang, Changsong Wen, Lingxi Xie, Xiaopeng Zhang, Wei Shen, and Qi Tian. Tackling view-dependent semantics in 3d language gaussian splatting.arXiv preprint arXiv:2505.24746, 2025. 1

    Pith/arXiv arXiv 2025

  6. [6]

    Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 14455–14465,

  7. [7]

    Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26428–26438, 2024. 1, 2, 7

    2024

  8. [8]

    Reasoning in space via grounding in the world.arXiv preprint arXiv:2510.13800, 2025

    Yiming Chen, Zekun Qi, Wenyao Zhang, Xin Jin, Li Zhang, and Peidong Liu. Reasoning in space via grounding in the world.arXiv preprint arXiv:2510.13800, 2025. 2

    arXiv 2025

  9. [9]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling.arXiv preprint arXiv:2412.05271, 2024. 7

    Pith/arXiv arXiv 2024

  10. [10]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.Science China Information Sciences, 67(12):220101,

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.Science China Information Sciences, 67(12):220101,

  11. [11]

    Long3r: Long sequence streaming 3d re- construction

    Zhuoguang Chen, Minghui Qin, Tianyuan Yuan, Zhe Liu, and Hang Zhao. Long3r: Long sequence streaming 3d re- construction. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5273–5284,

  12. [12]

    Spatial- rgpt: Grounded spatial reasoning in vision-language mod- els.Advances in Neural Information Processing Systems, 37:135062–135093, 2024

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Rui- han Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatial- rgpt: Grounded spatial reasoning in vision-language mod- els.Advances in Neural Information Processing Systems, 37:135062–135093, 2024. 3

    2024

  13. [13]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017. 5, 7

    2017

  14. [14]

    3d-llava: Towards generalist 3d lmms with omni superpoint transformer

    Jiajun Deng, Tianyu He, Li Jiang, Tianyu Wang, Feras Day- oub, and Ian Reid. 3d-llava: Towards generalist 3d lmms with omni superpoint transformer. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 3772–3782, 2025. 1, 2

    2025

  15. [15]

    Grounding consistency: Distilling spa- tial common sense for precise visual relationship detection

    Markos Diomataris, Nikolaos Gkanatsios, Vassilis Pitsikalis, and Petros Maragos. Grounding consistency: Distilling spa- tial common sense for precise visual relationship detection. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15911–15920, 2021. 1

    2021

  16. [16]

    Vlm-3r: Vision-language models aug- mented with instruction-aligned 3d reconstruction.arXiv preprint arXiv:2505.20279, 2025

    Zhiwen Fan, Jian Zhang, Renjie Li, Junge Zhang, Runjin Chen, Hezhen Hu, Kevin Wang, Huaizhi Qu, Dilin Wang, Zhicheng Yan, et al. Vlm-3r: Vision-language models aug- mented with instruction-aligned 3d reconstruction.arXiv preprint arXiv:2505.20279, 2025. 2, 3, 5, 6, 7, 13, 14, 15, 19, 20

    Pith/arXiv arXiv 2025

  17. [17]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 24108–24118, 2025. 6

    2025

  18. [18]

    Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context

    Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv:2403.05530, 2024. 6

    Pith/arXiv arXiv 2024

  19. [19]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xue- fei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13504–13514, 2024. 3

    2024

  20. [20]

    Mea- suring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mea- suring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020. 6

    Pith/arXiv arXiv 2009

  21. [21]

    Long short-term memory.Neural computation, 9(8):1735–1780, 1997

    Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory.Neural computation, 9(8):1735–1780, 1997. 2

    1997

  22. [22]

    3d-llm: In- jecting the 3d world into large language models.Advances in Neural Information Processing Systems, 36:20482–20494,

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: In- jecting the 3d world into large language models.Advances in Neural Information Processing Systems, 36:20482–20494,

  23. [23]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022. 5, 13 9

    2022

  24. [24]

    3dllm-mem: Long-term spatial-temporal memory for embodied 3d large language model.arXiv preprint arXiv:2505.22657, 2025

    Wenbo Hu, Yining Hong, Yanjun Wang, Leison Gao, Zibu Wei, Xingcheng Yao, Nanyun Peng, Yonatan Bitton, Idan Szpektor, and Kai-Wei Chang. 3dllm-mem: Long-term spatial-temporal memory for embodied 3d large language model.arXiv preprint arXiv:2505.22657, 2025. 2, 3

    arXiv 2025

  25. [25]

    Chat-3d v2: Bridging 3d scene and large language models with object identifiers.CoRR, 2023

    Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers.CoRR, 2023. 1, 2, 7

    2023

  26. [26]

    Chat-scene: Bridging 3d scene and large language models with object identifiers

    Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. Advances in Neural Information Processing Systems, 37: 113991–114017, 2024. 2, 7, 14

    2024

  27. [27]

    An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023. 2

    Pith/arXiv arXiv 2023

  28. [28]

    Memory forcing: Spatio-temporal memory for consistent scene generation on minecraft.arXiv preprint arXiv:2510.03198, 2025

    Junchao Huang, Xinting Hu, Boyao Han, Shaoshuai Shi, Zhuotao Tian, Tianyu He, and Li Jiang. Memory forcing: Spatio-temporal memory for consistent scene generation on minecraft.arXiv preprint arXiv:2510.03198, 2025. 3

    arXiv 2025

  29. [29]

    Mllms need 3d-aware representation supervision for scene under- standing.arXiv preprint arXiv:2506.01946, 2025

    Xiaohu Huang, Jingjing Wu, Qunyi Xie, and Kai Han. Mllms need 3d-aware representation supervision for scene under- standing.arXiv preprint arXiv:2506.01946, 2025. 1, 3, 7, 13, 14

    arXiv 2025

  30. [30]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 1, 6

    Pith/arXiv arXiv 2024

  31. [31]

    Goat-bench: A benchmark for multi-modal lifelong navigation

    Mukul Khanna, Ram Ramrakhya, Gunjan Chhablani, Sriram Yenamandra, Theophile Gervet, Matthew Chang, Zsolt Kira, Devendra Singh Chaplot, Dhruv Batra, and Roozbeh Mot- taghi. Goat-bench: A benchmark for multi-modal lifelong navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16373– 16383, 2024. 3

    2024

  32. [32]

    A path towards autonomous machine intelli- gence version 0.9

    Yann LeCun. A path towards autonomous machine intelli- gence version 0.9. 2, 2022-06-27.Open Review, 62(1):1–62,

    2022

  33. [33]

    LLaV A-onevision: Easy visual task transfer.TMLR, 2025

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, and Chunyuan Li. LLaV A-onevision: Easy visual task transfer.TMLR, 2025. 1, 2, 15

    2025

  34. [34]

    Guibiao Liao, Kaichen Zhou, Zhenyu Bao, Kanglin Liu, and Qing Li. Ov-nerf: Open-vocabulary neural radiance fields with vision and language foundation models for 3d semantic understanding.IEEE Transactions on Circuits and Systems for Video Technology, 2024. 1

    2024

  35. [35]

    Clip-gs: Clip-informed gaussian splat- ting for view-consistent 3d indoor semantic understanding

    Guibiao Liao, Jiankun Li, Zhenyu Bao, Xiaoqing Ye, Qing Li, and Kanglin Liu. Clip-gs: Clip-informed gaussian splat- ting for view-consistent 3d indoor semantic understanding. ACM Transactions on Multimedia Computing, Communica- tions and Applications, 21(8):1–24, 2025. 1

    2025

  36. [36]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for visual language models. InCVPR, 2024. 1, 15

    2024

  37. [37]

    Ssr: Enhancing depth perception in vision-language mod- els via rationale-guided spatial reasoning.arXiv preprint arXiv:2505.12448, 2025

    Yang Liu, Ming Ma, Xiaomin Yu, Pengxiang Ding, Han Zhao, Mingyang Sun, Siteng Huang, and Donglin Wang. Ssr: Enhancing depth perception in vision-language mod- els via rationale-guided spatial reasoning.arXiv preprint arXiv:2505.12448, 2025. 1, 2

    arXiv 2025

  38. [38]

    Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution.arXiv preprint arXiv:2409.12961, 2024

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Ji- wen Lu, and Yongming Rao. Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution.arXiv preprint arXiv:2409.12961, 2024. 7, 14

    Pith/arXiv arXiv 2024

  39. [39]

    3dsrbench: A comprehensive 3d spatial reasoning benchmark

    Wufei Ma, Haoyu Chen, Guofeng Zhang, Yu-Cheng Chou, Jieneng Chen, Celso de Melo, and Alan Yuille. 3dsrbench: A comprehensive 3d spatial reasoning benchmark. InProceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 6924–6934, 2025. 3

    2025

  40. [40]

    Sqa3d: Situated question answering in 3d scenes.arXiv preprint arXiv:2210.07474, 2022

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yi- tao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes.arXiv preprint arXiv:2210.07474, 2022. 2, 7, 13, 14

    Pith/arXiv arXiv 2022

  41. [41]

    Gpt-4o, 2024

    OpenAI. Gpt-4o, 2024. 2

    2024

  42. [42]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. InSC20: International Confer- ence for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. 13

    2020

  43. [43]

    Language embedded 3d gaussians for open- vocabulary scene understanding

    Jin-Chuan Shi, Miao Wang, Hao-Bin Duan, and Shao- Hua Guan. Language embedded 3d gaussians for open- vocabulary scene understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5333–5343, 2024. 1

    2024

  44. [44]

    Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics

    Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics. InProceedings of the Computer Vision and Pat- tern Recognition Conference, pages 15768–15780, 2025. 3

    2025

  45. [45]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18221–18232, 2024. 3

    2024

  46. [46]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    Qwen Team et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024. 5

    Pith/arXiv arXiv 2024

  47. [47]

    3d reconstruction with spatial memory.arXiv preprint arXiv:2408.16061, 2024

    Hengyi Wang and Lourdes Agapito. 3d reconstruction with spatial memory.arXiv preprint arXiv:2408.16061, 2024. 3

    Pith/arXiv arXiv 2024

  48. [48]

    Ross3d: Recon- structive visual instruction tuning with 3d-awareness.arXiv preprint arXiv:2504.01901, 2025

    Haochen Wang, Yucheng Zhao, Tiancai Wang, Haoqiang Fan, Xiangyu Zhang, and Zhaoxiang Zhang. Ross3d: Recon- structive visual instruction tuning with 3d-awareness.arXiv preprint arXiv:2504.01901, 2025. 1, 2, 7, 13, 14

    Pith/arXiv arXiv 2025

  49. [49]

    Vggt: Vi- sual geometry grounded transformer

    Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Vi- sual geometry grounded transformer. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 5294–5306, 2025. 2, 3, 8, 15 10

    2025

  50. [50]

    Continuous 3d per- ception model with persistent state

    Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A Efros, and Angjoo Kanazawa. Continuous 3d per- ception model with persistent state. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 10510–10522, 2025. 3, 5, 8, 15

    2025

  51. [51]

    Dust3r: Geometric 3d vi- sion made easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vi- sion made easy. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20697– 20709, 2024. 2

    2024

  52. [52]

    Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiangmiao Pang, Chunhua Shen, and Tong He.π 3: Scalable permutation-equivariant visual geometry learning.arXiv preprint arXiv:2507.13347,

    Pith/arXiv arXiv

  53. [53]

    Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes.arXiv preprint arXiv:2308.08769, 2023

    Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes.arXiv preprint arXiv:2308.08769, 2023. 2

    Pith/arXiv arXiv 2023

  54. [54]

    Spatial-mllm: Boosting mllm capabilities in visual-based spatial intelligence.arXiv preprint arXiv:2505.23747, 2025

    Diankun Wu, Fangfu Liu, Yi-Hsin Hung, and Yueqi Duan. Spatial-mllm: Boosting mllm capabilities in visual-based spatial intelligence.arXiv preprint arXiv:2505.23747, 2025. 2, 3, 6, 7, 13, 14, 15

    Pith/arXiv arXiv 2025

  55. [55]

    Video world models with long-term spatial memory.arXiv preprint arXiv:2506.05284,

    Tong Wu, Shuai Yang, Ryan Po, Yinghao Xu, Ziwei Liu, Dahua Lin, and Gordon Wetzstein. Video world models with long-term spatial memory.arXiv preprint arXiv:2506.05284,

    Pith/arXiv arXiv

  56. [56]

    Point3r: Streaming 3d reconstruction with explicit spatial pointer memory.arXiv preprint arXiv:2507.02863, 2025

    Yuqi Wu, Wenzhao Zheng, Jie Zhou, and Jiwen Lu. Point3r: Streaming 3d reconstruction with explicit spatial pointer memory.arXiv preprint arXiv:2507.02863, 2025. 3

    arXiv 2025

  57. [57]

    Learning 3d dynamic scene representations for robot manip- ulation.arXiv preprint arXiv:2011.01968, 2020

    Zhenjia Xu, Zhanpeng He, Jiajun Wu, and Shuran Song. Learning 3d dynamic scene representations for robot manip- ulation.arXiv preprint arXiv:2011.01968, 2020. 2

    Pith/arXiv arXiv 2011

  58. [58]

    Longvila: Scaling long-context vi- sual language models for long videos

    Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context vi- sual language models for long videos. InICLR, 2025. 15

    2025

  59. [59]

    Magma: A foundation model for multi- modal ai agents

    Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, et al. Magma: A foundation model for multi- modal ai agents. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 14203–14214, 2025. 3

    2025

  60. [60]

    Thinking in space: How mul- timodal large language models see, remember, and recall spaces

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How mul- timodal large language models see, remember, and recall spaces. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 10632–10643, 2025. 1, 2, 3, 5, 6, 7, 13, 14, 15, 16, 17, 18

    2025

  61. [61]

    Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents.arXiv preprint arXiv:2502.09560, 2025

    Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, et al. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents.arXiv preprint arXiv:2502.09560, 2025. 3

    Pith/arXiv arXiv 2025

  62. [62]

    3d-mem: 3d scene memory for embodied exploration and reasoning

    Yuncong Yang, Han Yang, Jiachen Zhou, Peihao Chen, Hongxin Zhang, Yilun Du, and Chuang Gan. 3d-mem: 3d scene memory for embodied exploration and reasoning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17294–17303, 2025. 2, 3

    2025

  63. [63]

    Scannet++: A high-fidelity dataset of 3d in- door scenes

    Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d in- door scenes. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 12–22, 2023. 5

    2023

  64. [64]

    Context as memory: Scene-consistent interactive long video generation with memory retrieval.arXiv preprint arXiv:2506.03141,

    Jiwen Yu, Jianhong Bai, Yiran Qin, Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Context as memory: Scene-consistent interactive long video generation with memory retrieval.arXiv preprint arXiv:2506.03141,

    Pith/arXiv arXiv

  65. [65]

    Empowering large language models with 3d situation awareness

    Zhihao Yuan, Yibo Peng, Jinke Ren, Yinghong Liao, Ya- tong Han, Chun-Mei Feng, Hengshuang Zhao, Guanbin Li, Shuguang Cui, and Zhen Li. Empowering large language models with 3d situation awareness. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 19435–19445, 2025. 1

    2025

  66. [66]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556– 9567, 2024. 6

    2024

  67. [67]

    Flash-vstream: Memory- based real-time understanding for long video streams.arXiv preprint arXiv:2406.08085, 2024

    Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory- based real-time understanding for long video streams.arXiv preprint arXiv:2406.08085, 2024. 3

    Pith/arXiv arXiv 2024

  68. [68]

    From flatland to space: Teaching vision-language models to perceive and reason in 3d.arXiv preprint arXiv:2503.22976, 2025

    Jiahui Zhang, Yurui Chen, Yanpeng Zhou, Yueming Xu, Ze Huang, Jilin Mei, Junhui Chen, Yu-Jie Yuan, Xinyue Cai, Guowei Huang, et al. From flatland to space: Teaching vision-language models to perceive and reason in 3d.arXiv preprint arXiv:2503.22976, 2025. 6, 7

    arXiv 2025

  69. [69]

    Lmms-eval: Re- ality check on the evaluation of large multimodal models

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Re- ality check on the evaluation of large multimodal models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 881–916, 2025. 13

    2025

  70. [70]

    Long context transfer from language to vision.arXiv:2406.16852, 2024

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision.arXiv:2406.16852, 2024. 15

    Pith/arXiv arXiv 2024

  71. [71]

    Llava- next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava- next: A strong zero-shot video understanding model, 2024. 1, 2, 6, 7, 15

    2024

  72. [72]

    Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024. 2, 3, 5, 7, 13, 14

    Pith/arXiv arXiv 2024

  73. [73]

    Learning from videos for 3d world: Enhancing mllms with 3d vision geometry priors.arXiv preprint arXiv:2505.24625,

    Duo Zheng, Shijia Huang, Yanyang Li, and Liwei Wang. Learning from videos for 3d world: Enhancing mllms with 3d vision geometry priors.arXiv preprint arXiv:2505.24625,

    arXiv

  74. [74]

    Video-3d llm: Learning position-aware video representation for 3d scene 11 understanding

    Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d llm: Learning position-aware video representation for 3d scene 11 understanding. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 8995–9006, 2025. 1, 2, 3, 7, 13, 14

    2025

  75. [75]

    Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024

    Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024. 2, 7, 14

    Pith/arXiv arXiv 2024

  76. [76]

    3d-vista: Pre-trained transformer for 3d vision and text alignment

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2911– 2921, 2023. 7, 14

    2023

  77. [77]

    Move to understand a 3d scene: Bridg- ing visual grounding and exploration for efficient and versa- tile embodied navigation

    Ziyu Zhu, Xilin Wang, Yixuan Li, Zhuofan Zhang, Xiaojian Ma, Yixin Chen, Baoxiong Jia, Wei Liang, Qian Yu, Zhi- dong Deng, et al. Move to understand a 3d scene: Bridg- ing visual grounding and exploration for efficient and versa- tile embodied navigation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 8120– 8132, 2025. 3 ...

    2025