Pith. sign in

REVIEW 3 major objections 6 minor 91 references

A structured pure-text description of every object, box, attribute, and relation can serve as the text equivalent of a 3D scene.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 06:11 UTC pith:U5HKBCEB

load-bearing objection Solid systems paper that defines a harder pure-text 3D captioning task, ships a 15K hybrid benchmark and a decoupled model that beats prior dense captioners and 3D LLMs on its own metric, with real robot and zero-shot detection support; absolute localization remains modest and the LLM-loop evaluation is the main soft spot. the 3 major comments →

arxiv 2607.02908 v1 pith:U5HKBCEB submitted 2026-07-03 cs.CV

Holo-Captioning: Toward the Text Equivalent of 3D Scenes

Kun-Yu Lin , Chengke Bu , Zhenguo Li , Kai Han This is my paper
classification cs.CV
keywords holo-captioning3D scene captioning3D LLMstructured textual descriptioninstance-aware pipelineHoloScanHoloScorepure-text localization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that existing 3D dense captioners and scene-description models stop short of a true text equivalent of a 3D scene: they cover few categories, produce short coarse sentences, and still need external detectors for locations. Holo-captioning is defined as producing one long structured text that lists every entity with its semantic tag, 9-DoF oriented box written in pure numbers, free-form attributes, and free-form relations. The authors supply an engine that builds such captions from multi-view images, a 15K-scene benchmark called HoloScan, an LLM-based metric called HoloScore, and a model called HoloScribe that discovers instances, links related pairs, and generates descriptions element by element without any auxiliary detector. If the claim holds, a single pure-text string becomes a usable, editable, robot-ready representation of an entire indoor scene.

Core claim

HoloScribe is the first 3D LLM that jointly localizes every entity as a pure-text 7/9-DoF box and then generates detailed attribute and relation descriptions, outperforming prior dense captioners and 3D LLM generalists on the new HoloScan benchmark under the four-part HoloScore metric.

What carries the argument

The instance-aware decoupled pipeline of HoloScribe: grounded instance discovery, anchor-aware instance linking on sampled subgraphs, and then grounded description generation conditioned on the discovered boxes.

Load-bearing premise

That the auto-generated training captions and the small human-curated test set of 83 scenes are accurate enough that large gains on the LLM-based HoloScore prove genuine 3D-text alignment.

What would settle it

Run the same HoloScribe pipeline on a fresh set of real indoor scans whose boxes, attributes and relations have been exhaustively re-annotated by humans, then check whether the HoloScore ranking and absolute gaps versus tuned SpatialLM and LL3DA remain essentially unchanged.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single pure-text holo-caption can be fed to an off-the-shelf 3D asset generator to reconstruct or edit an entire scene instance-by-instance.
  • Robots can navigate and manipulate objects by parsing only the text boxes and relations, without a separate detector.
  • Zero-shot detection on unseen real datasets becomes possible simply by extracting tags and boxes from the generated caption.
  • Future 3D LLMs can be trained to emit complete scene text rather than short captions or detector-dependent graphs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Pure-text 9-DoF localization may become a harder, more revealing benchmark for 3D-text alignment than traditional 3D detection mAP.
  • The same instance-centric decomposition could be applied to outdoor or dynamic scenes once suitable multi-view engines exist.
  • If HoloScore correlates with human judgment, other long-form 3D description tasks may adopt similar LLM-matching metrics instead of n-gram scores.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces holo-captioning: generating a structured pure-text description of a 3D scene that covers all entity instances along four dimensions—semantic tags, 3D oriented bounding-box locations, free-form attributes, and inter-entity relations—without auxiliary detectors or segmenters. It contributes HoloEngine (multi-view MLLM + LLM aggregation) to produce instance-centric captions, the HoloScan benchmark (~15K hybrid indoor scenes, 83 human-curated test scenes), the LLM-based HoloScore metric (four F-scores via grounded matching, granular descriptor extraction, and dual descriptor matching), and HoloScribe, a SpatialLM-initialized 3D LLM that uses an instance-aware decoupled pipeline (grounded discovery → anchor-aware subgraph linking → grounded attribute/relation generation). On HoloScan, HoloScribe reports large gains over dense captioners and tuned 3D LLMs (overall HoloScore ~116–117 vs. ~60–71); supporting evidence includes ablations (Table 4), zero-shot MultiScan detection (Table 1), scene reconstruction/editing, and a real-robot open-vocabulary navigation success rate of 75% vs. 25% for SpatialLM.

Significance. If the claims hold under independent scrutiny, the work is a meaningful step toward pure-text 3D–language alignment: it is, to the authors’ knowledge, the first 3D LLM that jointly emits entity locations as text boxes and detailed attribute/relation descriptions without an external detector. The large multi-source HoloScan resource, the explicit four-dimension task formulation, the clear ablations isolating decoupling and anchor-aware linking, the MultiScan transfer result, and the real-robot navigation demonstration are concrete strengths that go beyond a purely self-contained benchmark. Even with modest absolute localization scores, the engineering package (engine + benchmark + metric + model + downstream use) is likely to catalyze follow-on work on structured 3D captioning and text-mediated scene editing/robotics.

major comments (3)
  1. The headline superiority claim (Table 3: overall ~116–117 vs. ~60–71) rests primarily on HoloScore, whose components (grounded matching, Gemma3-1B granular extraction, Qwen3-Embedding dual matching; §3.2) operate on free-form attribute/relation text that is stylistically close to the HoloEngine training targets (Gemma3-12B multi-view + Qwen3 aggregation; §4.1). Appendix A4.2 reports only moderate human consistency (PCC 0.60). Relative ranking stability across thresholds is useful but does not rule out distributional/style match inflating relative gains. A load-bearing addition is needed: absolute human ratings (or non-LLM localization/attribute audits) on the 83-scene test set for both HoloScribe and the strongest tuned baselines, reported per dimension, so that the magnitude of the claimed advance can be judged independently of the LLM loop.
  2. Formulation (§3.1) and abstract claim pure-text 9-DoF oriented boxes, yet §5 states that training uses 7-DoF boxes with rx = ry = 0 recomputed from point clouds “to simplify learning.” Table 3 localization F-scores remain modest (~22.3 val / ~22.7 test). The paper itself notes pure-text localization is highly challenging (§7). The manuscript should (i) quantify the accuracy cost of the 7-DoF restriction on the human test set, (ii) report 9-DoF vs. 7-DoF numbers if any 9-DoF runs exist, and (iii) temper claims of “text equivalent” / “precise spatial coordinates in pure text” so they match the implemented representation and the absolute localization level actually achieved.
  3. The human-curated test set comprises only 83 scenes (§4.2). Relation F-scores are low in absolute terms (8.74 test / 11.08 val) and are the dimension most sensitive to pair sampling and free-form phrasing. With such a small N, confidence intervals or bootstrap estimates on the four HoloScore components (and on the MultiScan transfer numbers in Table 1) are needed to establish that the large gaps vs. SpatialLM-Tuned / LL3DA-Tuned are stable. Without them, the central “significantly outperforms” claim is only weakly supported for the relation dimension and for cross-dataset localization.
minor comments (6)
  1. Table 2 “Word” column is ambiguous (average words per caption? per instance?); clarify units and how multi-instance scenes are aggregated.
  2. Fig. 1 and Fig. 3 are helpful but the full structured output format is only fully shown in the appendix JSON example; a short main-text schema box would improve readability.
  3. Implementation details scatter coefficients (δ_tag, δ_loc, δ_attr, δ_rel, α, n_g, angular interval) across §3.2, §5, and §6; a single hyperparameter table would help reproducibility.
  4. Several citation/author-list typos and spacing artifacts appear in the compiled text (e.g., “Thisworkintroduces”, “Tosupportcomprehensive”); a careful copy-edit pass is needed.
  5. Robot navigation (Appendix A4.3) reports 75% vs. 25% over four scenes × 10 trials; state the exact success criterion and whether failures are localization vs. planning vs. control.
  6. Related-work discussion of 3D scene graphs and panoptic captioning is useful; a short explicit contrast table (what is predicted purely in text vs. what requires auxiliary models) would sharpen novelty claims.

Circularity Check

0 steps flagged

No derivation-chain circularity; mild self-reference from author-introduced data/metric is mitigated by external baselines, MultiScan zero-shot, robot deployment, and human-curated test set.

full rationale

This is an empirical systems paper introducing a task (holo-captioning), data engine (HoloEngine), benchmark (HoloScan), model (HoloScribe), and metric (HoloScore). There is no mathematical derivation, uniqueness theorem, fitted-parameter-as-prediction, or ansatz smuggled via self-citation that reduces a claimed result to its inputs by construction. The performance claims are experimental comparisons against independent baselines (Vote2Cap-DETR, LL3DA, SpatialLM) on both the authors' validation/test splits and an unseen MultiScan detection set, plus a real-robot navigation success-rate experiment. The sole self-citation (prior panoptic captioning) appears only in related work as a complementary 2D precursor and is not load-bearing for any quantitative claim. Human curation of the 83-scene test set and a reported human-consistency analysis for HoloScore further break any pure auto-LLM loop. Absolute localization scores remain modest, but that is a correctness/validity concern, not circularity of derivation. Score 1 reflects only the ordinary mild self-reference of any paper that both creates and evaluates on its own benchmark; the central claims do not reduce by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 3 invented entities

The central empirical claim rests on standard 3D-LLM assumptions plus several paper-specific design choices (instance decoupling, anchor subgraphs, LLM-based scoring thresholds, auto-generated training captions). No new physical entities are postulated; free parameters are the usual training hyper-parameters and metric cut-offs.

free parameters (3)
  • HoloScore thresholds (δ_tag=0.5, δ_loc=0.3, δ_attr=δ_rel=0.6) and α=10
    Chosen by preliminary human-consistency experiments; absolute scores change with thresholds although rankings stay stable.
  • LoRA rank r=64, α=128, learning rate 5e-4, batch size 12, 1 epoch
    Standard finetuning hyper-parameters that affect final HoloScore numbers.
  • Subgraph size n_g and angular sampling interval (e.g. 30°)
    Design choices that control which relations are considered and how multi-view captions are aggregated.
axioms (4)
  • domain assumption A 3D indoor scene can be adequately represented by a finite set of oriented bounding boxes plus free-form attribute and pairwise relation text.
    Stated in §3.1 task formulation; underpins the entire claim of ‘text equivalence’.
  • ad hoc to paper MLLM-generated multi-view captions aggregated by an LLM (HoloEngine) are sufficiently accurate to serve as training targets and, after light human curation, as evaluation ground truth.
    Core of data construction (§4); reliability argued via pilot experiments and human-consistency of validation vs test trends.
  • domain assumption 7-DoF boxes (rx=ry=0) are an acceptable simplification of 9-DoF boxes for training.
    Explicitly adopted in grounded instance discovery (§5) to simplify learning.
  • domain assumption Meaningful inter-entity relations occur predominantly among spatially proximate instances.
    Used to restrict relation candidates in both HoloEngine and anchor-aware linking.
invented entities (3)
  • Holo-caption (structured four-dimension pure-text scene description) no independent evidence
    purpose: Defines the target output of the new task.
    New task object; no independent physical existence outside the paper’s formulation.
  • HoloScore (LLM-based four-F-score metric) no independent evidence
    purpose: Evaluate long structured captions where n-gram metrics fail.
    New evaluation construct whose absolute values depend on chosen LLM judges and thresholds.
  • Anchor-aware instance linking no independent evidence
    purpose: Efficiently identify relational pairs inside sampled subgraphs.
    Algorithmic device introduced by the paper; effectiveness shown only via ablation on HoloScan.

pith-pipeline@v1.1.0-grok45 · 30406 in / 3280 out tokens · 26822 ms · 2026-07-12T06:11:49.917849+00:00 · methodology

0 comments
read the original abstract

This work introduces holo-captioning, a novel task that strives to seek the text equivalent of 3D scenes. As the initial step, we formulate holo-captioning as generating a structured textual description that comprehensively depicts all entities within a 3D scene -- including their semantic tags, spatial locations, attributes, and inter-entity relations. To tackle this challenging task, we first develop an effective captioning engine to produce detailed descriptions of individual entity instances and instance pairs, and contribute a large-scale benchmark comprising over 15K scenes for training and evaluation. Building upon this foundation, we propose HoloScribe, a novel model that features an instance-aware decoupled pipeline for generating structured holo-captions, and further incorporates anchor-aware instance linking to identify relational instance pairs. Additionally, we propose a comprehensive evaluation metric named HoloScore, and provide a human-curated test set to ensure reliable model assessment. Experimental results demonstrate that HoloScribe significantly outperforms state-of-the-art 3D dense captioners and 3D LLM generalists, underscoring the effectiveness of our approach. Project page: https://visual-ai.github.io/holocap/

Figures

Figures reproduced from arXiv: 2607.02908 by Chengke Bu, Kai Han, Kun-Yu Lin, Zhenguo Li.

Figure 1
Figure 1. Figure 1: An example to demonstrate our proposed holo-captioning task, which is for￾mulated as generating a comprehensive structured textual description for a 3D scene. A holo-caption comprehensively depicts all entity instances within a 3D scene, includ￾ing their respective semantic tags, spatial locations and attributes, as well as relations between entity instances. Best viewed in color. relations. These omission… view at source ↗
Figure 2
Figure 2. Figure 2: An overview of our HoloScore metric. HoloScore first groups holo-caption el￾ements into three aspects: semantic tags with spatial locations, entity attributes and inter-entity relations. It then performs grounded instance matching to assess tagging and localization, applies granular descriptor extraction to process attributes, and per￾forms dual descriptor matching to assess attribute and relation dimensio… view at source ↗
Figure 3
Figure 3. Figure 3: An overview of our HoloScribe. Given an input 3D scene, HoloScribe generates holo-captions following an instance-aware decoupling pipeline, consisting of grounded instance discovery, anchor-aware instance linking, and grounded description generation. connector, and a decoder-only LLM. In each phase, we employ a specific prompt to guide HoloScribe in producing desired outputs in pure text form. Grounded Ins… view at source ↗
Figure 4
Figure 4. Figure 4: Demonstration of (a) a real scene from HoloScan, (b) the HoloScribe-based reconstruction, and (c) LL3DA-based reconstruction. We highlight entities by 3D boxes. attribute and relation prediction mechanism (“w/ instance-aware prediction”), the overall performance of HoloScribe improves significantly, where relation pre￾diction is performed for every instance pair. By further incorporating our anchor￾aware l… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

91 extracted references · 16 linked inside Pith

  1. [1]

    In: European Conference on Computer Vision (2020) 4, 5, 6, 10

    Achlioptas, P., Abdelreheem, A., Xia, F., Elhoseiny, M., Guibas, L.J.: ReferIt3D: Neural listeners for fine-grained 3D object identification in real-world scenes. In: European Conference on Computer Vision (2020) 4, 5, 6, 10

    2020

  2. [2]

    In: IEEE/CVF International Conference on Computer Vision (2019) 4

    Armeni, I., He, Z., Zamir, A., Gwak, J., Malik, J., Fischer, M., Savarese, S.: 3D Scene Graph: A structure for unified semantics, 3D space, and camera. In: IEEE/CVF International Conference on Computer Vision (2019) 4

    2019

  3. [3]

    In: European Conference on Computer Vision (2024) 1, 2, 5, 6, 10

    Avetisyan, A., Xie, C., Howard-Jenkins, H., Yang, T., Aroudj, S., Patra, S., Zhang, F., Frost, D.P., Holland, L., Orme, C., Engel, J.J., Miller, E., Newcombe, R.A., Balntas, V.: SceneScript: Reconstructing scenes with an autoregressive structured language model. In: European Conference on Computer Vision (2024) 1, 2, 5, 6, 10

    2024

  4. [4]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022) 5

    Azuma, D., Miyanishi, T., Kurita, S., Kawanabe, M.: ScanQA: 3D question an- swering for spatial scene understanding. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022) 5

    2022

  5. [5]

    arXiv preprint arXiv:2510.10903 (2025) 1, 5

    Bai, S., Song, W., Chen, J., Ji, Y., Zhong, Z., Yang, J., Zhao, H., Zhou, W., Zhao, W., Li, Z., Ding, P., Chi, C., Li, H., Xu, C., Zheng, X., Wang, D., Zhang, S., Chen,B.:Towardsaunifiedunderstandingofrobotmanipulation:Acomprehensive survey. arXiv preprint arXiv:2510.10903 (2025) 1, 5

    arXiv 2025

  6. [6]

    In: Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005 (2005) 6

    Banerjee, S., Lavie, A.: METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005 (2005) 6

    2005

  7. [7]

    arXiv preprint arXiv:2410.24164 (2024) 1, 5

    Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., Jakubczak, S., Jones, T., Ke, L., Levine, S., Li-Bell, A., Mothukuri, M., Nair, S., Pertsch, K., Shi, L.X., Tanner, J., Vuong, Q., Walling, A., Wang, H., Zhilinsky, U.:π0: A vision-language-action flow model for general robot control. arXiv pre...

    Pith/arXiv arXiv 2024

  8. [8]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023) 5

    Brazil, G., Kumar, A., Straub, J., Ravi, N., Johnson, J., Gkioxari, G.: Omni3D: A large benchmark and model for 3D object detection in the wild. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023) 5

    2023

  9. [9]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Cai, D., Zhao, L., Zhang, J., Sheng, L., Xu, D.: 3DJCG: A unified framework for joint dense captioning and visual grounding on 3D point clouds. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16464–16473 (2022) 4

    2022

  10. [10]

    arXiv preprint arXiv:2008.09403 (2020) 21

    Campari, T., Eccher, P., Serafini, L., Ballan, L.: Exploiting scene-specific features for object goal navigation. arXiv preprint arXiv:2008.09403 (2020) 21

    Pith/arXiv arXiv 2008

  11. [11]

    In: International Conference on 3D Vision (2017) 9

    Chang, A., Dai, A., Funkhouser, T., Halber, M., Nießner, M., Savva, M., Song, S., Zeng, A., Zhang, Y.: Matterport3D: Learning from rgb-d data in indoor environ- ments. In: International Conference on 3D Vision (2017) 9

    2017

  12. [12]

    In: Conference on Robot Learning (2023) 6

    Chang, H., Boyalakuntla, K., Lu, S., Cai, S., Jing, E.P., Keskar, S., Geng, S., Abbas, A., Zhou, L., Bekris, K.E., Boularias, A.: Context-aware entity grounding with open-vocabulary 3D scene graphs. In: Conference on Robot Learning (2023) 6

    2023

  13. [13]

    In: European Conference on Computer Vision (2020) 4, 5, 6, 10, 13

    Chen, D.Z., Chang, A.X., Nießner, M.: ScanRefer: 3D object localization in RGB-D scans using natural language. In: European Conference on Computer Vision (2020) 4, 5, 6, 10, 13

    2020

  14. [14]

    In: IEEE Conference on Computer Vision and Pattern Recognition (2021) 1, 2, 4, 5, 6, 16 26 K.-Y

    Chen, D.Z., Gholami, A., Nießner, M., Chang, A.X.: Scan2Cap: Context-aware dense captioning in RGB-D scans. In: IEEE Conference on Computer Vision and Pattern Recognition (2021) 1, 2, 4, 5, 6, 16 26 K.-Y. Lin et al

    2021

  15. [15]

    In: European Confer- ence on Computer Vision

    Chen, D.Z., Wu, Q., Nießner, M., Chang, A.X.: D3Net: A unified speaker-listener architecture for 3D dense captioning and visual grounding. In: European Confer- ence on Computer Vision. Springer-Verlag, Berlin, Heidelberg (2022) 4

    2022

  16. [16]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024) 3, 5, 12, 13, 20, 23

    Chen, S., Chen, X., Zhang, C., Li, M., Yu, G., Fei, H., Zhu, H., Fan, J., Chen, T.: LL3DA: visual interactive instruction tuning for omni-3D understanding, reason- ing, and planning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024) 3, 5, 12, 13, 20, 23

    2024

  17. [17]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023) 1, 2, 4, 5, 6, 12, 13, 20

    Chen, S., Zhu, H., Chen, X., Lei, Y., Yu, G., Chen, T.: End-to-end 3D dense captioning with vote2cap-detr. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023) 1, 2, 4, 5, 6, 12, 13, 20

    2023

  18. [18]

    IEEE Transactions on Pattern Analysis and Machine Intelligence 46(2024) 4, 6, 12, 13

    Chen, S., Zhu, H., Li, M., Chen, X., Guo, P., Lei, Y., Yu, G., Li, T., Chen, T.: Vote2Cap-DETR++: Decoupling localization and describing for end-to-end 3D dense captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence 46(2024) 4, 6, 12, 13

    2024

  19. [19]

    In: IEEE Conference on Computer Vision and Pattern Recognition (2017) 9, 10

    Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: Scan- Net: Richly-annotated 3D reconstructions of indoor scenes. In: IEEE Conference on Computer Vision and Pattern Recognition (2017) 9, 10

    2017

  20. [20]

    In: Neural Information Processing Systems Datasets and Benchmarks Track (2021) 5, 9

    Dehghan, A., Baruch, G., Chen, Z., Feigin, Y., Fu, P., Gebauer, T., Kurz, D., Dimry, T., Joffe, B., Schwartz, A., Shulman, E.: ARKitScenes: A diverse real- world dataset for 3D indoor scene understanding using mobile RGB-D data. In: Neural Information Processing Systems Datasets and Benchmarks Track (2021) 5, 9

    2021

  21. [21]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2025) 5

    Deng, J., He, T., Jiang, L., Wang, T., Dayoub, F., Reid, I.D.: 3D-LLaVA: Towards generalist 3D lmms with omni superpoint transformer. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2025) 5

    2025

  22. [22]

    arXiv preprint arXiv:2405.19092 (2024) 6, 20

    Dong, H., Li, J., Wu, B., Wang, J., Zhang, Y., Guo, H.: Benchmarking and im- proving detail image caption. arXiv preprint arXiv:2405.19092 (2024) 6, 20

    Pith/arXiv arXiv 2024

  23. [23]

    arXiv preprint arXiv:2407.21783 (2024) 4

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024) 4

    Pith/arXiv arXiv 2024

  24. [24]

    arXiv preprint arXiv:2509.14981 (2025) 1

    Fang, C., Li, H., Liang, Y., Zheng, J., Mao, Y., Liu, Y., Tang, R., Zhou, Z., Tan, P.: SPATIALGEN: Layout-guided 3D indoor scene generation. arXiv preprint arXiv:2509.14981 (2025) 1

    arXiv 2025

  25. [25]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024) 4

    Ge, Y., Zeng, X., Huffman, J.S., Lin, T., Liu, M., Cui, Y.: Visual Fact Checker: Enabling high-fidelity detailed caption generation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024) 4

    2024

  26. [26]

    In: IEEE International Conference on Robotics and Automation (2024) 6

    Gu, Q., Kuwajerwala, A., Morin, S., Jatavallabhula, K.M., Sen, B., Agarwal, A., Rivera, C., Paul, W., Ellis, K., Chellappa, R., Gan, C., de Melo, C.M., Tenenbaum, J.B., Torralba, A., Shkurti, F., Paull, L.: ConceptGraphs: Open-vocabulary 3D scene graphs for perception and planning. In: IEEE International Conference on Robotics and Automation (2024) 6

    2024

  27. [27]

    Guo, K., Huang, Y., Jia, T., Song, X., Sun, S., Wei, H., Han, X.F., Huang, S., Strisciuglio, N., Li, S.: Visual Grounding in 2D and 3D: A unified perspective and survey. Inf. Fusion (2026) 4

    2026

  28. [28]

    In: Neural Information Processing Systems (2023) 5

    Hong, Y., Zhen, H., Chen, P., Zheng, S., Du, Y., Chen, Z., Gan, C.: 3D-LLM: In- jecting the 3D world into large language models. In: Neural Information Processing Systems (2023) 5

    2023

  29. [29]

    In: Neural Information Processing Systems (2024) 5 Holo-Captioning 27

    Huang, H., Chen, Y., Wang, Z., Huang, R., Xu, R., Wang, T., Liu, L., Cheng, X., Zhao, Y., Pang, J., Zhao, Z.: Chat-Scene: Bridging 3D scene and large language models with object identifiers. In: Neural Information Processing Systems (2024) 5 Holo-Captioning 27

    2024

  30. [30]

    In: International Conference on Machine Learning (2024) 5, 12, 13, 23

    Huang, J., Yong, S., Ma, X., Linghu, X., Li, P., Wang, Y., Li, Q., Zhu, S., Jia, B., Huang, S.: An embodied generalist agent in 3D world. In: International Conference on Machine Learning (2024) 5, 12, 13, 23

    2024

  31. [31]

    arXiv preprint arXiv:2507.07781 (2025) 5

    Huang, J., Li, Z., Zhang, H., Chen, R., He, X., Guo, Y., Wang, W., Liu, T., Gong, M.: SURPRISE3D: A dataset for spatial understanding and reasoning in complex 3D scenes. arXiv preprint arXiv:2507.07781 (2025) 5

    Pith/arXiv arXiv 2025

  32. [32]

    arXiv preprint arXiv:2507.23478 (2025) 5

    Huang, T., Zhang, Z., Tang, H.: 3D-R1: Enhancing reasoning in 3D VLMs for unified scene understanding. arXiv preprint arXiv:2507.23478 (2025) 5

    Pith/arXiv arXiv 2025

  33. [33]

    In: IEEE Conference on Computer Vision and Pattern Recognition (2026) 5

    Huang, W.J., Zhang, Y.Y., Wei, Y.L., Xia, Z.W., Tan, J., Li, Y.M., Zhao, Z., Zheng, W.S.: Beyond Mimicry: Learning whole-body human-humanoid interaction from human-human demonstrations. In: IEEE Conference on Computer Vision and Pattern Recognition (2026) 5

    2026

  34. [34]

    In: Neural Information Processing Systems (2025) 5

    Huang, X., Wu, J., Xie, Q., Han, K.: 3DRS: MLLMs Need 3D-Aware Represen- tation Supervision for Scene Understanding. In: Neural Information Processing Systems (2025) 5

    2025

  35. [35]

    In: European Conference on Computer Vision (2024) 5, 9

    Jia, B., Chen, Y., Yu, H., Wang, Y., Niu, X., Liu, T., Li, Q., Huang, S.: Scen- eVerse: Scaling 3D vision-language learning for grounded scene understanding. In: European Conference on Computer Vision (2024) 5, 9

    2024

  36. [36]

    In: IEEE/CVF International Conference on Computer Vision (2025) 5

    Jiang, J., Wu, X., He, Y., Zeng, L., Wei, Y., Zhang, D., Zheng, W.: Rethinking bimanual robotic manipulation: Learning with decoupled interaction framework. In: IEEE/CVF International Conference on Computer Vision (2025) 5

    2025

  37. [37]

    In: European Conference on Computer Vision

    Jiao, Y., Chen, S., Jie, Z., Chen, J., Ma, L., Jiang, Y.G.: MORE: Multi-order relation mining for dense captioning in 3D scenes. In: European Conference on Computer Vision. pp. 528–545. Springer Nature Switzerland, Cham (2022) 4

    2022

  38. [38]

    arXiv preprint arXiv:2503.19786 (2025) 13

    Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., et al.: Gemma 3 technical report. arXiv preprint arXiv:2503.19786 (2025) 13

    Pith/arXiv arXiv 2025

  39. [39]

    In: Findings of the Association for Computational Linguistics (2024) 4

    Kim, M., Lim, H., Kim, S.H., Lee, S., Kim, B., Kim, G.: See It All: Contextual- ized late aggregation for 3D dense captioning. In: Findings of the Association for Computational Linguistics (2024) 4

    2024

  40. [40]

    Koch, S., Vaskevicius, N., Colosi, M., Hermosilla, P., Ropinski, T.: Open3DSG: Open-vocabulary 3D scene graphs from point clouds with queryable objects and open-setrelationships.In:IEEE/CVFConferenceonComputerVisionandPattern Recognition (2024) 4

    2024

  41. [41]

    arXiv preprint arXiv:2509.16721 (2025) 5

    Li, H., Liu, R., Fan, H., Yang, Y.: Text-scene: A scene-to-language parsing frame- work for 3D scene understanding. arXiv preprint arXiv:2509.16721 (2025) 5

    arXiv 2025

  42. [42]

    In: European Conference on Computer Vision

    Li, Y.M., Huang, W.J., Wang, A.L., Zeng, L.A., Meng, J.K., Zheng, W.S.: EgoExo- Fitness: Towards egocentric and exocentric full-body action understanding. In: European Conference on Computer Vision. Springer (2024) 5

    2024

  43. [43]

    arXiv preprint arXiv:2209.07753 (2022) 23

    Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., Zeng, A.: Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753 (2022) 23

    Pith/arXiv arXiv 2022

  44. [44]

    In: Neural Information Processing Systems (2025) 4

    Lin, K.Y., Wang, H., Ren, W., Han, K.: Panoptic Captioning: An equivalence bridge for image and text. In: Neural Information Processing Systems (2025) 4

    2025

  45. [45]

    In: Neural Information Processing Systems Datasets and Benchmarks Track (2024) 5

    Linghu, X., Huang, J., Niu, X., Ma, X., Jia, B., Huang, S.: Multi-modal situated reasoning in 3D scenes. In: Neural Information Processing Systems Datasets and Benchmarks Track (2024) 5

    2024

  46. [46]

    In: Neural Information Processing Systems (2023) 4

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: Neural Information Processing Systems (2023) 4

    2023

  47. [47]

    In: European Conference on Computer Vision (2024) 4 28 K.-Y

    Luo,T.,Johnson,J.,Lee,H.:Viewselectionfor3Dcaptioningviadiffusionranking. In: European Conference on Computer Vision (2024) 4 28 K.-Y. Lin et al

    2024

  48. [48]

    In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S

    Luo, T., Rockwell, C., Lee, H., Johnson, J.: Scalable 3D captioning with pretrained models. In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Neural Information Processing Systems (2023) 4

    2023

  49. [49]

    In: Neural Information Processing Systems (2024) 5, 9

    Lyu, R., Lin, J., Wang, T., Yang, S., Mao, X., Chen, Y., Xu, R., Huang, H., Zhu, C., Lin, D., Pang, J.: MMScan: A multi-modal 3D scene dataset with hierarchical grounded language annotations. In: Neural Information Processing Systems (2024) 5, 9

    2024

  50. [50]

    In: International Conference on Learning Representations (2023) 5

    Ma, X., Yong, S., Zheng, Z., Li, Q., Liang, Y., Zhu, S.C., Huang, S.: SQA3D: Situated question answering in 3D scenes. In: International Conference on Learning Representations (2023) 5

    2023

  51. [51]

    In: Neural Information Processing Systems (2022) 3, 14

    Mao, Y., Zhang, Y., Jiang, H., Chang, A.X., Savva, M.: MultiScan: Scalable RGBD scanning for 3D environments with articulated objects. In: Neural Information Processing Systems (2022) 3, 14

    2022

  52. [52]

    In: Neural Information Processing Systems (2025) 1, 2, 3, 5, 6, 10, 13, 18, 19, 20, 23

    Mao, Y., Zhong, J., Fang, C., Zheng, J., Tang, R., Zhu, H., Tan, P., Zhou, Z.: SpatialLM: Training large language models for structured indoor modeling. In: Neural Information Processing Systems (2025) 1, 2, 3, 5, 6, 10, 13, 18, 19, 20, 23

    2025

  53. [53]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2025) 5

    Mei, G., Lin, W., Riz, L., Wu, Y., Poiesi, F., Wang, Y.: PerLA: Perceptive 3D language assistant. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2025) 5

    2025

  54. [54]

    In: Annual Meeting of the Association for Compu- tational Linguistics (2002) 3, 6, 22

    Papineni, K., Roukos, S., Ward, T., Zhu, W.: Bleu: a method for automatic eval- uation of machine translation. In: Annual Meeting of the Association for Compu- tational Linguistics (2002) 3, 6, 22

    2002

  55. [55]

    arXiv preprint arXiv:2501.01428 (2025) 5

    Qi, Z., Zhang, Z., Fang, Y., Wang, J., Zhao, H.: GPT4Scene: Understand 3D scenes from videos with vision-language models. arXiv preprint arXiv:2501.01428 (2025) 5

    Pith/arXiv arXiv 2025

  56. [56]

    arXiv preprint arXiv:2511.21631 (2025) 23

    Qwen Team: Qwen3-VL technical report. arXiv preprint arXiv:2511.21631 (2025) 23

    Pith/arXiv arXiv 2025

  57. [57]

    In: Conference on Robot Learning (2023) 6

    Rana, K., Haviland, J., Garg, S., Abou-Chakra, J., Reid, I.D., Sünderhauf, N.: SayPlan: Grounding large language models using 3D scene graphs for scalable robot task planning. In: Conference on Robot Learning (2023) 6

    2023

  58. [58]

    In: IEEE International Conference on Robotics and Automation (2023) 4

    Schult, J., Engelmann, F., Hermans, A., Litany, O., Tang, S., Leibe, B.: Mask3D: Mask transformer for 3D semantic instance segmentation. In: IEEE International Conference on Robotics and Automation (2023) 4

    2023

  59. [59]

    arXiv preprint arXiv:2506.15442 (2025) 14, 20

    Team Hunyuan3D, Yang, S., Yang, M., Feng, Y., Huang, X., et al.: Hunyuan3D 2.1: From images to high-fidelity 3D assets with production-ready PBR material. arXiv preprint arXiv:2506.15442 (2025) 14, 20

    Pith/arXiv arXiv 2025

  60. [60]

    Journal of Mech- anisms and Robotics15(2), 020801 (2022) 21

    Thakar, S., Srinivasan, S., Al-Hussaini, S., Bhatt, P.M., Rajendran, P., Jung Yoon, Y., Dhanaraj, N., Malhan, R.K., Schmid, M., Krovi, V.N., Gupta, S.K.: A survey of wheeled mobile manipulation: A decision-making perspective. Journal of Mech- anisms and Robotics15(2), 020801 (2022) 21

    2022

  61. [61]

    In: IEEE Conference on Computer Vision and Pattern Recognition (2015) 3, 6, 22

    Vedantam, R., Zitnick, C.L., Parikh, D.: CIDEr: Consensus-based image descrip- tion evaluation. In: IEEE Conference on Computer Vision and Pattern Recognition (2015) 3, 6, 22

    2015

  62. [62]

    In: IEEE/CVF Interna- tional Conference on Computer Vision (2019) 9

    Wald, J., Avetisyan, A., Navab, N., Tombari, F., Niessner, M.: RIO: 3D object instance re-localization in changing indoor environments. In: IEEE/CVF Interna- tional Conference on Computer Vision (2019) 9

    2019

  63. [63]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020) 4 Holo-Captioning 29

    Wald, J., Dhamo, H., Navab, N., Tombari, F.: Learning 3D semantic scene graphs from 3D indoor reconstructions. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020) 4 Holo-Captioning 29

    2020

  64. [64]

    In: AAAI (2025) 4

    Wang, A., Shan, B., Shi, W., Lin, K., Fei, X., Tang, G., Liao, L., Tang, J., Huang, C., Zheng, W.: ParGo: Bridging vision-language with partial and global views. In: AAAI (2025) 4

    2025

  65. [65]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024) 5, 9

    Wang, T., Mao, X., Zhu, C., Xu, R., Lyu, R., Li, P., Chen, X., Zhang, W., Chen, K., Xue, T., Liu, X., Lu, C., Lin, D., Pang, J.: EmbodiedScan: A holistic multi- modal 3D perception suite towards embodied AI. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024) 5, 9

    2024

  66. [66]

    arXiv preprint arXiv:2308.08769 (2023) 5

    Wang, Z., Huang, H., Zhao, Y., Zhang, Z., Zhao, Z.: Chat-3D: Data-efficiently tuning large language model for universal dialogue of 3D scenes. arXiv preprint arXiv:2308.08769 (2023) 5

    Pith/arXiv arXiv 2023

  67. [67]

    In: Neural Information Processing Systems (2024) 5

    Wei, Y.L., Jiang, J.J., Xing, C., Tan, X.T., Wu, X.M., Li, H., Cutkosky, M., Zheng, W.S.: Grasp as you say: Language-guided dexterous grasp generation. In: Neural Information Processing Systems (2024) 5

    2024

  68. [68]

    In: IEEE/CVF International Conference on Computer Vi- sion

    Wei, Y.L., Lin, M., Lin, Y., Jiang, J.J., Wu, X.M., Zeng, L.A., Zheng, W.S.: AffordDexGrasp: Open-set language-guided dexterous grasp with generalizable- instructive affordance. In: IEEE/CVF International Conference on Computer Vi- sion. pp. 11818–11828 (2025) 5

    2025

  69. [69]

    In: Robotics: Science and Systems (2024) 1, 5

    Werby, A., Huang, C., Büchner, M., Valada, A., Burgard, W.: Hierarchical open- vocabulary 3D scene graphs for language-grounded robot navigation. In: Robotics: Science and Systems (2024) 1, 5

    2024

  70. [70]

    In: IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (2025) 4

    Wu, S., Fei, H., Chua, T.: Universal scene graph generation. In: IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (2025) 4

    2025

  71. [71]

    Neural Comput

    Wu, W., Chang, T., Li, X., Yin, Q., Hu, Y.: Vision-language navigation: a survey and taxonomy. Neural Comput. Appl. (2024) 1, 5

    2024

  72. [72]

    In: European Conference on Computer Vi- sion (2024) 5

    Wu, X.M., Cai, J.F., Jiang, J.J., Zheng, D., Wei, Y.L., Zheng, W.S.: An economic framework for 6-DoF grasp detection. In: European Conference on Computer Vi- sion (2024) 5

    2024

  73. [73]

    In: International Conference on Machine Learning (2026) 5

    Wu, X.M., Fan, B., Liao, K., Jiang, J.J., Yang, R., Luo, Y., Wu, Z., Zheng, W.S., Loy, C.C.: VLANeXt: Recipes for building strong VLA models. In: International Conference on Machine Learning (2026) 5

    2026

  74. [74]

    In: IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (2025) 13, 18

    Wu, X., DeTone, D., Frost, D.P., Shen, T., Xie, C., Yang, N., Engel, J.J., New- combe, R.A., Zhao, H., Straub, J.: Sonata: Self-supervised learning of reliable point representations. In: IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (2025) 13, 18

    2025

  75. [75]

    arXiv preprint arXiv:2503.11806 (2025) 1

    Xie, C., Avetisyan, A., Howard-Jenkins, H., Siddiqui, Y., Straub, J., Newcombe, R.A., Balntas, V., Engel, J.J.: Human-in-the-loop local corrections of 3D scene layouts via infilling. arXiv preprint arXiv:2503.11806 (2025) 1

    Pith/arXiv arXiv 2025

  76. [76]

    In: European Conference on Computer Vision (2024) 4

    Yan,J.,Gao,Y.,Yang,Q.,Wei,X.,Xie,X.,Wu,A.,Zheng,W.:DreamView:Inject- ing view-specific text guidance into text-to-3d generation. In: European Conference on Computer Vision (2024) 4

    2024

  77. [77]

    arXiv preprint arXiv:2505.09388 (2025) 13

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025) 13

    Pith/arXiv arXiv 2025

  78. [78]

    arXiv preprint arXiv:2412.15115 (2024) 18

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., et al.: Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 (2024) 18

    Pith/arXiv arXiv 2024

  79. [79]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recog- nition

    Yang, J., Chen, X., Madaan, N., Iyengar, M., Qian, S., Fouhey, D.F., Chai, J.: 3D-GRAND: A million-scale dataset for 3D-llms with better grounding and less hallucination. In: IEEE/CVF Conference on Computer Vision and Pattern Recog- nition. pp. 29501–29512 (2025) 5

    2025

  80. [80]

    In: International Con- ference on Learning Representations (2025) 6 30 K.-Y

    Ye, Q., Zeng, X., Li, F., Li, C., Fan, H.: Painting with Words: Elevating detailed image captioning with benchmark and alignment learning. In: International Con- ference on Learning Representations (2025) 6 30 K.-Y. Lin et al

    2025

Showing first 80 references.