Pith. sign in

REVIEW 5 major objections 8 minor 54 references

Hypo3D: Exploring Hypothetical Reasoning in 3D

T0 review · 5 major / 8 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Hypo3D is a 3D question-answering benchmark that forces models to imagine a described scene change before answering, and the best current models score under 50% where humans exceed 85%.

desk verdict A genuinely new 3D VQA benchmark for hypothetical reasoning, but the headline human-model gap compares numbers from two different test sets and needs rework before the central claim is trusted. read the letter →

arxiv 2502.00954 v3 pith:K6IQ77JW submitted 2025-02-02 cs.CV

Ye Mao , Weixun Luo , Junpeng Jing , Anlan Qiu , Krystian Mikolajczyk This is my paper · ORCID
classification cs.CV
keywords hypotheticalreasoning3Dvisualquestionansweringscenechangesimulationbenchmarkfoundationmodelsspatialdirectionalanchor-basedworldframe
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Hypo3D proposes hypothetical reasoning as a distinct, testable capability: given a static 3D scene, a textual description of a change, and a question, a system must mentally apply the change and answer from the imagined scene rather than from a fresh scan. The paper builds this into 7,727 context changes and 14,885 question-answer pairs over 700 indoor scenes, spanning five change types (movement, removal, attribute, addition, replacement) and an anchor-based world frame that fixes what directional words mean in each scene. On this benchmark, ten state-of-the-art foundation models top out below 50% exact match, while humans reach 91.00%, with the largest shortfalls on movement changes and direction-based questions. The authors argue this exposes a missing capability — mental simulation of unobserved scene states — that existing 3D reasoning benchmarks never test because they supply the current scene itself.

What carries the argument

The load-bearing mechanism is the task tuple, and inside it, the anchor-based world frame. Since a 3D scan has no intrinsic orientation, each scene receives a textual frame statement — for example, 'the desk is located to the left of the scene' — and the right-hand rule derives the remaining directions, so terms like 'left of the laptop' refer to one global coordinate frame instead of the observer's viewpoint or an object's own geometry. The context change $c$ is one of five types (movement, removal, attribute, addition, replacement), and the benchmark forces the model to compute the aligned scene $\tilde{S}$, apply the change mentally, and answer from the imagined updated scene. This combination is what makes directional questions unambiguous while also requiring the model to fill a perceptual gap, the property that distinguishes Hypo3D from prior 3D VQA benchmarks that provide the current scene.

What would settle it

Render a sample of Hypo3D context changes in a 3D editor so the changed scene is actually visible, present the rendered changed scene (not the change description) to the same models with the same questions, and compare exact match against the imagination condition; if scores rise sharply, the reported gap is specifically the inability to simulate unobserved changes, whereas flat scores would mean the benchmark measures general spatial reasoning. A complementary check is to have fresh annotators re-answer a random sample and compare their agreement against the published ground-truth answers.

Watch

Extended reading notes

Core claim

The central claim is that current foundation models cannot reason about a 3D scene they have not observed in its changed state, at anything close to human level. Hypo3D formalizes the task as a tuple $\langle S, F, c, q\rangle$: the scene, an anchor-based world frame that fixes the global orientation, a context change of one of five types, and an open-ended question that is answerable only by combining the original scene with the change. A model must align the scene to the frame, imagine the scene after the change, and then answer. Assembled with human crowdsourcing plus LLM generation and heavy human refinement, the benchmark reports that the best model (GPT-4o on semantic top-view maps) achieves 45.50% exact match overall against 91.00% for humans, with movement and replacement changes hardest and direction-based questions the weakest area. The paper also reports a systematic failure: when a described change cannot affect the answer, models still revise their answers and lose accuracy, showing they do not reliably track which parts of the scene their answer depends on.

Load-bearing premise

The benchmark's validity depends on each question having exactly one correct answer and on the anchor-based frame being interpreted the same way by everyone, but the paper asserts the uniqueness criterion without reporting inter-annotator agreement or spot checks.

Editorial extensions

If this is right

  • Robots and AR systems that rely on a stored map of a room cannot afford frequent re-scans, so the imagination step is a practical requirement; Hypo3D turns that requirement into a score.
  • Directional questions are only well-defined under a fixed world frame, so the anchor-based frame offers a recipe other 3D reasoning datasets can adopt.
  • Movement and replacement changes are the hardest for models, implying that progress will require simulating spatial re-layouts rather than recalling object attributes.
  • The hallucination result implies that knowing when to ignore a change is a distinct skill from knowing when to apply one, and models currently lack both.
  • Since the best open model nearly matches the best closed model, the limiting factor is not model scale or API quality but the hypothetical-reasoning operation itself.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A test the paper leaves unrun: render the changed scenes for real and feed the rendered images to the same models; a sharp jump in exact match would confirm the imagination step as the bottleneck, while a flat score would mean the benchmark also measures general spatial reasoning.
  • The paper asserts but never verifies answer uniqueness; an inter-annotator agreement study on a sample would show whether some model errors are actually annotation ambiguity, especially for addition and replacement changes where human accuracy also dips.
  • The same $\langle$scene, frame, change, question$\rangle$ format transfers to other costly-to-refresh domains such as outdoor navigation, autonomous driving, and warehouse scenes, so Hypo3D's pipeline could serve as a template.
  • The large gap between semantic and non-semantic map inputs suggests object recognition, not reasoning, is often the binding constraint for 2D VLMs; isolating the mental-transformation component from recognition error is where the benchmark's next analysis would be most informative.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper introduces Hypo3D, a 3D visual question answering benchmark for hypothetical reasoning. Given a 3D scene, an anchor-based textual world frame, a context change description (movement, removal, attribute, addition, or replacement), and a question, models must imagine the changed scene and answer. The benchmark comprises 7,727 context changes and 14,885 QA pairs across 700 indoor scenes, built through a human-in-the-loop pipeline in which GPT-4o generates raw context changes and questions and human reviewers edit and filter them (91% of raw questions are discarded). Ten foundation models are evaluated zero-shot in three input modalities: LLMs with scene captions, 2D VLMs with top-view maps, and 3D VLMs with point clouds/RGB-D. The paper reports that the best model, GPT-4o with a semantic top-view map, achieves 45.50% overall exact match, while human evaluators score 91.00% on a 250-item sample, and that models struggle most on movement changes and direction-based questions and often change their answers when the context change is irrelevant.

Significance. The proposed task is well motivated: real-time 3D scene updates are costly, and evaluating whether models can reason about imagined scene changes is a meaningful extension of existing 3D VQA benchmarks. The anchor-based world frame is a reasonable attempt to standardize directional terms, and the public dataset, code, and prompts are a useful community resource. The paper also evaluates a broad set of models across modalities and includes several diagnostic analyses (frame ablation, irrelevant-change hallucination, Chain-of-Thought and ICL effects). If the validity concerns about the human comparison and annotation consistency are addressed, the benchmark could become a standard evaluation for hypothetical spatial reasoning. The central claim that current models substantially underperform humans on this task is plausible, but the quantitative evidence as presented is not yet controlled.

major comments (5)
  1. [§4.3, Table 1] The headline human–model comparison is computed on mismatched evaluation sets. The human row in Table 1 is based on a sample of 50 scenes and 250 QA pairs (50 per context-change type), while model rows are evaluated on the full 14,885-pair benchmark; subtracting the full-benchmark GPT-4o EM (45.50) from the subset human EM (91.00) to claim a 45.5% gap is therefore uncontrolled. The gap could be inflated or deflated depending on how the human subset differs from the full benchmark. Please report model EM/PM on the exact same 250-pair subset used for the human evaluation, or replace the headline comparison with a matched evaluation, and include confidence intervals for the human scores given the small sample size.
  2. [§3.2 'Question-Answer Preparation'] Criterion (3) that each question has a unique and unambiguous answer is asserted rather than demonstrated. The pipeline filters 91% of GPT-4o-generated questions via human reviewers, but the paper reports no inter-annotator agreement on this filtering/editing step and no spot-check of the final question–answer pairs. Since the benchmark's validity and the human reference performance both depend on unambiguous answers, please provide an agreement study (e.g., two independent annotators re-answering a sample of questions, with agreement rates per question type) and report the distribution of reasons for filtering.
  3. [§3.2 'Post-processing'] The SBERT cosine-similarity threshold of 0.8 used to deduplicate context changes and questions is a free parameter, and the specific SBERT model is not identified. No sensitivity analysis is given, so it is unknown whether the benchmark composition and subsequent model rankings are stable to this choice. Please report a threshold sweep (e.g., 0.7, 0.75, 0.85, 0.9) with resulting dataset sizes and, if feasible, model EM on the resulting variants, and cite the exact SBERT checkpoint.
  4. [§4.4 'Insights 4 and 5', Tables 3 and 4] The small-sample analyses in Tables 3 and 4 are reported without error bars, without the number of scenes involved, and without stating whether the 250 sampled context changes overlap with the human-evaluation subset used in Table 1. The conclusion that hypothetical change degrades model performance (Insight 4) is based on differences of a few percentage points on 250 pairs; these differences may be within sampling noise. Please provide paired bootstrap confidence intervals for the w/o-change vs. w/change differences and specify the sampling procedure.
  5. [§3.2 'Anchor-Based World Frame Annotation'] The claim that a single primary orientation plus the right-hand rule yields an unambiguous world frame is not empirically validated. Direction-based questions rely on this frame, and Table 2 tests only a handful of models; if annotators or models interpret 'left/right' inconsistently, the observed model failures on direction questions could partly reflect annotation ambiguity. Please provide a human consistency check (e.g., re-annotate a sample of directional questions with the frame description and compute agreement on the intended answers) and report per-question-type agreement.
minor comments (8)
  1. [§4.3] The sentence 'we sampled 50 scenes and 250 context changes with 50 questions per change type' should specify the total number of human-annotated QA pairs (250) and how the 50 questions per change type are distributed across the five change types.
  2. [Appendix A.2] Table 5 has a caption that belongs to the direction-based experiments in Appendix B.3.2; the table body is missing and the numbering conflicts with Table 11. Please correct the placement and numbering.
  3. [Appendix B.3.1, Table 10] The caption says 'and human evaluators' but the table contains no human row; either add the row or adjust the caption.
  4. [Throughout] The model name is written inconsistently as 'LLaV A-OV' and 'LLaVA-OV'; please unify with the official name.
  5. [References] The SBERT reference (Reimers, 2019) is incomplete; this is a library rather than a paper, and the actual method should be cited (Reimers and Gurevych, EMNLP 2019).
  6. [§4.4, Insight 5] The statement that 'all 2D and 3D VLMs exhibit performance degradation' is contradicted by Table 4, where LLaVA-3D 7B shows PM 40.17 → 40.17 and Llama-3.2 3B improves; please qualify the claim to 'most' and discuss the exceptions.
  7. [Appendix B.3.2, Figure 14] Figure 14 uses the SBERT metric but never defines how the cosine-similarity scores are aggregated; add a one-sentence definition in the caption or text.
  8. [§3.2] The pipeline states that 21 raw questions are generated per context change; clarify that this comes from seven question-type templates each producing three questions, as implied by Appendix A.2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Hypo3D's benchmark construction and zero-shot evaluation are independent of model fitting.

full rationale

The paper's central deliverables are a human-refined 3D VQA dataset and zero-shot evaluations of pre-existing models. The derivation chain therefore has no fitted parameter or predicted variable that is defined in terms of the evaluation outcome. GPT-4o contributes raw context-change and question candidates, but Section 3.2 reports that 91% of raw questions are filtered out and the remainder are edited by independent human reviewers, with human-provided answers; the final benchmark is not a re-description of any model's output. The anchor-based world frame is a data-definition convention for directional terms, not a theorem derived from model behavior. Self-citations (e.g., Mao et al. 2024; Jing et al. 2025) appear only as contextual references on general VLMs and stereo matching and are not load-bearing for the benchmark's validity or for any uniqueness claim. The human-model gap in Section 4.3 is computed by comparing human EM on a 250-item sample with model EM on the full 14,885-item benchmark, which is a statistical/validity weakness rather than a circularity: the model scores are not constructed from, nor fitted to, the human sample, so the comparison does not reduce to an identity. Overall, the evaluation is self-contained against external, fixed ground-truth answers, and no circular step was found.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The benchmark's central measurement rests on annotation quality, frame definition, and representative human sampling. No free parameters are fitted to model performance, but the 0.8 SBERT threshold is a design choice. No invented physical entities are introduced.

free parameters (1)
  • SBERT cosine similarity filter threshold = 0.8
    Used in post-processing to remove semantically similar context changes and questions. Not tuned on model performance, but it is a hand-chosen design decision.
assumptions (3)
  • domain assumption Human annotator and GPT-4o generated descriptions correctly reflect true spatial changes in the scene.
    The benchmark assumes imperfect LLM outputs were corrected by human reviewers, but no inter-annotator agreement or validation study is reported.
  • domain assumption An anchor-based world frame defined by one prominent object and the right-hand rule uniquely determines left, right, front, and back.
    The paper assumes a single primary orientation is sufficient to infer all directions; this may be under-specified for scenes with symmetric layouts or ambiguous anchor descriptions.
  • domain assumption Human evaluator performance on 50 sampled scenes generalizes to the full benchmark.
    The human baseline is computed on a small subset with no confidence intervals or sampling details, yet it is used as the reference for the human-model gap.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hypo3D: Exploring Hypothetical Reasoning in 3D." pith.science (2026). https://pith.science/paper/K6IQ77JW

@misc{pith2026250200954,
  author       = {Pith},
  title        = {Pith review of: Hypo3D: Exploring Hypothetical Reasoning in 3D},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K6IQ77JW}},
  note         = {Machine review of arXiv:2502.00954}
}
read the original abstract

The rise of vision-language foundation models marks an advancement in bridging the gap between human and machine capabilities in 3D scene reasoning. Existing 3D reasoning benchmarks assume real-time scene accessibility, which is impractical due to the high cost of frequent scene updates. To this end, we introduce Hypothetical 3D Reasoning, namely Hypo3D, a benchmark designed to evaluate models' ability to reason without access to real-time scene data. Models need to imagine the scene state based on a provided change description before reasoning. Hypo3D is formulated as a 3D Visual Question Answering (VQA) benchmark, comprising 7,727 context changes across 700 indoor scenes, resulting in 14,885 question-answer pairs. An anchor-based world frame is established for all scenes, ensuring consistent reference to a global frame for directional terms in context changes and QAs. Extensive experiments show that state-of-the-art foundation models struggle to reason in hypothetically changed scenes. This reveals a substantial performance gap compared to humans, particularly in scenarios involving movement changes and directional reasoning. Even when the context change is irrelevant to the question, models often incorrectly adjust their answers. Project website: https://matchlab-imperial.github.io/Hypo3D/

Figures

Figures reproduced from arXiv: 2502.00954 by the authors.

Figure 1
Figure 1. Overview of the Hypo3D benchmark. ⃝1 Examples of five context change types. ⃝2 Sample questions, including scale-based and direction-based questions requiring spatial reasoning, as well as semantic questions, all of which have open-ended answers. ⃝3 The radar chart highlights a notable performance gap between models and humans, especially in direction-based questions. their answers accordingly. Based on the Hypo3D t… view at source ↗
Figure 2
Figure 2. Example of hypothetical reasoning in a 3D scene. Given a 3D scene and an anchor-based frame description (Scene Orientation), models first align the scene to the specified frame. Then, based on a context change description and a question, models hypothetically modify the aligned scene and answer questions about the changed scene. Various models (e.g., LLMs, 2D VLMs, 3D VLMs) can tackle this task using corresponding s… view at source ↗
Figure 3
Figure 3. Dataset Generation Pipeline. The Hypo3D collection pipeline consists of five stages: Stage ⃝1 curates scenes (50 hours per person), Stage ⃝2 defines world frames (10 h/p), Stages ⃝3 and ⃝4 collect context changes and QA descriptions from human annotators (thousands of hours) and LLM, and Stage ⃝5 conducts grammar checks and filters data based on semantic similarity. (25 h/p). answer a to q after applying c to S˜ [P… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Dataset Statistics. ⃝1 Word cloud representing context change descriptions. ⃝2 Frequency distribution of context change types across 7,727 instances. ⃝3 Distribution of question types across change categories, with question frequency consistently highest for scale-base…
Figure 5
Figure 5. Figure 5: Model and human EM performance across question types. Humans consistently achieve strong performance, whereas models struggle, particularly with direction-based questions [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Guidelines for movement change collection in crowdsourcing. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Prompt template for addition change generation. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Prompt template for generating raw proximity questions based on context changes and the object list of the 3D scene. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Prompt template for generating raw relative position questions based on context changes and the object list of the 3D scene. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Question distribution in Hypo3D. A.4. Dataset Statistics [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Prompt template for the main hypothetical reasoning experiments, with differences between prompts for 3D VLM, 2D VLM, and LLM underlined. B.2. Reasoning Prompts The prompt template used for the main results in [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Prompt template for evaluating model performance without using an anchor-based world frame. Reason without Context Change Given a top-view of a 3D scene, mentally rotate the image to align with the specified orientation. Scene Orientation: {} Then, answer a question b…
Figure 13
Figure 13. Figure 13: Prompt template for evaluating models in static scenes without context changes. B.3. More Quantitative Results B.3.1. MORE COMPLETE MAIN RESULTS. We present the complete main results for each context change type and question type in [PITH_FULL_IMAGE:figures/full_fig_…
Figure 14
Figure 14. Figure 14: Model and human SBERT scores across question types. Models struggle the most with direction-based questions, followed by scale-based and semantic questions. B.3.2. MORE RESULTS ON DIRECTIONAL QUESTIONS The radar chart in [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Qualitative Results. The changed object described in the context change is highlighted with a red bounding box. Model outputs are shown in green for correct and red for incorrect predictions. Results indicate that while models struggle with most examples, 2D VLMs are …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 33 canonical work pages

  1. [1]

    Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes

    Achlioptas, P., Abdelreheem, A., Xia, F., Elhoseiny, M., and Guibas, L. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part I 16, pp.\ 422--440. Springer, 2020

    work page 2020

  2. [2]

    Claude 3.5 sonnet, 2024

    Anthropic. Claude 3.5 sonnet, 2024. URL https://www.anthropic.com/news/claude-3-5-sonnet

    2024

  3. [3]

    Scanqa: 3d question answering for spatial scene understanding

    Azuma, D., Miyanishi, T., Kurita, S., and Kawanabe, M. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 19129--19139, 2022

    work page 2022

  4. [4]

    Qwen technical report

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  5. [5]

    Z., Chang, A

    Chen, D. Z., Chang, A. X., and Nie ner, M. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pp.\ 202--221. Springer, 2020

    work page 2020

  6. [6]

    Gaussianeditor: Swift and controllable 3d editing with gaussian splatting

    Chen, Y., Chen, Z., Zhang, C., Wang, F., Yang, X., Wang, Y., Cai, Z., Yang, L., Liu, H., and Lin, G. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 21476--21485, 2024

    work page 2024

  7. [7]

    X., Savva, M., Halber, M., Funkhouser, T., and Nie ner, M

    Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., and Nie ner, M. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 5828--5839, 2017

    work page 2017

  8. [8]

    3D Scanning: A Comprehensive Survey

    Daneshmand, M., Helmi, A., Avots, E., Noroozi, F., Alisinanoglu, F., Arslan, H. S., Gorbova, J., Haamer, R. E., Ozcinar, C., and Anbarjafari, G. 3d scanning: A comprehensive survey. arXiv preprint arXiv:1801.08863, 2018

    work page Pith review arXiv 2018

Show all 54 references
  1. [9]

    S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al

    Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023

    2023 arXiv

  2. [10]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  3. [11]

    Scene-llm: Extending language model for 3d visual understanding and reasoning

    Fu, R., Liu, J., Chen, X., Nie, Y., and Xiong, W. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401, 2024

    2024 arXiv

  4. [12]

    Relightable 3d gaussians: Realistic point cloud relighting with brdf decomposition and ray tracing

    Gao, J., Gu, C., Lin, Y., Li, Z., Zhu, H., Cao, X., Zhang, L., and Yao, Y. Relightable 3d gaussians: Realistic point cloud relighting with brdf decomposition and ray tracing. In European Conference on Computer Vision, pp.\ 73--89. Springer, 2025

    2025

  5. [13]

    Artificial general intelligence: concept, state of the art, and future prospects

    Goertzel, B. Artificial general intelligence: concept, state of the art, and future prospects. Journal of Artificial General Intelligence, 5 0 (1): 0 1, 2014

    2014

  6. [14]

    Hamilton, W. R. Lectures on quaternions. (No Title), 2008

    2008

  7. [15]

    3d-llm: Injecting the 3d world into large language models

    Hong, Y., Zhen, H., Chen, P., Zheng, S., Du, Y., Chen, Z., and Gan, C. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36: 0 20482--20494, 2023

    2023

  8. [16]

    An embodied generalist agent in 3d world

    Huang, J., Yong, S., Ma, X., Linghu, X., Li, P., Wang, Y., Li, Q., Zhu, S.-C., Jia, B., and Huang, S. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871, 2023

    2023 arXiv

  9. [17]

    Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes

    Huang, Y.-H., Sun, Y.-T., Yang, Z., Lyu, X., Cao, Y.-P., and Qi, X. Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 4220--4230, 2024

    2024

  10. [18]

    Sceneverse: Scaling 3d vision-language learning for grounded scene understanding

    Jia, B., Chen, Y., Yu, H., Wang, Y., Niu, X., Liu, T., Li, Q., and Huang, S. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. In European Conference on Computer Vision, pp.\ 289--310. Springer, 2025

  11. [19]

    Match-stereo-videos: Bidirectional alignment for consistent dynamic stereo matching

    Jing, J., Mao, Y., and Mikolajczyk, K. Match-stereo-videos: Bidirectional alignment for consistent dynamic stereo matching. In European Conference on Computer Vision, pp.\ 415--432. Springer, 2025

    2025

  12. [20]

    M., Kowalski, M., Trzci \'n ski, T., and Tagliasacchi, A

    Kania, K., Yi, K. M., Kowalski, M., Trzci \'n ski, T., and Tagliasacchi, A. Conerf: Controllable neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 18623--18632, 2022

    2022

  13. [21]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp.\ 1207--1216, Stanford, CA, 2000. Morgan Kaufmann

    2000

  14. [22]

    Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

    Lewis, M. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019

    1910 arXiv

  15. [23]

    Llava-onevision: Easy visual task transfer

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024 a

    2024 arXiv

  16. [24]

    Topviewrs: Vision-language models as top-view spatial reasoners

    Li, C., Zhang, C., Zhou, H., Collier, N., Korhonen, A., and Vuli \'c , I. Topviewrs: Vision-language models as top-view spatial reasoners. arXiv preprint arXiv:2406.02537, 2024 b

    2024 arXiv

  17. [25]

    Multi-modal situated reasoning in 3d scenes

    Linghu, X., Huang, J., Niu, X., Ma, X., Jia, B., and Huang, S. Multi-modal situated reasoning in 3d scenes. arXiv preprint arXiv:2409.02389, 2024

    2024 arXiv

  18. [26]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems, 36, 2024

    2024

  19. [27]

    Editing conditional radiance fields

    Liu, S., Zhang, X., Zhang, Z., Zhang, R., Zhu, J.-Y., and Russell, B. Editing conditional radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 5773--5783, 2021

    2021

  20. [28]

    Mmscan: A multi-modal 3d scene dataset with hierarchical grounded language annotations

    Lyu, R., Lin, J., Wang, T., Mao, X., Chen, Y., Xu, R., Huang, H., Zhu, C., Lin, D., and Pang, J. Mmscan: A multi-modal 3d scene dataset with hierarchical grounded language annotations. Advances in Neural Information Processing Systems, 37: 0 50898--50924, 2024

    2024

  21. [29]

    Sqa3d: Situated question answering in 3d scenes

    Ma, X., Yong, S., Zheng, Z., Li, Q., Liang, Y., Zhu, S.-C., and Huang, S. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022

    2022 arXiv

  22. [30]

    Opendlign: Open-world point cloud understanding with depth-aligned images

    Mao, Y., Jing, J., and Mikolajczyk, K. Opendlign: Open-world point cloud understanding with depth-aligned images. arXiv preprint arXiv:2404.16538, 2024

    2024 arXiv

  23. [31]

    Moulton, S. T. and Kosslyn, S. M. Imagining predictions: mental imagery as mental emulation. Philosophical Transactions of the Royal Society B: Biological Sciences, 364 0 (1521): 0 1273--1280, 2009

    2009

  24. [32]

    A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A

    Newcombe, R. A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohi, P., Shotton, J., Hodges, S., and Fitzgibbon, A. Kinectfusion: Real-time dense surface mapping and tracking. In 2011 10th IEEE international symposium on mixed and augmented reality, pp.\ 12...

    2011

  25. [33]

    Real-time 3d reconstruction at scale using voxel hashing

    Nie ner, M., Zollh \"o fer, M., Izadi, S., and Stamminger, M. Real-time 3d reconstruction at scale using voxel hashing. ACM Transactions on Graphics (ToG), 32 0 (6): 0 1--11, 2013

    2013

  26. [34]

    Hello gpt-4o, 2024

    OpenAI. Hello gpt-4o, 2024. URL https://openai.com/index/hello-gpt-4o/

    2024

  27. [35]

    Instruction tuning with gpt-4

    Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023

    2023 arXiv

  28. [36]

    Pylyshyn, Z. W. Mental imagery: In search of a theory. Behavioral and brain sciences, 25 0 (2): 0 157--182, 2002

    2002

  29. [37]

    Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario

    Qian, T., Chen, J., Zhuo, L., Jiao, Y., and Jiang, Y.-G. Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 4542--4550, 2024

    2024

  30. [38]

    Artificial general intelligence: Roadmap to achieving human-level capabilities, 2023

    Rayhan, A., Rayhan, R., and Rayhan, S. Artificial general intelligence: Roadmap to achieving human-level capabilities, 2023

    2023

  31. [39]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Reimers, N. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019

    1908 arXiv

  32. [40]

    M., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R

    Seitz, S. M., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R. A comparison and evaluation of multi-view stereo reconstruction algorithms. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06), volume 1, pp.\ 519--528. IEEE, 2006

    2006

  33. [41]

    Games: Mesh-based adapting and modification of gaussian splatting

    Waczy \'n ska, J., Borycki, P., Tadeja, S., Tabor, J., and Spurek, P. Games: Mesh-based adapting and modification of gaussian splatting. arXiv preprint arXiv:2402.01459, 2024

    2024 arXiv

  34. [42]

    Rio: 3d object instance re-localization in changing indoor environments

    Wald, J., Avetisyan, A., Navab, N., Tombari, F., and Nie ner, M. Rio: 3d object instance re-localization in changing indoor environments. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 7658--7667, 2019

    2019

  35. [43]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  36. [44]

    and Spetsakis, M

    Wong, B. and Spetsakis, M. Scene reconstruction and robot navigation using dynamic fields. Autonomous Robots, 8: 0 71--86, 2000

    2000

  37. [45]

    Recent advances in 3d gaussian splatting

    Wu, T., Yuan, Y.-J., Zhang, L.-X., Yang, J., Cao, Y.-P., Yan, L.-Q., and Gao, L. Recent advances in 3d gaussian splatting. Computational Visual Media, 10 0 (4): 0 613--642, 2024

    2024

  38. [46]

    Gaussian grouping: Segment and edit anything in 3d scenes

    Ye, M., Danelljan, M., Yu, F., and Ke, L. Gaussian grouping: Segment and edit anything in 3d scenes. In European Conference on Computer Vision, pp.\ 162--179. Springer, 2025

    2025

  39. [47]

    3d question answering

    Ye, S., Chen, D., Han, S., and Liao, J. 3d question answering. arXiv preprint arXiv:2112.08359, 2021

    2021 arXiv

  40. [48]

    A survey of autonomous driving: Common practices and emerging technologies

    Yurtsever, E., Lambert, J., Carballo, A., and Takeda, K. A survey of autonomous driving: Common practices and emerging technologies. IEEE access, 8: 0 58443--58469, 2020

    2020

  41. [49]

    Pegasus: Pre-training with extracted gap-sentences for abstractive summarization

    Zhang, J., Zhao, Y., Saleh, M., and Liu, P. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International conference on machine learning, pp.\ 11328--11339. PMLR, 2020

    2020

  42. [50]

    Spartun3d: Situated spatial understanding of 3d world in large language models

    Zhang, Y., Xu, Z., Shen, Y., Kordjamshidi, P., and Huang, L. Spartun3d: Situated spatial understanding of 3d world in large language models. arXiv preprint arXiv:2410.03878, 2024

    2024 arXiv

  43. [51]

    Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness

    Zhu, C., Wang, T., Zhang, W., Pang, J., and Liu, X. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness. arXiv preprint arXiv:2409.18125, 2024

    2024 arXiv

  44. [52]

    3d-vista: Pre-trained transformer for 3d vision and text alignment

    Zhu, Z., Ma, X., Chen, Y., Deng, Z., Huang, S., and Li, Q. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 2911--2921, 2023

    2023

  45. [53]

    Real-time non-rigid reconstruction using an rgb-d camera

    Zollh \"o fer, M., Nie ner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., et al. Real-time non-rigid reconstruction using an rgb-d camera. ACM Transactions on Graphics (ToG), 33 0 (4): 0 1--12, 2014

    2014

  46. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.