Embodied Spatial Intelligence: from Implicit Scene Modeling to Spatial Reasoning

Jiading Fang

REVIEW 3 major objections 5 minor 293 references

Embodied Spatial Intelligence: from Implicit Scene Modeling to Spatial Reasoning

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Robust 3D understanding is essential for embodied agents; inductive biases are learnable; text bridges perception to reasoning.

desk verdict A solid PhD thesis compiling six strong papers, but the headline 'practical path to embodied spatial intelligence' is a research agenda, not a demonstrated result. read the letter →

arxiv 2509.00465 v1 pith:63KKXPH5 submitted 2025-08-30 cs.RO cs.AIcs.CV

Jiading Fang This is my paper

classification cs.ROcs.AIcs.CV

keywords embodiedspatialintelligenceimplicitneuralrepresentations3Dsceneunderstandinglargelanguagemodelsreasoningreferringexpressionresolutioncameraself-calibrationlong-horizonplanning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This thesis defines embodied spatial intelligence as the ability to perceive a 3D world and act on it through natural language, decomposing it into a perception step that builds a 3D representation and an action step that reasons over it. It claims that the geometric inductive biases traditionally hard-coded into vision systems are largely learnable from data, and that implicit neural representations offer a robust, generalizable, and scalable way to build those 3D representations. On the reasoning side, it claims that large language models, though weak at mapping and navigation on their own, can be made spatially competent by transcribing 3D scenes into text, executing code for quantitative checks, and maintaining an explicit state summary during long tasks. If these claims hold, robots could recalibrate their cameras from raw video, transfer depth models across domains without retraining, fuse independently built maps, and follow complex language instructions in cluttered real environments.

What carries the argument

The thesis decomposes the action-perception mapping into V (perception from observations to a 3D representation) and F (from that representation to actions). It relies on implicit scene representations—continuous depth fields, neural radiance fields, and modular NeRF sub-maps—and on unified camera models with closed-form differentiable projection. For reasoning, it uses text as the unifying medium: detected objects are transcribed into structured descriptions, an LLM iterates with a Python interpreter for spatial calculations, and a second LLM maintains a concise state summary for long-horizon planning.

What would settle it

Take Transcrib3D to referring expressions that require exact metric distances or occlusion ordering, such as 'the cup at least 30 centimeters from the wall, behind the tall bottle,' and see if accuracy drops to chance while humans succeed; if so, the claim that transcribed text retains the geometry needed for spatial reasoning is falsified.

Watch

Extended reading notes

Core claim

Robust 3D understanding is essential for embodied agents; many of the inductive biases needed for that understanding are learnable at scale; and a practical path toward embodied spatial intelligence lies in strategically integrating high-quality 3D structure with large-scale 2D foundation models, using text as a unifying medium. The thesis supports this with six results: sub-pixel self-calibration from raw video, a generalist depth network that transfers zero-shot, NeRF registration and blending, a benchmark showing LLM mapping gaps, 3D referring-expression resolution via transcribed scenes, and dual-LLM state-maintained planning.

Load-bearing premise

The load-bearing premise is that transcribing 3D scenes into text preserves enough geometric information for language-guided spatial tasks; the thesis itself concedes this transcription is lossy and may discard metric distances, occlusions, or fine orientations.

Editorial extensions

If this is right

Robots can recover from camera calibration drift on the fly using raw video and a self-supervised depth-and-pose objective.
Generalist transformer architectures with 3D data augmentation can replace hand-engineered geometric modules for depth estimation and still transfer across domains.
Large-scale environments can be represented as independently trained NeRFs, enabling distributed and privacy-preserving map building.
Language models show systematic gaps in mapping and navigation; providing explicit map information improves their downstream performance.
Transcribing 3D scenes into text with iterative code execution gives state-of-the-art 3D referring-expression resolution.
Maintaining an explicit world state via a second LLM improves long-horizon language-guided task execution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If 3D inductive biases are genuinely learnable at scale, specialized modules like cost volumes and epipolar losses may become optional; a direct test is to scale a generalist transformer on more diverse 3D data and see if the gap closes.
The text-as-unifying-medium strategy has a measurable ceiling: construct referring expressions that require metric precision, occlusion reasoning, or fine orientation; if accuracy drops to chance, the claim that transcribed text retains geometry is falsified.
The dual-LLM state-maintenance idea likely extends to belief maintenance in partially observable environments, storing uncertainty and hypotheses rather than just past actions.
The MANGO result suggests that explicit memory or map modules, not just larger models, may be the missing piece for spatial reasoning in language models.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my read of arXiv:2509.00465. It's a PhD thesis, and it reads like one: six previously published papers stitched together with an introductory framing. The framing—Embodied Spatial Intelligence, the M = F ∘ V decomposition, 'implicit scene modeling' then 'spatial reasoning'—is the only genuinely new material. That's normal for a dissertation, but worth knowing if you're expecting a monograph.

The individual chapters are the real value. The self-calibration work gets sub-pixel reprojection error without calibration targets; DeFiNe beats published zero-shot transfer to 7-Scenes by a wide margin; NeRFuser adds a practical registration/blending method for large-scale NeRFs; MANGO is a clean benchmark showing LLMs still fail at text-based mapping/navigation; Transcrib3D sets state-of-the-art on ReferIt3D/ScanRefer with a text-based bridge; Statler improves long-horizon decisions with a maintained state. There are ablations (Table 2.7, Figures 2.20–2.22), code releases, and the author flags the lossy compression of translating 3D scenes to text.

Where it's soft: the thesis-level claim (c) — that a practical path lies in 'strategically integrating high-quality 3D structure with large-scale 2D foundation models' — is never tested. The chapter evaluations are disjoint. Transcrib3D uses Mask3D detections, not DeFiNe or NeRFuser; DeFiNe isn't a front-end for any reasoning module. So the integration is a synthesis, not a demonstrated outcome. The author's own caveat about lossy text transcription chips at the 'text as unifying medium' mechanism, though the broader issue is that no component-wise composition is evaluated. That's a missing-support problem for the thesis-level narrative, not an internal inconsistency. Minor stuff: a few tables lack error bars, and there are hand-tuned hyperparameters (σv, γ, τ). For a thesis, these are acceptable; for a paper claiming an integrated path, they'd need addressing.

Verdict: this deserves a serious referee. It's a competent compilation of research that was already peer-reviewed, with a plausible if underevidenced synthesis. If it lands in a venue that publishes dissertations or survey-thesis hybrids, send it out. If the claim is an end-to-end system, the referee should ask for an integrated benchmark or a clearer limitation statement. I'd bring it to a reading group to discuss how to frame multi-paper theses, and I'd cite the individual papers rather than the thesis for specific results.

Referee Report

3 major / 5 minor

Summary. The manuscript is a PhD thesis that defines Embodied Spatial Intelligence and decomposes the robot control mapping as M = F ∘ V (Eq. 1.2). Chapter 2 develops implicit 3D scene representations: self-supervised camera self-calibration, the DeFiNe depth-field network, and NeRFuser for NeRF registration and blending. Chapter 3 develops LLM-based spatial reasoning: the MANGO text-navigation benchmark, Transcrib3D for 3D referring-expression resolution via scene transcription, and Statler for state-maintaining long-horizon planning. The thesis-level claims in Section 1.3 are that (a) robust 3D understanding is essential, (b) many inductive biases are learnable at scale, and (c) a practical path lies in integrating high-quality 3D structure with large-scale 2D foundation models. Each chapter is a previously published work evaluated on external benchmarks, with ablations for key design choices.

Significance. If the thesis-level synthesis were established, this would be a useful integrative statement of a research program: the per-chapter contributions are individually strong, with concrete benchmark results, ablations (Table 2.7, Figures 2.20–2.22), and public code for NeRFuser and Transcrib3D, as well as a hosted benchmark for MANGO. The works have already received peer review in their original venues. However, the manuscript's own central claim is not demonstrated: the scene-representation and reasoning components are never composed. The thesis is honest about the lossy nature of 3D-to-text transcription, but it does not analyze the consequences of that loss. The result is a credible research agenda rather than a validated architectural thesis.

major comments (3)

[Section 1.3, Chapters 2–3] The thesis-level claim (c) is never tested end-to-end. Chapter 2 modules are evaluated on self-calibration, depth estimation, and NeRF registration/blending benchmarks; Chapter 3 modules are evaluated on text-based navigation, 3D referring-expression, and pick-and-place benchmarks. No experiment feeds a Chapter 2 representation into a Chapter 3 reasoner. In particular, Transcrib3D (§3.2) consumes Mask3D detections and an LLM code interpreter, not DeFiNe or NeRFuser, and Statler (§3.3) uses an LLM state summary without Chapter 2 representations. Thus 'a practical path... strategically integrating high-quality 3D structure with large-scale 2D foundation models' is an assertion, not a demonstrated result. Either add a concrete integration experiment or explicitly recast (c) as a research agenda.
[Chapter 3 introduction (p. 64)] The author acknowledges that 'transcribing 3D scenes into language is a form of lossy compression, potentially discarding geometric details crucial for certain tasks.' This caveat directly affects the Transcrib3D mechanism, which uses text as the unifying medium for 3D referring expression resolution and downstream pick-and-place (§3.2.5). The manuscript does not quantify which geometric details are lost or when they matter for the benchmark results in Table 3.2. A concrete fidelity analysis is needed: compare transcript-based reasoning against a variant with structured numeric 3D data, and characterize failure cases as semantic vs. geometric. Without this, the 'practical path' claim is vulnerable exactly where the author identifies the risk.
[Section 3.1] MANGO is presented as evidence about LLMs' navigation and mapping abilities, and the thesis uses this to motivate the broader embodied-spatial-intelligence claim. However, the benchmark is text-only and the mazes are 2D projections of text-game graphs (Figure 3.1), not 3D environments. The results therefore support a narrower claim about textual graph navigation than about 3D spatial reasoning. The thesis should explicitly delimit MANGO's scope so that the per-chapter claims are not read as supporting the 3D-specific central claim more than they do.

minor comments (5)

[Eq. (2.4b)] The definition of r² reads 'm2_x + m2_x'; presumably this should be m_x² + m_y².
[§3.1.4.1] The sentence 'we used the same walkthrough prefixes and questions as GPT-3.5 for a fair comparison' is repeated verbatim in consecutive paragraphs.
[Figure 3.6 caption] Typo: 'necessary to to follow' should be 'necessary to follow'.
[Figure 3.16 caption] Typo: 'However, the the baseline Code-as-Policies' should read 'However, the baseline Code-as-Policies'.
[Prompt 3.5] Typo: 'Dome Roomm' should be 'Dome Room'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: each chapter's results are measured against external benchmarks or held-out data, and the untested central synthesis is a missing-support issue, not a circular derivation.

full rationale

I walked the claimed derivation chain chapter by chapter. Section 2.1 learns camera intrinsics by minimizing a photometric view-synthesis loss and then evaluates the resulting parameters against AprilTag/Basalt calibration and LiDAR-based depth (EuRoC, KITTI); the evaluation targets are external to the training objective, so the sub-pixel reprojection claim is not a fitted input renamed as a prediction. Section 2.2 (DeFiNe) is trained on ScanNet RGB-D and evaluated on held-out ScanNet, 7-Scenes, and virtual-camera projections; the novel-view ground truth is generated from sensor depth and available point clouds, not from DeFiNe's own output, so the zero-shot transfer and query-vs-projection comparisons are genuine generalization tests. Section 2.3 (NeRFuser) registers and blends pretrained NeRFs and checks against ground-truth RGB-D-fused point clouds and held-out imagery; the registration and blending metrics are not constructed from the method's own predictions. The LLM chapters (MANGO, Transcrib3D, Statler) are evaluated on external benchmarks (Jericho games, ReferIt3D, ScanRefer) and simulator/real-robot rollouts, and the LLM responses are not used to define the ground-truth labels. The thesis is compiled from the author's own prior papers, so self-citation is pervasive, but no load-bearing argument reduces to a self-citation chain: the cited works are independently benchmarked, and the strongest external supports (e.g., VGGT for learnable inductive biases) are not the author's own. The main weaknesses are evidentiary, not circular: Section 1.3's claim that a practical path lies in integrating 3D structure with 2D foundation models is never tested as an end-to-end composed system, and the Section 3 preface explicitly concedes that 'transcribing 3D scenes into language is a form of lossy compression, potentially discarding geometric details crucial for certain tasks.' These are limitations of support and correctness risk, not cases where an equation or prediction reduces by construction to its own input. Accordingly, no circular step is identified.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The thesis introduces a conceptual framing but no new physical entities. Its central claims rest on standard ML assumptions and several hand-chosen hyperparameters. The free parameters are typical for deep learning systems and are ablated in part, but they do contribute to the final benchmark numbers.

free parameters (5)

DeFiNe loss weights lambda_s, lambda_v = tuned (ablated in Table 2.7)
Balance depth and view synthesis losses; chosen by hand, sensitivity shown in ablation.
DeFiNe augmentation noise scales sigma_v, sigma_t, sigma_r = e.g., sigma_v=0.25m used in Figure 2.11b
Control the strength of 3D augmentations; tuned on validation.
NeRFuser blending rate gamma = 10 (Mission Bay)
Governs IDW blending sharpness; ablated in Figure 2.22.
NeRFuser proximity threshold tau = 1.2
Filters distant NeRFs in blending; chosen by hand.
NeRFuser distant accumulation distance d = 0.3
Threshold for pose filtering; chosen by hand.

assumptions (5)

domain assumption The Unified Camera Model family (UCM/EUCM/DS) can represent the cameras considered (perspective, fisheye, catadioptric).
Relies on Usenko et al. [1]; not derived in thesis, but needed for self-calibration validity.
domain assumption Input NeRFs are trained from poses that have been preprocessed in a standardized way (centered, rotated, up-aligned, scaled to [-1,1]).
Stated in Section 2.3.3.1; necessary for pose sampling in NeRF registration. If not met, registration fails.
domain assumption LLMs can reason reliably over text transcripts of 3D scenes when given code execution tools and principles.
Core to Transcrib3D and Statler; empirically tested but not guaranteed across all scenes.
domain assumption Mask3D object detector provides sufficient recall of objects relevant to referring expressions.
Transcrib3D depends on the detector's output; if objects are missed, the referent may be absent.
standard math Standard backpropagation and Transformer architectures are taken as given.
Used throughout; standard ML machinery.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Embodied Spatial Intelligence: from Implicit Scene Modeling to Spatial Reasoning." pith.science (2026). https://pith.science/paper/63KKXPH5

@misc{pith2026250900465,
  author       = {Pith},
  title        = {Pith review of: Embodied Spatial Intelligence: from Implicit Scene Modeling to Spatial Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/63KKXPH5}},
  note         = {Machine review of arXiv:2509.00465}
}

read the original abstract

This thesis introduces "Embodied Spatial Intelligence" to address the challenge of creating robots that can perceive and act in the real world based on natural language instructions. To bridge the gap between Large Language Models (LLMs) and physical embodiment, we present contributions on two fronts: scene representation and spatial reasoning. For perception, we develop robust, scalable, and accurate scene representations using implicit neural models, with contributions in self-supervised camera calibration, high-fidelity depth field generation, and large-scale reconstruction. For spatial reasoning, we enhance the spatial capabilities of LLMs by introducing a novel navigation benchmark, a method for grounding language in 3D, and a state-feedback mechanism to improve long-horizon decision-making. This work lays a foundation for robots that can robustly perceive their surroundings and intelligently act upon complex, language-based commands.

Figures

Figures reproduced from arXiv: 2509.00465 by the authors.

**Figure 2.1.** Our self-supervised self-calibration procedure can recover accurate parameters for a wide range of cameras using a structure-from-motion objective on raw videos (EuRoC dataset, top), enabling on-the-fly re-calibration and robustness to intrinsics perturbation (bottom). Camera calibration is integral to robotics and computer vision algorithms that seek to infer geometric properties of the scene from visual input stre… view at source ↗

**Figure 2.2.** Our self-supervised self-calibration architecture. We use gradients from the photometric loss to update the parameters of a unified camera model ( [PITH_FULL_IMAGE:figures/full_fig_p031_2_2.png] view at source ↗

**Figure 2.3.** The Unified Camera Model [1] used in our self-calibration pipeline. Points are projected onto a unit sphere before being projected onto an image plane of a standard pinhole camera offset by α 1−α from the sphere center. aforementioned view synthesis objective alone. At the start of self-supervised depth and pose training, rather than pre-calibrating the camera parameters, we initialize the camera with “default” valu… view at source ↗

Figures from the paper (41 more)

**Figure 2.4.** Figure 2.4: EuRoC rectification results using images from the calibration sequences. Each column visualizes the results rendered using (left) the Basalt calibrated intrinsics and (right) our learned intrinsics. The top row shows that detected (small circles) and reprojected (big…

**Figure 2.** Figure 2: , we see that our learned parameters allow for the rectified grid to [PITH_FULL_IMAGE:figures/full_fig_p035_2.png]

**Figure 2.5.** Figure 2.5: EuRoC perturbation test, showing how our proposed learningbased method is able to recover from changes in camera parameters for online self-calibration. a camera based on a potentially incorrect prior calibration. Generally, this requires the capture of new calibrat…

**Figure 2.6.** Figure 2.6: Self-supervised monocular pointcloud for EuRoC, obtained by unprojecting predicted depth with our learned camera parameters (input image on the bottom right). lenging dataset that involves cluttered indoor sequences with six-DoF motion. Compared to the per-frame dist…

**Figure 2.7.** Figure 2.7: Qualitative depth estimation results on non-pinhole datasets with (a) fisheye and (b) catadioptric images. 2.1.5 Conclusion We proposed a procedure to self-calibrate a family of general camera models using self-supervised depth and pose estimation as a proxy task. We…

**Figure 2.8.** Figure 2.8: Our Depth Field Networks (DeFiNe) achieve state of the art in multi-view depth estimation, while also enabling predictions from arbitrary viewpoints. Modern 3D computer vision leverages learning to boost geometric reasoning, mapping image data to classical structures…

**Figure 2.9.** Figure 2.9: Overview of our proposed DeFiNe architecture, and the embeddings used to encode and decode information for depth and view synthesis. 2.2.3.1 Perceiver IO Perceiver IO [85] is a recent extension of the Perceiver [89] architecture. The Perceiver architecture alleviate…

**Figure 2.10.** Figure 2.10: Geometric augmentations. (a) Information from camera i is projected onto a virtual camera at Tv, creating additional supervision from other viewpoints. (b) Noise T ′ 0 is introduced to the canonical camera at T0, and then propagated to other cameras to preserve rela…

**Figure 2.11.** Figure 2.11: Depth estimation results on ScanNet-Stereo. (a) We outperform contemporary methods by a large margin. (b) Depth estimation results on virtual cameras using different values for σv at training and test time. use a custom split to select 2000 samples from 90 scenes no…

**Figure 2.12.** Figure 2.12: Reconstructed two-view pointclouds, from ScanNet-Stereo. DeFiNe pointclouds are generated using both depth maps and RGB images queried from our learned latent representation. GT DeFiNe GT DeFiNe [PITH_FULL_IMAGE:figures/full_fig_p055_2_12.png]

**Figure 2.13.** Figure 2.13: Depth estimation and view synthesis results on ScanNet. Although view synthesis is not our primary goal, it can be achieved with minimal modifications, and we show that it improves depth estimation performance. of traditional relative viewing rays; ( [PITH_FULL_IMA…

**Figure 2.14.** Figure 2.14: Depth estimation results from novel viewpoints. frames onto these new viewpoints. We evaluate both standard depth estimation networks [15, 68, 96] as well as DeFiNe itself, that can be used to either explicitly project information from encoded frames onto new viewpo…

**Figure 2.** Figure 2: reports results in terms of root mean squared error (RMSE) [PITH_FULL_IMAGE:figures/full_fig_p058_2.png]

**Figure 2.15.** Figure 2.15: Overall framework of NeRFuser We present NeRFuser , a novel framework that extends the representational capacity of neural radiance fields (NeRFs) to produce high-fidelity representations of large-scale scenes. Integral to our approach is the decomposition of spati…

**Figure 2.16.** Figure 2.16: Qualitative comparison of blending methods. Our proposed [PITH_FULL_IMAGE:figures/full_fig_p062_2_16.png]

**Figure 2.17.** Figure 2.17: Illustration of NeRF renderings and their accumulations. Column [PITH_FULL_IMAGE:figures/full_fig_p067_2_17.png]

**Figure 2.18.** Figure 2.18: Illustration of IDW-based blending methods: IDW-2D, IDW-3D [PITH_FULL_IMAGE:figures/full_fig_p069_2_18.png]

**Figure 2.19.** Figure 2.19: NeRF blending with IDW-based methods on the Mission Bay [PITH_FULL_IMAGE:figures/full_fig_p071_2_19.png]

**Figure 2.** Figure 2: , distant accumulation-based filtering removes mostly bad images [PITH_FULL_IMAGE:figures/full_fig_p073_2.png]

**Figure 2.20.** Figure 2.20: Registration error and time consumption against varying [PITH_FULL_IMAGE:figures/full_fig_p074_2_20.png]

**Figure 2.21.** Figure 2.21: Effect of re-rendering poses on NeRF registration. [PITH_FULL_IMAGE:figures/full_fig_p075_2_21.png]

**Figure 2.22.** Figure 2.22: Effect of blending rate γ in IDW-based blending. Ablation of γ in IDW-based blending We study the effect of blending rate γ in IDW-based blending on Object-Centric Indoor Scenes. Specifically, we use ground-truth transformations and set distance test ratio τ = 1.8. …

**Figure 3.1.** Figure 3.1: Map of Zork-I. Arrows denote the direction of travel during the [PITH_FULL_IMAGE:figures/full_fig_p080_3_1.png]

**Figure 3.2.** Figure 3.2: Success rates of the examined models on (a) DF and (b) RF [PITH_FULL_IMAGE:figures/full_fig_p089_3_2.png]

**Figure 3.** Figure 3: presents the success rates of all models. For each kind of question [PITH_FULL_IMAGE:figures/full_fig_p089_3.png]

**Figure 3.3.** Figure 3.3: Success rates of GPT-3.5 and GPT-4 broken down into individual [PITH_FULL_IMAGE:figures/full_fig_p091_3_3.png]

**Figure 3.4.** Figure 3.4: Playing minigames. corners of the hard-question plots in [PITH_FULL_IMAGE:figures/full_fig_p093_3_4.png]

**Figure 3.5.** Figure 3.5: The overall Transcrib3D framework, which takes as input the colored point-cloud and referring expression (in green), and outputs the ID or bounding box of the referent object. To resolve the referring expression “the chair in the corner of the room, between the white…

**Figure 3.6.** Figure 3.6: Transcrib3D enables a robot to resolve complex 3D referring [PITH_FULL_IMAGE:figures/full_fig_p097_3_6.png]

**Figure 3.** Figure 3: illustrates our proposed Transcrib3D framework. Given the input [PITH_FULL_IMAGE:figures/full_fig_p099_3.png]

**Figure 3.7.** Figure 3.7: Illustration of the iterative code generation and reasoning process. [PITH_FULL_IMAGE:figures/full_fig_p101_3_7.png]

**Figure 3.8.** Figure 3.8: Qualitative comparisons between Transcrib3D (ours, in [PITH_FULL_IMAGE:figures/full_fig_p105_3_8.png]

**Figure 3.9.** Figure 3.9: Qualitative comparison of the grounding performance of (top) CaP [PITH_FULL_IMAGE:figures/full_fig_p109_3_9.png]

**Figure 3.10.** Figure 3.10: Typical failure cases in Transcrib3D, with ground-truth bounding [PITH_FULL_IMAGE:figures/full_fig_p110_3_10.png]

**Figure 3.11.** Figure 3.11: Overall Framework of Statler There has been a significant research interest in employing large language models to empower intelligent robots with complex reasoning. Existing work focuses on harnessing their abilities to reason about the histories of their actions an…

**Figure 3.12.** Figure 3.12: Model accuracies on the three-cups-and-a-ball shell game. LLM+State is a simplified version of our proposed Statler framework. For each method, the solid line shows how its accuracy a(n) changes with the number of swaps n. The dashed line is the relative accuracy: r…

**Figure 3.13.** Figure 3.13: Examples of simulations that show the result of executing different [PITH_FULL_IMAGE:figures/full_fig_p117_3_13.png]

**Figure 3.14.** Figure 3.14: The simulated domains we consider include a Pick-and-Place; b [PITH_FULL_IMAGE:figures/full_fig_p119_3_14.png]

**Figure 3.15.** Figure 3.15: Examples that show the result of querying LLMs with and without [PITH_FULL_IMAGE:figures/full_fig_p120_3_15.png]

**Figure 3.16.** Figure 3.16: A comparison of the resulting behavior for (top) Code-as-Policies [PITH_FULL_IMAGE:figures/full_fig_p123_3_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

293 extracted references · 18 canonical work pages

[1]

The double sphere camera model,

V. Usenko, N. Demmel, and D. Cremers, “The double sphere camera model,” in Proceedings of the International Conference on 3D Vision (3DV) , 2018, pp. 552–560

2018
[2]

Visual- inertial mapping with non-linear factor recovery,

V. Usenko, N. Demmel, D. Schubert, J. Stueckler, and D. Cremers, “Visual- inertial mapping with non-linear factor recovery,” IEEE Robotics and Automa- tion Letters, vol. 5, no. 2, pp. 422–429, 2020

2020
[3]

The EuRoC micro aerial vehicle datasets,

M. Burri, J. Nikolic, P. Gohl, T. Schneider, J. Rehder, S. Omari, M. W. Achtelik, and R. Siegwart, “The EuRoC micro aerial vehicle datasets,” International Journal of Robotics Research, vol. 35, no. 10, pp. 1157–1163, 2016

2016
[4]

Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras,

A. Gordon, H. Li, R. Jonschkowski, and A. Angelova, “Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019

2019
[5]

PointGroup: Dual-set point grouping for 3D instance segmentation,

L. Jiang, H. Zhao, S. Shi, S. Liu, C.-W. Fu, and J. Jia, “PointGroup: Dual-set point grouping for 3D instance segmentation,” in Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , 2020

2020
[6]

Mask3D: Mask transformer for 3D semantic instance segmentation,

J. Schult, F. Engelmann, A. Hermans, O. Litany, S. Tang, and B. Leibe, “Mask3D: Mask transformer for 3D semantic instance segmentation,” in Proc. IEEE Int’l Conf. on Robotics and Automation (ICRA) , 2023

2023
[7]

What’s “up

A. Kamath, J. Hessel, and K.-W. Chang, “What’s “up” with vision-language models? investigating their struggle with spatial reasoning,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

2023
[8]

Spa- tialvlm: Endowing vision-language models with spatial reasoning capabilities,

B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia, “Spa- tialvlm: Endowing vision-language models with spatial reasoning capabilities,” 115 in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

2024

Show all 293 references

[9]

Spatialrgpt: Grounded spatial reasoning for vision-language models,

X. Cheng, X. Wang, R. Zhao, X. Wang, L. Zhang, and S. Z. Li, “Spatialrgpt: Grounded spatial reasoning for vision-language models,” in Advances in Neural Information Processing Systems (NeurIPS) , 2024

2024
[10]

PointNet++: Deep hierarchical feature learning on point sets in a metric space,

C. Qi, L. Yi, H. Su, and L. J. Guibas, “PointNet++: Deep hierarchical feature learning on point sets in a metric space,” in Advances in Neural Information Processing Systems (NeurIPS), 2017

2017
[11]

KinectFusion: Real-time dense surface mapping and tracking,

R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon, “KinectFusion: Real-time dense surface mapping and tracking,” in Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR) , 2011

2011
[12]

Octomap: An efficient probabilistic 3d mapping framework based on octrees,

A. Hornung, K. M. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard, “Octomap: An efficient probabilistic 3d mapping framework based on octrees,” Autonomous Robots, vol. 34, no. 3, pp. 189–206, 2013

2013
[13]

Dtam: Dense track- ing and mapping in real-time,

R. A. Newcombe, S. J. Lovegrove, and A. J. Davison, “Dtam: Dense track- ing and mapping in real-time,” in 2011 IEEE International Conference on Computer Vision (ICCV) . IEEE, 2011, pp. 2320–2327

2011
[14]

PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume,

D. Sun, X. Yang, M. Liu, and J. Kautz, “PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 2018, pp. 8934–8943

2018
[15]

Digging into self- supervised monocular depth prediction,

C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow, “Digging into self- supervised monocular depth prediction,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2019

2019
[16]

Depth field networks for generalizable multi-view scene representation,

V. Guizilini, I. Vasiljevic, J. Fang, R. Ambru, G. Shakhnarovich, M. R. Walter, and A. Gaidon, “Depth field networks for generalizable multi-view scene representation,” in European Conference on Computer Vision. Springer, 2022, pp. 245–262

2022
[17]

Input- level inductive biases for 3D reconstruction,

W. Yifan, C. Doersch, R. Arandjelovi´ c, J. Carreira, and A. Zisserman, “Input- level inductive biases for 3D reconstruction,” arXiv preprint arXiv:2112.03243 , 2021. 116

2021 arXiv
[18]

Delira: Self-supervised depth, light, and radiance fields,

V. Guizilini, I. Vasiljevic, J. Fang, R. Ambrus, S. Zakharov, V. Sitzmann, and A. Gaidon, “Delira: Self-supervised depth, light, and radiance fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 17 935–17 945

2023
[19]

Vggt: Visual geometry grounded transformer,

J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny, “Vggt: Visual geometry grounded transformer,” in Proceedings of the Computer Vision and Pattern Recognition Conference , 2025, pp. 5294–5306

2025
[20]

NeRF: Representing scenes as neural radiance fields for view synthesis,

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “NeRF: Representing scenes as neural radiance fields for view synthesis,” in Proceedings of the European Conference on Computer Vision (ECCV), 2020

2020
[21]

Nerfuser: Large-scale scene repre- sentation by nerf fusion,

J. Fang, S. Lin, I. Vasiljevic, V. Guizilini, R. Ambrus, A. Gaidon, G. Shakhnarovich, and M. R. Walter, “Nerfuser: Large-scale scene repre- sentation by nerf fusion,” 2023

2023
[22]

Di-nerf: Distributed nerf for collabora- tive learning with relative pose refinement,

M. Asadi, K. Zareinia, and S. Saeedi, “Di-nerf: Distributed nerf for collabora- tive learning with relative pose refinement,” IEEE Robotics and Automation Letters, 2024

2024
[23]

Transcrib3d: 3d referring expression resolution through large language models,

J. Fang, X. Tan, S. Lin, I. Vasiljevic, V. Guizilini, H. Mei, R. Ambrus, G. Shakhnarovich, and M. R. Walter, “Transcrib3d: 3d referring expression resolution through large language models,” arXiv preprint arXiv:2404.19221 , 2024

2024 arXiv
[24]

Perceiving layout and knowing distances: The integration, relative potency, and contextual use of different information about depth,

J. E. Cutting and P. M. Vishton, “Perceiving layout and knowing distances: The integration, relative potency, and contextual use of different information about depth,” in Perception of space and motion . Elsevier, 1995, pp. 69–117

1995
[25]

Mango: A benchmark for evaluating mapping and navigation abilities of large language models,

P. Ding, J. Fang, P. Li, K. Wang, X. Zhou, M. Yu, J. Li, M. R. Walter, and H. Mei, “Mango: A benchmark for evaluating mapping and navigation abilities of large language models,” arXiv preprint arXiv:2403.19913 , 2024

2024 arXiv
[26]

Do as I can, 117 not as I say: Grounding language in robotic affordances,

b. Ichter, A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, D. Kalashnikov, S. Levine, Y. Lu, C. Parada, K. Rao, P. Sermanet, A. T. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, M. Yan, N. Brown, M. Ahn, O. Cortes, N. Sieve...

2023
[28]

Self-supervised camera self-calibration from video,

J. Fang, I. Vasiljevic, V. Guizilini, R. Ambrus, G. Shakhnarovich, A. Gaidon, and M. R. Walter, “Self-supervised camera self-calibration from video,” in 2022 International Conference on Robotics and Automation (ICRA) , 2022, pp. 8468–8475

2022
[29]

Structure-from-motion revisited,

J. L. Sch¨ onberger and J.-M. Frahm, “Structure-from-motion revisited,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4104–4113

2016
[30]

Pixelwise view selection for unstructured multi-view stereo,

J. L. Sch¨ onberger, E. Zheng, M. Pollefeys, and J.-M. Frahm, “Pixelwise view selection for unstructured multi-view stereo,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2016, pp. 501–518

2016
[31]

Orb-slam: a versatile and accurate monocular slam system,

R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos, “Orb-slam: a versatile and accurate monocular slam system,” IEEE transactions on robotics , vol. 31, no. 5, pp. 1147–1163, 2015

2015
[32]

NeRF: Representing scenes as neural radiance fields for view synthesis,

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “NeRF: Representing scenes as neural radiance fields for view synthesis,” in Proceedings of the European Conference on Computer Vision (ECCV), 2020, pp. 405–421

2020
[33]

Implicit neural representations with periodic activation functions,

V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein, “Implicit neural representations with periodic activation functions,” Advances in neural information processing systems, vol. 33, pp. 7462–7473, 2020

2020
[34]

A flexible new technique for camera calibration,

Z. Zhang, “A flexible new technique for camera calibration,”IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 22, no. 11, pp. 1330–1334, 2000

2000
[35]

A flexible technique for accurate omnidirectional camera calibration and structure from motion,

D. Scaramuzza, A. Martinelli, and R. Siegwart, “A flexible technique for accurate omnidirectional camera calibration and structure from motion,” in Proceedings of the IEEE International Conference on Computer Vision Systems (ICVS), 2006, pp. 45–45. 118

2006
[36]

A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses,

J. Kannala and S. S. Brandt, “A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 28, no. 8, pp. 1335–1340, 2006

2006
[37]

A general imaging model and a method for finding its parameters,

M. D. Grossberg and S. K. Nayar, “A general imaging model and a method for finding its parameters,” in Proceedings of the International Conference on Computer Vision (ICCV) , vol. 2, 2001, pp. 108–115

2001
[38]

Using vanishing points for camera calibration,

B. Caprile and V. Torre, “Using vanishing points for camera calibration,” International Journal on Computer Vision , vol. 4, no. 2, pp. 127–139, 1990

1990
[39]

A stratified approach to metric self-calibration,

M. Pollefeys and L. Van Gool, “A stratified approach to metric self-calibration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1997, pp. 407–412

1997
[40]

Camera calibration from vanishing points in image of architectural scenes,

R. Cipolla, T. Drummond, and D. P. Robertson, “Camera calibration from vanishing points in image of architectural scenes,” in Proceedings of the British Machine Vision Conference (BMVC) , 1999, pp. 382–391

1999
[41]

DeepCalib: A deep learning approach for automatic intrinsic calibration of wide field-of-view cameras,

O. Bogdan, V. Eckstein, F. Rameau, and J.-C. Bazin, “DeepCalib: A deep learning approach for automatic intrinsic calibration of wide field-of-view cameras,” in Proceedings of the ACM SIGGRAPH European Conference on Visual Media Production, 2018

2018
[42]

Unsupervised CNN for single view depth estimation: Geometry to the rescue,

R. Garg, V. K. Bg, G. Carneiro, and I. Reid, “Unsupervised CNN for single view depth estimation: Geometry to the rescue,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2016, pp. 740–756

2016
[43]

Unsupervised learning of depth and ego-motion from video,

T. Zhou, M. Brown, N. Snavely, and D. G. Lowe, “Unsupervised learning of depth and ego-motion from video,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2017, pp. 1851–1858

2017
[44]

Neural ray surfaces for self-supervised learning of depth and ego-motion,

I. Vasiljevic, V. Guizilini, R. Ambrus, S. Pillai, W. Burgard, G. Shakhnarovich, and A. Gaidon, “Neural ray surfaces for self-supervised learning of depth and ego-motion,” in Proceedings of the International Conference on 3D Vision (3DV), 2020

2020
[45]

Hartley and A

R. Hartley and A. Zisserman, Multiple view geometry in computer vision . Cambridge university press, 2003

2003
[46]

Bradski and A

G. Bradski and A. Kaehler, “OpenCV,” Dr. Dobb’s Journal of Software Tools , vol. 3, 2000. 119

2000
[47]

Extending kalibr: Calibrating the extrinsics of multiple IMUs and of individual axes,

J. Rehder, J. Nikolic, T. Schneider, T. Hinzmann, and R. Siegwart, “Extending kalibr: Calibrating the extrinsics of multiple IMUs and of individual axes,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2016, pp. 4304–4311

2016
[48]

A perception-driven autonomous urban vehicle,

J. Leonard, J. How, S. Teller, M. Berger, S. Campbell, G. Fiore, L. Fletcher, E. Frazzoli, A. Huang, S. Karaman, O. Koch, Y. Kuwata, D. Moore, E. Olson, S. Peters, J. Teo, R. Truax, M. Walter, D. Barrett, A. Epstein, K. Maheloni, K. Moyer, T. Jones, R. Buckley, M. Antone, R. G...

2008
[49]

Autonomous driving in urban environments: Boss and the Urban Challenge,

C. Urmson, J. Anhalt, D. Bagnell, C. Baker, R. Bittner, M. Clark, J. Dolan, D. Duggins, T. Galatali, C. Geyer et al. , “Autonomous driving in urban environments: Boss and the Urban Challenge,” Journal of Field Robotics , vol. 25, no. 8, pp. 425–466, 2008

2008
[50]

Lens distortion for close-range photogrammetry,

J. G. Fryer and D. C. Brown, “Lens distortion for close-range photogrammetry,” Photogrammetric Engineering and Remote Sensing , vol. 52, pp. 51–58, 1986

1986
[51]

A unifying theory for central panoramic systems and practical implications,

C. Geyer and K. Daniilidis, “A unifying theory for central panoramic systems and practical implications,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2000, pp. 445–461

2000
[52]

An enhanced unified camera model,

B. Khomutenko, G. Garcia, and P. Martinet, “An enhanced unified camera model,” IEEE Robotics and Automation Letters , vol. 1, no. 1, pp. 137–144, 2015

2015
[53]

DeepFo- cal: A method for direct focal length estimation,

S. Workman, C. Greenwell, M. Zhai, R. Baltenberger, and N. Jacobs, “DeepFo- cal: A method for direct focal length estimation,” in Proceedings of the IEEE International Conference on Image Processing (ICIP) , 2015, pp. 1369–1373

2015
[54]

Horizon lines in the wild,

S. Workman, M. Zhai, and N. Jacobs, “Horizon lines in the wild,” arXiv preprint arXiv:1604.02129, 2016

2016 arXiv
[55]

Radial lens distortion correction using convolutional neural networks trained with synthesized images,

J. Rong, S. Huang, Z. Shang, and X. Ying, “Radial lens distortion correction using convolutional neural networks trained with synthesized images,” in Proceedings of the Asian Conference on Computer Vision , 2016, pp. 35–49

2016
[56]

FishEyeRectNet: A multi-context collaborative deep network for fisheye image rectification,

X. Yin, X. Wang, J. Yu, M. Zhang, P. Fua, and D. Tao, “FishEyeRectNet: A multi-context collaborative deep network for fisheye image rectification,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2018, pp. 469–484. 120

2018
[57]

Deep single image camera calibration with radial distortion,

M. Lopez, R. Mari, P. Gargallo, Y. Kuang, J. Gonzalez-Jimenez, and G. Haro, “Deep single image camera calibration with radial distortion,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 11 817–11 825

2019
[58]

A perceptual measure for deep single image camera calibration,

Y. Hold-Geoffroy, K. Sunkavalli, J. Eisenmann, M. Fisher, E. Gambaretto, S. Hadap, and J.-F. Lalonde, “A perceptual measure for deep single image camera calibration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2018

2018
[59]

Single view metrology in the wild,

R. Zhu, X. Yang, Y. Hold-Geoffroy, F. Perazzi, J. Eisenmann, K. Sunkavalli, and M. Chandraker, “Single view metrology in the wild,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2020, pp. 316–333

2020
[60]

DeepPTZ: Deep self-calibration for PTZ cameras,

C. Zhang, F. Rameau, J. Kim, D. M. Argaw, J.-C. Bazin, and I. S. Kweon, “DeepPTZ: Deep self-calibration for PTZ cameras,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2020, pp. 1041–1049

2020
[61]

Learning perspective undistortion of portraits,

Y. Zhao, Z. Huang, T. Li, W. Chen, C. LeGendre, X. Ren, A. Shapiro, and H. Li, “Learning perspective undistortion of portraits,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2019, pp. 7849–7859

2019
[62]

Learning to recover 3D scene shape from a single image,

W. Yin, J. Zhang, O. Wang, S. Niklaus, L. Mai, S. Chen, and C. Shen, “Learning to recover 3D scene shape from a single image,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021, pp. 204–213

2021
[63]

Progressively complementary network for fisheye image rectification using appearance flow,

S. Yang, C. Lin, K. Liao, C. Zhang, and Y. Zhao, “Progressively complementary network for fisheye image rectification using appearance flow,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 6348–6357

2021
[64]

A deep ordinal distortion estimation approach for distortion rectification,

K. Liao, C. Lin, and Y. Zhao, “A deep ordinal distortion estimation approach for distortion rectification,” IEEE Transactions on Image Processing , vol. 30, pp. 3362–3375, 2021

2021
[65]

Deep geometry-aware camera self- calibration from video,

A. Hagemann, M. Knorr, and C. Stiller, “Deep geometry-aware camera self- calibration from video,” in Proceedings of the IEEE/CVF International Con- ference on Computer Vision , 2023, pp. 3438–3448

2023
[66]

Robust self-supervised extrinsic self-calibration,

T. Kanai, I. Vasiljevic, V. Guizilini, A. Gaidon, and R. Ambrus, “Robust self-supervised extrinsic self-calibration,” in 2023 IEEE/RSJ International 121 Conference on Intelligent Robots and Systems (IROS) . IEEE, 2023, pp. 1932–1939

2023
[67]

Digging into self- supervised monocular depth estimation,

C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow, “Digging into self- supervised monocular depth estimation,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2019, pp. 3828–3838

2019
[68]

3D packing for self-supervised monocular depth estimation,

V. Guizilini, R. Ambrus, S. Pillai, A. Raventos, and A. Gaidon, “3D packing for self-supervised monocular depth estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020

2020
[69]

SuperDepth: Self-supervised, super- resolved monocular depth estimation,

S. Pillai, R. Ambrus, and A. Gaidon, “SuperDepth: Self-supervised, super- resolved monocular depth estimation,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) , 2019

2019
[70]

Are we ready for autonomous driv- ing? The KITTI vision benchmark suite,

A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driv- ing? The KITTI vision benchmark suite,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2012, pp. 3354–3361

2012
[71]

nuScenes: A multimodal dataset for au- tonomous driving,

H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuScenes: A multimodal dataset for au- tonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020, pp. 11...

2020
[72]

Depth map prediction from a single image using a multi-scale deep network,

D. Eigen, C. Puhrsch, and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” arXiv preprint arXiv:1406.2283, 2014

2014 arXiv
[73]

Calibrating and centering quasi- central catadioptric cameras,

M. Sch¨ onbein, T. Strauß, and A. Geiger, “Calibrating and centering quasi- central catadioptric cameras,” in Proceedings of the IEEE International Con- ference on Robotics and Automation (ICRA) , 2014, pp. 4443–4450

2014
[74]

Automatic differentiation in PyTorch,

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in PyTorch,” in Advances in Neural Information Processing Systems (NeurIPS) , 2017

2017
[75]

AprilTag: A robust and flexible visual fiducial system,

E. Olson, “AprilTag: A robust and flexible visual fiducial system,” in Pro- ceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2011, pp. 3400–3407

2011
[76]

End-to-end learning of geometry and context for deep stereo 122 regression,

A. Kendall, H. Martirosyan, S. Dasgupta, P. Henry, R. Kennedy, A. Bachrach, and A. Bry, “End-to-end learning of geometry and context for deep stereo 122 regression,” in Proceedings of the International Conference on Computer Vision (ICCV), 2017, pp. 66–75

2017
[77]

RAFT-Stereo: Multilevel recurrent field transforms for stereo matching,

L. Lipson, Z. Teed, and J. Deng, “RAFT-Stereo: Multilevel recurrent field transforms for stereo matching,” in Proceedings of the International Conference on 3D Vision (3DV) , 2021, pp. 218–227

2021
[78]

RAFT: Recurrent all-pairs field transforms for optical flow,

Z. Teed and J. Deng, “RAFT: Recurrent all-pairs field transforms for optical flow,” in Proceedings of the European Conference on Computer Vision (ECCV), 2020

2020
[79]

DeepV2D: Video to depth with differentiable structure from motion,

——, “DeepV2D: Video to depth with differentiable structure from motion,” in Proceedings of the International Conference on Learning Representations (ICLR), 2020

2020
[80]

Neural fields in visual computing and beyond,

Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. Sitzmann, and S. Sridhar, “Neural fields in visual computing and beyond,” arXiv preprint arXiv:2111.11426 , 2021

2021 arXiv
[81]

pixelNeRF: Neural radiance fields from one or few images,

A. Yu, V. Ye, M. Tancik, and A. Kanazawa, “pixelNeRF: Neural radiance fields from one or few images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021

2021
[82]

An image is worth 16x16 words: Transformers for image recognition at scale,

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Un- terthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al. , “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020

2010 arXiv
[83]

Revisiting stereo depth estimation from a sequence-to-sequence perspective with transformers,

Z. Li, X. Liu, N. Drenkow, A. Ding, F. X. Creighton, R. H. Taylor, and M. Unberath, “Revisiting stereo depth estimation from a sequence-to-sequence perspective with transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 6197–6206

2021
[84]

Vision transformers for dense prediction,

R. Ranftl, A. Bochkovskiy, and V. Koltun, “Vision transformers for dense prediction,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 12 179–12 188

2021
[85]

Perceiver IO: A general architecture for structured inputs & outputs,

A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, D. Zoran, A. Brock, E. Shelhamer et al. , “Perceiver IO: A general architecture for structured inputs & outputs,” arXiv preprint arXiv:2107.14795, 2021. 123

2021 arXiv
[86]

Taming transformers for high-resolution image synthesis,

P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high-resolution image synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 12 873–12 883

2021
[87]

Geometry-free view synthesis: Trans- formers and no 3D priors,

R. Rombach, P. Esser, and B. Ommer, “Geometry-free view synthesis: Trans- formers and no 3D priors,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2021, pp. 14 356–14 366

2021
[88]

Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations,

M. S. Sajjadi, H. Meyer, E. Pot, U. Bergmann, K. Greff, N. Radwan, S. Vora, M. Lucic, D. Duckworth, A. Dosovitskiy, J. Uszkoreit, T. Funkhouser, and A. Tagliasacchi, “Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations,...

2021 arXiv
[89]

Perceiver: General perception with iterative attention,

A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira, “Perceiver: General perception with iterative attention,” in Proceedings of the International Conference on Machine Learning (ICML) , 2021, pp. 4651–4664

2021
[90]

ScanNet: Richly-annotated 3D reconstructions of indoor scenes,

A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “ScanNet: Richly-annotated 3D reconstructions of indoor scenes,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5828–5839

2017
[91]

Scene coordinate regression forests for camera relocalization in RGB-D images,

J. Shotton, B. Glocker, C. Zach, S. Izadi, A. Criminisi, and A. Fitzgibbon, “Scene coordinate regression forests for camera relocalization in RGB-D images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 2930–2937

2013
[92]

Learning depth from single monocular images,

A. Saxena, S. Chung, and A. Ng, “Learning depth from single monocular images,” in Advances in Neural Information Processing Systems (NeurIPS) , 2005

2005
[93]

Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture,

D. Eigen and R. Fergus, “Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2015, pp. 2650–2658

2015
[94]

Deep ordinal regression network for monocular depth estimation,

H. Fu, M. Gong, C. Wang, K. Batmanghelich, and D. Tao, “Deep ordinal regression network for monocular depth estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2018, pp. 2002–2011. 124

2018
[95]

Deeper depth prediction with fully convolutional residual networks,

I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, and N. Navab, “Deeper depth prediction with fully convolutional residual networks,” in Proceedings of the International Conference on 3D Vision (3DV) , 2016, pp. 239–248

2016
[96]

From big to small: Multi- scale local planar guidance for monocular depth estimation,

J. H. Lee, M.-K. Han, D. W. Ko, and I. H. Suh, “From big to small: Multi- scale local planar guidance for monocular depth estimation,” arXiv:1907.10326, 2019

1907 arXiv
[97]

Unsupervised monocular depth estimation with left-right consistency,

C. Godard, O. Mac Aodha, and G. J. Brostow, “Unsupervised monocular depth estimation with left-right consistency,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2017, pp. 270–279

2017
[98]

Feature-metric loss for self-supervised learning of depth and egomotion,

C. Shu, K. Yu, Z. Duan, and K. Yang, “Feature-metric loss for self-supervised learning of depth and egomotion,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2020, pp. 572–588

2020
[99]

The temporal opportunist: Self-supervised multi-frame monocular depth,

J. Watson, O. Mac Aodha, V. Prisacariu, G. Brostow, and M. Firman, “The temporal opportunist: Self-supervised multi-frame monocular depth,” in Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR) , 2021, pp. 1164–1174

2021
[100]

Towards zero- shot scale-aware monocular depth estimation,

V. Guizilini, I. Vasiljevic, D. Chen, R. Ambru s, , and A. Gaidon, “Towards zero- shot scale-aware monocular depth estimation,” inProceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 9233–9243

2023
[101]

Structure-from-motion revisited,

J. L. Schonberger and J.-M. Frahm, “Structure-from-motion revisited,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4104–4113

2016
[102]

Pyramid stereo matching network,

J.-R. Chang and Y.-S. Chen, “Pyramid stereo matching network,” in Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 5410–5418

2018
[103]

MVSNet: Depth inference for unstructured multi-view stereo,

Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan, “MVSNet: Depth inference for unstructured multi-view stereo,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2018, pp. 767–783

2018
[104]

DPSNet: End-to-end deep plane sweep stereo,

S. Im, H.-G. Jeon, S. Lin, and I. S. Kweon, “DPSNet: End-to-end deep plane sweep stereo,” in Proceedings of the International Conference on Learning Representations (ICLR), 2019. 125

2019
[105]

Raft-3D: Scene flow using rigid-motion embeddings,

Z. Teed and J. Deng, “Raft-3D: Scene flow using rigid-motion embeddings,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 8375–8384

2021
[106]

DROID-SLAM: Deep visual SLAM for monocular, stereo, and RGB-D cameras,

——, “DROID-SLAM: Deep visual SLAM for monocular, stereo, and RGB-D cameras,” in Advances in Neural Information Processing Systems (NeurIPS) , 2021

2021
[107]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS) , 2017

2017
[108]

Unifying flow, stereo and depth estimation,

H. Xu, J. Zhang, J. Cai, H. Rezatofighi, F. Yu, D. Tao, and A. Geiger, “Unifying flow, stereo and depth estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023

2023
[109]

DeMoN: Depth and motion network for learning monocular stereo,

B. Ummenhofer, H. Zhou, J. Uhrig, N. Mayer, E. Ilg, A. Dosovitskiy, and T. Brox, “DeMoN: Depth and motion network for learning monocular stereo,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5038–5047

2017
[110]

DeepTAM: Deep tracking and mapping,

H. Zhou, B. Ummenhofer, and T. Brox, “DeepTAM: Deep tracking and mapping,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 822–838

2018
[111]

Consistent video depth estimation,

X. Luo, J.-B. Huang, R. Szeliski, K. Matzen, and J. Kopf, “Consistent video depth estimation,” ACM Transactions on Graphics (TOG) , vol. 39, no. 4, pp. 71–1, 2020

2020
[112]

Robust consistent video depth estimation,

J. Kopf, X. Rong, and J.-B. Huang, “Robust consistent video depth estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 1611–1621

2021
[113]

Multi-view depth estimation using epipolar spatio-temporal networks,

X. Long, L. Liu, W. Li, C. Theobalt, and W. Wang, “Multi-view depth estimation using epipolar spatio-temporal networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021, pp. 8258–8267

2021
[114]

NeuralRecon: Real-time coherent 3D reconstruction from monocular video,

J. Sun, Y. Xie, L. Chen, X. Zhou, and H. Bao, “NeuralRecon: Real-time coherent 3D reconstruction from monocular video,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2021, pp. 15 598–15 607. 126

2021
[115]

Neural RGB-D surface reconstruction,

D. Azinovi´ c, R. Martin-Brualla, D. B. Goldman, M. Nießner, and J. Thies, “Neural RGB-D surface reconstruction,” arXiv preprint arXiv:2104.04532, 2021

2021 arXiv
[116]

Depth-supervised NeRF: Fewer views and faster training for free,

K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan, “Depth-supervised NeRF: Fewer views and faster training for free,” arXiv preprint arXiv:2107.02791 , 2021

2021 arXiv
[117]

Urban radiance fields,

K. Rematas, A. Liu, P. P. Srinivasan, J. T. Barron, A. Tagliasacchi, T. Funkhouser, and V. Ferrari, “Urban radiance fields,” arXiv preprint arXiv:2111.14643, 2021

2021 arXiv
[118]

NerfingMVS: Guided op- timization of neural radiance fields for indoor multi-view stereo,

Y. Wei, S. Liu, Y. Rao, W. Zhao, J. Lu, and J. Zhou, “NerfingMVS: Guided op- timization of neural radiance fields for indoor multi-view stereo,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2021

2021
[119]

NICE-SLAM: Neural implicit scalable encoding for SLAM,

Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R. Oswald, and M. Pollefeys, “NICE-SLAM: Neural implicit scalable encoding for SLAM,” arXiv preprint arXiv:2112.12130 , 2021

2021 arXiv
[120]

Light field networks: Neural scene representations with single-evaluation rendering,

V. Sitzmann, S. Rezchikov, B. Freeman, J. Tenenbaum, and F. Durand, “Light field networks: Neural scene representations with single-evaluation rendering,” in Advances in Neural Information Processing Systems (NeurIPS) , 2021

2021
[121]

Deep residual learning for image recognition,

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2016. [Online]. Available: https://openaccess.thecvf.com/content cvpr 2016/papers/He Deep Residual...

2016
[122]

A survey on image data augmentation for deep learning,

C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” Journal of Big Data , vol. 6, no. 1, pp. 1–48, 2019

2019
[123]

Learning 3D object categories by looking around them,

D. Novotny, D. Larlus, and A. Vedaldi, “Learning 3D object categories by looking around them,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2017, pp. 5218–5227

2017
[124]

Normal assisted stereo depth estimation,

U. Kusupati, S. Cheng, R. Chen, and H. Su, “Normal assisted stereo depth estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020, pp. 2189–2199

2020
[125]

BA-Net: Dense bundle adjustment network,

C. Tang and P. Tan, “BA-Net: Dense bundle adjustment network,” arXiv preprint arXiv:1806.04807, 2018. 127

2018 arXiv
[126]

Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,

R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun, “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2020

2020
[127]

Neural RGB → D sensing: Depth and uncertainty from a video camera,

C. Liu, J. Gu, K. Kim, S. G. Narasimhan, and J. Kautz, “Neural RGB → D sensing: Depth and uncertainty from a video camera,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2019, pp. 10 986–10 995

2019
[128]

Occlusion-aware depth estimation with adaptive normal constraints,

X. Long, L. Liu, C. Theobalt, and W. Wang, “Occlusion-aware depth estimation with adaptive normal constraints,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2020, pp. 640–657

2020
[129]

Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age,

C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. Reid, and J. J. Leonard, “Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age,” IEEE Transactions on Robotics, vol. 32, no. 6, pp. 1309–1332, December 2016

2016
[130]

Neural fields in visual computing and beyond,

Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. sitzmann, and S. Sridhar, “Neural fields in visual computing and beyond,” Computer Graphics Forum , vol. 41, no. 2, pp. 641–676, May
[131]

Instant neural graphics primi- tives with a multiresolution hash encoding,

T. M¨ uller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primi- tives with a multiresolution hash encoding,” ACM Transactions on Graphics (TOG), vol. 41, no. 4, 2022

2022
[132]

Zip-NeRF: Anti-aliased grid-based neural radiance fields,

J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Zip-NeRF: Anti-aliased grid-based neural radiance fields,” arXiv preprint arXiv:2304.06706, 2023

2023 arXiv
[133]

Block-NeRF: Scalable large scene neural view synthesis,

M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar, “Block-NeRF: Scalable large scene neural view synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022, pp. 8238–8248

2022
[134]

Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields,

J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan, “Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2021, pp. 5835–5844. 128

2021
[135]

Mip-NeRF 360: Unbounded anti-aliased neural radiance fields,

J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Mip-NeRF 360: Unbounded anti-aliased neural radiance fields,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 5460–5469

2021
[136]

Plenoxels: Radiance fields without neural networks,

A. Yu, S. Fridovich-Keil, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa, “Plenoxels: Radiance fields without neural networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021, pp. 5491–5500

2021
[137]

Direct voxel grid optimization: Super- fast convergence for radiance fields reconstruction,

C. Sun, M. Sun, and H.-T. Chen, “Direct voxel grid optimization: Super- fast convergence for radiance fields reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021, pp. 5449–5459

2021
[138]

Nerfstudio: A modular framework for neural radiance field development,

M. Tancik, E. Weber, E. Ng, R. Li, B. Yi, J. Kerr, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, A. Ahuja, D. McAllister, and A. Kanazawa, “Nerfstudio: A modular framework for neural radiance field development,” arXiv preprint arXiv:2302.04264, 2023

2023 arXiv
[139]

CityNeRF: Building NeRF at city scale,

Y. Xiangli, L. Xu, X. Pan, N. Zhao, A. Rao, C. Theobalt, B. Dai, and D. Lin, “CityNeRF: Building NeRF at city scale,” arXiv preprint arXiv:2112.05504 , 2021

2021 arXiv
[140]

NeRFusion: Fusing radiance fields for large-scale scene reconstruction,

X. Zhang, S. Bi, K. Sunkavalli, H. Su, and Z. Xu, “NeRFusion: Fusing radiance fields for large-scale scene reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022, pp. 5439–5448

2022
[141]

iMAP: Implicit mapping and positioning in real-time,

E. Sucar, S. Liu, J. Ortiz, and A. J. Davison, “iMAP: Implicit mapping and positioning in real-time,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2021, pp. 6209–6218

2021
[142]

NICE-SLAM: Neural implicit scalable encoding for SLAM,

Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R. Oswald, and M. Pollefeys, “NICE-SLAM: Neural implicit scalable encoding for SLAM,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 12 786–12 796

2022
[143]

NICER-SLAM: Neural implicit scene encoding for RGB SLAM,

Z. Zhu, S. Peng, V. Larsson, Z. Cui, M. R. Oswald, A. Geiger, and M. Pollefeys, “NICER-SLAM: Neural implicit scene encoding for RGB SLAM,”arXiv preprint arXiv:2302.03594, 2023. 129

2023 arXiv
[144]

Mega-NeRF: Scalable con- struction of large-scale NeRFs for virtual fly-throughs,

H. Turki, D. Ramanan, and M. Satyanarayanan, “Mega-NeRF: Scalable con- struction of large-scale NeRFs for virtual fly-throughs,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021, pp. 12 912–12 921

2021
[145]

KiloNeRF: Speeding up neural ra- diance fields with thousands of tiny MLPs,

C. Reiser, S. Peng, Y. Liao, and A. Geiger, “KiloNeRF: Speeding up neural ra- diance fields with thousands of tiny MLPs,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2021, pp. 14 315–14 325

2021
[146]

From coarse to fine: Robust hierarchical localization at large scale,

P.-E. Sarlin, C. Cadena, R. Siegwart, and M. Dymczyk, “From coarse to fine: Robust hierarchical localization at large scale,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2019, pp. 12 716–12 725

2019
[147]

SuperGlue: Learning feature matching with graph neural networks,

P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, “SuperGlue: Learning feature matching with graph neural networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020, pp. 4938–4947

2020
[148]

SuperPoint: Self-supervised interest point detection and description,

D. DeTone, T. Malisiewicz, and A. Rabinovich, “SuperPoint: Self-supervised interest point detection and description,” Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition Workshops (CVPR W) , pp. 224–236, 2018

2018
[149]

R2D2: Repeatable and reliable detector and descriptor,

J. Revaud, P. Weinzaepfel, C. R. de Souza, N. Pion, G. Csurka, Y. Cabon, and M. Humenberger, “R2D2: Repeatable and reliable detector and descriptor,” arXiv preprint arXiv:1906.06195 , 2019

1906 arXiv
[150]

D2-Net: A trainable CNN for joint description and detection of local features,

M. Dusmanu, I. Rocco, T. Pajdla, M. Pollefeys, J. Sivic, A. Torii, and T. Sattler, “D2-Net: A trainable CNN for joint description and detection of local features,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 8084–8093

2019
[151]

iMeRR: Inverting neural radiance fields for pose estimation,

L. Yen-Chen, P. Florence, J. T. Barron, A. Rodriguez, P. Isola, and T.-Y. Lin, “iMeRR: Inverting neural radiance fields for pose estimation,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021, pp. 1323–1330

2021
[152]

Parallel inversion of neural radiance fields for robust pose estimation,

Y. Lin, T. M¨ uller, J. Tremblay, B. Wen, S. Tyree, A. Evans, P. A. Vela, and S. Birchfield, “Parallel inversion of neural radiance fields for robust pose estimation,” arXiv preprint arXiv:2210.10108 , 2022. 130

2022 arXiv
[153]

Vision-only robot navigation in a neural radiance world,

M. Adamkiewicz, T. Chen, A. Caccavale, R. Gardner, P. Culbertson, J. Bohg, and M. Schwager, “Vision-only robot navigation in a neural radiance world,” IEEE Robotics and Automation Letters , vol. 7, no. 2, pp. 4606–4613, 2021

2021
[154]

Loc-NeRF: Monte Carlo localization using neural radiance fields,

D. Maggio, M. Abate, J. Shi, C. Mario, and L. Carlone, “Loc-NeRF: Monte Carlo localization using neural radiance fields,” arXiv preprint arXiv:2209.09050, 2022

2022 arXiv
[155]

Catnips: Collision avoidance through neural implicit probabilistic scenes,

T. Chen, P. Culbertson, and M. Schwager, “Catnips: Collision avoidance through neural implicit probabilistic scenes,” arXiv preprint arXiv:2302.12931 , 2023

2023 arXiv
[156]

LENS: Localization enhanced by NeRF synthesis,

A. Moreau, N. Piasco, D. Tsishkou, B. Stanciulescu, and A. de La Fortelle, “LENS: Localization enhanced by NeRF synthesis,” in Proceedings of the Con- ference on Robot Learning (CoRL) , 2022, pp. 1347–1356

2022
[157]

BARF: Bundle-adjusting neural radiance fields,

C.-H. Lin, W.-C. Ma, A. Torralba, and S. Lucey, “BARF: Bundle-adjusting neural radiance fields,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2021, pp. 5741–5751

2021
[158]

NeRF–: Neural radiance fields without known camera parameters,

Z. Wang, S. Wu, W. Xie, M. Chen, and V. A. Prisacariu, “NeRF–: Neural radiance fields without known camera parameters,” arXiv preprint arXiv:2102.07064, 2021

2021 arXiv
[159]

Self- calibrating neural radiance fields,

Y. Jeong, S. Ahn, C. Choy, A. Anandkumar, M. Cho, and J. Park, “Self- calibrating neural radiance fields,” in Proceedings of the International Confer- ence on Computer Vision (ICCV) , 2021, pp. 5846–5854

2021
[160]

nerf2nerf: Pair- wise registration of neural radiance fields,

L. Goli, D. Rebain, S. Sabour, A. Garg, and A. Tagliasacchi, “nerf2nerf: Pair- wise registration of neural radiance fields,” arXiv preprint arXiv:2211.01600 , 2022

2022 arXiv
[161]

Zero NeRF: Registration with zero overlap,

C. Peat, O. Batchelor, R. Green, and J. Atlas, “Zero NeRF: Registration with zero overlap,” arXiv preprint arXiv:2211.12544 , 2022

2022 arXiv
[162]

Registering neural radiance fields as 3D density images,

H. Jiang, R. Li, H. Sun, Y.-W. Tai, and C.-K. Tang, “Registering neural radiance fields as 3D density images,” arXiv preprint arXiv:2305.12843 , 2023

2023 arXiv
[163]

Dreg-nerf: Deep registration for neural radiance fields,

Y. Chen and G. H. Lee, “Dreg-nerf: Deep registration for neural radiance fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 22 703–22 713

2023
[164]

The Laplacian pyramid as a compact image code,

P. J. Burt and E. H. Adelson, “The Laplacian pyramid as a compact image code,” IEEE Transactions on Communications , vol. 31, pp. 532–540, 1983. 131

1983
[165]

Recognising panoramas,

M. A. Brown and D. G. Lowe, “Recognising panoramas,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2003, pp. 1218–1225

2003
[166]

Nerflets: Local radiance fields for efficient structure-aware 3D scene representation from 2D supervision,

X. Zhang, A. Kundu, T. A. Funkhouser, L. J. Guibas, H. Su, and K. Genova, “Nerflets: Local radiance fields for efficient structure-aware 3D scene representation from 2D supervision,” arXiv preprint arXiv:2303.03361 , 2023. [Online]. Available: https://api.semanticscholar.org/C...

2023 arXiv
[167]

Blended-NeRF: Zero- shot object generation and blending in existing neural radiance fields,

O. Gordon, O. Avrahami, and D. Lischinski, “Blended-NeRF: Zero- shot object generation and blending in existing neural radiance fields,” arXiv preprint arXiv:2306.12760 , 2023. [Online]. Available: https: //api.semanticscholar.org/CorpusID:259224726

2023 arXiv
[168]

3d gaussian splatting for real-time radiance field rendering

B. Kerbl, G. Kopanas, T. Leimk¨ uhler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering.” ACM Trans. Graph., vol. 42, no. 4, pp. 139–1, 2023

2023
[169]

Nerfstudio: A framework for neural radiance field development,

M. Tancik*, E. Weber*, E. Ng*, R. Li, B. Yi, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, A. Ahuja, D. McAllister, and A. Kanazawa, “Nerfstudio: A framework for neural radiance field development,” 2022. [Online]. Available: https://github.com/nerfstudio-project/nerfstudio

2022
[170]

Density-aware NeRF ensembles: Quantifying predictive uncertainty in neural radiance fields,

N. S¨ underhauf, J. Abou-Chakra, and D. Miller, “Density-aware NeRF ensembles: Quantifying predictive uncertainty in neural radiance fields,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) , 2022, pp. 9370–9376. [Online]. Available: https...

2022
[171]

ScanNet: Richly-annotated 3D reconstructions of indoor scenes,

A. Dai, A. X. Chang, M. Savva, M. Halber, T. A. Funkhouser, and M. Nießner, “ScanNet: Richly-annotated 3D reconstructions of indoor scenes,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2432–2443

2017
[172]

Efficient variants of the ICP algorithm,

S. M. Rusinkiewicz and M. Levoy, “Efficient variants of the ICP algorithm,” in Proceedings of the International Conference on 3-D Digital Imaging and Modeling, 2001, pp. 145–152

2001
[173]

Fast global registration,

Q.-Y. Zhou, J. Park, and V. Koltun, “Fast global registration,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2016, pp. 766–782

2016
[174]

Fast point feature histograms (FPFH) for 3D registration,

R. B. Rusu, N. Blodow, and M. Beetz, “Fast point feature histograms (FPFH) for 3D registration,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) , 2009, pp. 3212–3217. 132

2009
[175]

Understanding natural language commands for robotic navigation and mobile manipulation,

S. Tellex, T. Kollar, S. Dickerson, M. Walter, A. Banerjee, S. Teller, and N. Roy, “Understanding natural language commands for robotic navigation and mobile manipulation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 25, no. 1, 2011, pp. 1507–1514

2011
[176]

Language models are few-shot learners,

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. ...

2020
[177]

Tree of thoughts: Deliberate problem solving with large language models,

S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024
[178]

Kahneman, Thinking, fast and slow

D. Kahneman, Thinking, fast and slow . macmillan, 2011

2011
[179]

Bottom up top down detection transformers for language grounding in images and point clouds,

A. Jain, N. Gkanatsios, I. Mediratta, and K. Fragkiadaki, “Bottom up top down detection transformers for language grounding in images and point clouds,” arXiv preprint arXiv:2112.08879 , 2021

2021 arXiv
[180]

3D-VisTA: Pre-trained transformer for 3D vision and text alignment,

Z. Zhu, X. Ma, Y. Chen, Z. Deng, S. Huang, and Q. Li, “3D-VisTA: Pre-trained transformer for 3D vision and text alignment,”arXiv preprint arXiv:2308.04352, 2023

2023 arXiv
[181]

Lost in the middle: How language models use long contexts,

N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics , vol. 12, pp. 157–173, 2024

2024
[183]

Language as a disruptive technology: Abstract concepts, embodiment and the flexible mind,

G. Dove, “Language as a disruptive technology: Abstract concepts, embodiment and the flexible mind,” Philosophical Transactions of the Royal Society B , vol. 373, no. 1752, p. 20170135, 2018

2018
[184]

Tomasello, A Natural History of Human Thinking

M. Tomasello, A Natural History of Human Thinking . Cambridge, MA: Harvard University Press, 2014. 133

2014
[185]

Pinker, The Language Instinct: How the Mind Creates Language

S. Pinker, The Language Instinct: How the Mind Creates Language . New York: William Morrow and Company, 1994

1994
[186]

The road towards understanding embodied decisions,

J. Gordon, A. Maselli, G. L. Lancia, T. Thiery, P. Cisek, and G. Pezzulo, “The road towards understanding embodied decisions,”Neuroscience & Biobehavioral Reviews, vol. 131, pp. 722–736, 2021

2021
[187]

Explaining math: Gesturing lightens the load,

S. Goldin-Meadow, H. Nusbaum, S. D. Kelly, and S. Wagner, “Explaining math: Gesturing lightens the load,” Psychological Science, vol. 12, no. 6, pp. 516–522, 2001

2001
[188]

Thoughts, behaviour, and brain dynamics during navigation in the real world,

H. J. Spiers and E. A. Maguire, “Thoughts, behaviour, and brain dynamics during navigation in the real world,” Neuroimage, vol. 31, no. 4, pp. 1826–1840,
[189]

The cognitive map in humans: spatial navigation and beyond,

R. A. Epstein, E. Z. Patai, J. B. Julian, and H. J. Spiers, “The cognitive map in humans: spatial navigation and beyond,” Nature Neuroscience, no. 11, pp. 1504– 1513, 2017. [Online]. Available: https://www.nature.com/articles/nn.4656

2017
[190]

Solving the detour problem in navigation: A model of prefrontal and hippocampal interactions,

H. J. Spiers and S. J. Gilbert, “Solving the detour problem in navigation: A model of prefrontal and hippocampal interactions,” Frontiers in Human Neuroscience , 2015. [Online]. Available: https: //pubmed.ncbi.nlm.nih.gov/25852515/

2015
[191]

Hippocampal and prefrontal processing of network topology to simulate the future,

A.-H. Javadi, B. Emo, L. R. Howard, F. E. Zisch, Y. Yu, R. Knight, J. Pinelo Silva, and H. J. Spiers, “Hippocampal and prefrontal processing of network topology to simulate the future,” Nature Communications, 2017. [Online]. Available: https://www.nature.com/articles/ncomms14652

2017
[192]

Foun- dation models for decision making: Problems, methods, and opportunities,

S. Yang, O. Nachum, Y. Du, J. Wei, P. Abbeel, and D. Schuurmans, “Foun- dation models for decision making: Problems, methods, and opportunities,” arXiv preprint arXiv:2303.04129 , 2023

2023 arXiv
[193]

GLUE: A multi-task benchmark and analysis platform for natural language understanding,

A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” arXiv preprint arXiv:1804.07461 , 2018. [Online]. Available: https://arxiv.org/pdf/1804.07461.pdf

2018 arXiv
[194]

SuperGLUE: A stickier benchmark for general-purpose language understanding systems,

A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “SuperGLUE: A stickier benchmark for general-purpose language understanding systems,” arXiv preprint 1905.00537 , 2019. [Online]. Available: https://arxiv.org/pdf/1905.00537.pdf 134

1905 arXiv
[195]

MCTest: A challenge dataset for the open-domain machine comprehension of text,

M. Richardson, C. J. Burges, and E. Renshaw, “MCTest: A challenge dataset for the open-domain machine comprehension of text,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) ,
[196]

Race: Large-scale reading comprehension dataset from examinations,

G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy, “Race: Large-scale reading comprehension dataset from examinations,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2017. [Online]. Available: https://arxiv.org/abs/1704.04683

2017 arXiv
[197]

HellaSwag: Can a machine really finish your sentence?

R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, “HellaSwag: Can a machine really finish your sentence?” in Proceedings of the Association for Computational Linguistics (ACL) , 2019. [Online]. Available: https://arxiv.org/abs/1905.07830

2019 arXiv
[198]

PIQA: Reasoning about physical commonsense in natural language,

Y. Bisk, R. Zellers, J. Gao, and Y. Choi, “PIQA: Reasoning about physical commonsense in natural language,” in Proceedings of the National Conference on Artificial Intelligence (AAAI) , 2020. [Online]. Available: https://arxiv.org/abs/1911.11641

2020 arXiv
[199]

Cosmos QA: Machine reading comprehension with contextual commonsense reasoning,

L. Huang, R. L. Bras, C. Bhagavatula, and Y. Choi, “Cosmos QA: Machine reading comprehension with contextual commonsense reasoning,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2019. [Online]. Available: https://arxiv.org/abs/...

2019 arXiv
[200]

CommonsenseQA: A question answering challenge targeting commonsense knowledge,

A. Talmor, J. Herzig, N. Lourie, and J. Berant, “CommonsenseQA: A question answering challenge targeting commonsense knowledge,” in Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2019. [Online]. Available:...

2019 arXiv
[201]

A diverse corpus for evaluating and developing english math word problem solvers,

S.-Y. Miao, C.-C. Liang, and K.-Y. Su, “A diverse corpus for evaluating and developing english math word problem solvers,” in Proceedings of the Association for Computational Linguistics (ACL) , 2020. [Online]. Available: https://arxiv.org/abs/2106.15772

2020 arXiv
[202]

Training verifiers to solve math word problems,

K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano et al. , “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168 , 2021. [Online]. Available: https://arxiv.org/abs/2110.14168 135

2021 arXiv
[203]

Are nlp models really able to solve simple math word problems?

A. Patel, S. Bhattamishra, and N. Goyal, “Are nlp models really able to solve simple math word problems?” in Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) ,
[204]

Think you have solved question answering? Try ARC, the AI2 Reasoning Challenge,

P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? Try ARC, the AI2 Reasoning Challenge,” arXiv preprint arXiv:1803.05457 , 2018. [Online]. Available: https://arxiv.org/abs/1803.05457

2018 arXiv
[205]

Can a suit of armor conduct electricity? a new dataset for open book question answering,

T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? a new dataset for open book question answering,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2018. [Online]. Available: https://arxiv....

2018 arXiv
[206]

Qasc: A dataset for question answering via sentence composition,

T. Khot, P. Clark, M. Guerquin, P. Jansen, and A. Sabharwal, “Qasc: A dataset for question answering via sentence composition,” in Proceedings of the National Conference on Artificial Intelligence (AAAI) , 2020. [Online]. Available: https://arxiv.org/abs/1910.11473

2020 arXiv
[207]

From F to A on the NY Regents Science Exams: An overview of the Aristo project,

P. Clark, O. Etzioni, T. Khot, D. Khashabi, B. Mishra, K. Richardson, A. Sabharwal, C. Schoenick, O. Tafjord, N. Tandon et al. , “From F to A on the NY Regents Science Exams: An overview of the Aristo project,” AI Magazine , 2020. [Online]. Available: https: //ojs.aaai.org/aim...

2020
[208]

Measuring massive multitask language understanding,

D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” in Proceedings of the International Conference on Learning Representations (ICLR), 2021. [Online]. Available: https://arxiv.org/abs/2009.03300

2021 arXiv
[209]

Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,

A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso et al. , “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,” Transactions on Machine Learning Research, 2022. ...

2022 arXiv
[212]

Inner monologue: Embodied reasoning through planning with language models,

W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tomp- son, I. Mordatch, Y. Chebotar et al. , “Inner monologue: Embodied reasoning through planning with language models,” arXiv preprint arXiv:2207.05608 , 2022

2022 arXiv
[213]

ChatGPT for robotics: Design principles and model abilities,

S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, “ChatGPT for robotics: Design principles and model abilities,” Microsoft Auton. Syst. Robot. Res , 2023

2023
[214]

Voyager: An open-ended embodied agent with large language models,

G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar, “Voyager: An open-ended embodied agent with large language models,” arXiv preprint arXiv:2305.16291 , 2023

2023 arXiv
[215]

PaLM-E: An embodied multimodal language model,

D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu et al. , “PaLM-E: An embodied multimodal language model,” arXiv preprint arXiv:2303.03378 , 2023

2023 arXiv
[217]

Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,

W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,” in Proceedings of the International Conference on Machine Learning (ICML) , 2022

2022
[218]

Programmatically grounded, compositionally generalizable robotic manipulation,

R. Wang, J. Mao, J. Hsu, H. Zhao, J. Wu, and Y. Gao, “Programmatically grounded, compositionally generalizable robotic manipulation,” in Proceedings of the International Conference on Learning Representations (ICLR) , 2023

2023
[219]

Lever- aging language for accelerated learning of tool manipulation,

A. Z. Ren, B. Govil, T.-Y. Yang, K. R. Narasimhan, and A. Majumdar, “Lever- aging language for accelerated learning of tool manipulation,” in Proceedings of the Conference on Robot Learning (CoRL) , 2023

2023
[220]

Improving vision-and-language navigation with image-text pairs from the Web,

A. Majumdar, A. Shrivastava, S. Lee, P. Anderson, D. Parikh, and D. Batra, “Improving vision-and-language navigation with image-text pairs from the Web,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2020. 137

2020
[221]

Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation,

S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song, “Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. [Online]. Available: ...

2023 arXiv
[222]

LM-Nav: Robotic navigation with large pre-trained models of language, vision, and action,

D. Shah, B. Osi´ nski, S. Levine et al. , “LM-Nav: Robotic navigation with large pre-trained models of language, vision, and action,” in Proceedings of the Conference on Robot Learning (CoRL) , 2023. [Online]. Available: https://arxiv.org/abs/2207.04429

2023 arXiv
[223]

Visual language maps for robot navigation,

C. Huang, O. Mees, A. Zeng, and W. Burgard, “Visual language maps for robot navigation,” arXiv preprint arXiv:2210.05714 , 2022. [Online]. Available: https://arxiv.org/abs/2210.05714

2022 arXiv
[224]

Virtualhome: Simulating household activities via programs,

X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, and A. Torralba, “Virtualhome: Simulating household activities via programs,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. [Online]. Available: https://arxiv.org/abs/1806.07011

2018 arXiv
[225]

Alfred: A benchmark for interpreting grounded instructions for everyday tasks,

M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox, “Alfred: A benchmark for interpreting grounded instructions for everyday tasks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020. [...

2020 arXiv
[226]

MineDojo: Building open-ended embodied agents with internet-scale knowledge,

L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and A. Anandkumar, “MineDojo: Building open-ended embodied agents with internet-scale knowledge,” in Advances in Neural Information Processing Systems (NeurIPS) , 2022. [Online]. Available:...

2022 arXiv
[227]

Inferring maps and behaviors from natural language instructions,

F. Duvallet, M. R. Walter, T. Howard, S. Hemachandra, J. Oh, S. Teller, N. Roy, and A. Stentz, “Inferring maps and behaviors from natural language instructions,” in Proceedings of the International Symposium on Experimental Robotics (ISER) , 2014. [Online]. Available: https://...

2014 doi
[228]

Listen, attend, and walk: Neural mapping of navigational instructions to action sequences,

H. Mei, M. Bansal, and M. Walter, “Listen, attend, and walk: Neural mapping of navigational instructions to action sequences,” in Proceedings of the National Conference on Artificial Intelligence (AAAI) , 2016. 138

2016
[229]

Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,

P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. S¨ underhauf, I. D. Reid, S. Gould, and A. van den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” in Proceedings of the IEEE/CVF Conference on Compute...

2017
[230]

Speaker- follower models for vision-and-language navigation,

D. Fried, R. Hu, V. Cirik, A. Rohrbach, J. Andreas, L.-P. Morency, T. Berg-Kirkpatrick, K. Saenko, D. Klein, and T. Darrell, “Speaker- follower models for vision-and-language navigation,” in Advances in Neural Information Processing Systems (NeurIPS) , Dec. 2018. [Online]. Ava...

2018 arXiv
[231]

Vision-language navigation with self-supervised auxiliary reasoning tasks,

F. Zhu, Y. Zhu, X. Chang, and X. Liang, “Vision-language navigation with self-supervised auxiliary reasoning tasks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , Jun. 2020

2020
[232]

FILM: Following instructions in language with modular methods,

S. Y. Min, D. S. Chaplot, P. Ravikumar, Y. Bisk, and R. Salakhutdinov, “FILM: Following instructions in language with modular methods,” arXiv preprint arXiv:2110.07342, 2021

2021 arXiv
[233]

Multigrid neural memory,

T. Huynh, M. Maire, and M. R. Walter, “Multigrid neural memory,” in Proceedings of the International Conference on Machine Learning (ICML) ,
[234]

Emergence of maps in the memories of blind navigation agents,

E. Wijmans, M. Savva, I. Essa, S. Lee, A. S. Morcos, and D. Batra, “Emergence of maps in the memories of blind navigation agents,” in Proceedings of the International Conference on Learning Representations (ICLR) , 2023. [Online]. Available: https://arxiv.org/abs/2301.13261

2023 arXiv
[235]

ORB-SLAM: A versatile and accurate monocular SLAM system,

R. Mur-Artal, J. M. M. Montiel, and J. D. Tard´ os, “ORB-SLAM: A versatile and accurate monocular SLAM system,” IEEE Transactions on Robotics , vol. 31, pp. 1147–1163, 2015. [Online]. Available: https: //ieeexplore.ieee.org/document/7219438

2015
[236]

Learning semantic maps from natural language descriptions,

M. R. Walter, S. Hemachandra, B. Homberg, S. Tellex, and S. J. Teller, “Learning semantic maps from natural language descriptions,” The International Journal of Robotics Research , 2013. [Online]. Available: https://www.roboticsproceedings.org/rss09/p04.pdf

2013
[237]

Information-theoretic dialog to improve spatial-semantic representations,

S. Hemachandra and M. R. Walter, “Information-theoretic dialog to improve spatial-semantic representations,” in Proceedings of the IEEE/RSJ 139 International Conference on Intelligent Robots and Systems (IROS) , 2015. [Online]. Available: https://ieeexplore.ieee.org/document/7354097

2015
[238]

Interactive fiction games: A colossal adventure,

M. Hausknecht, P. Ammanabrolu, M.-A. Cˆ ot´ e, and X. Yuan, “Interactive fiction games: A colossal adventure,” in Proceedings of the National Conference on Artificial Intelligence (AAAI) , 2020. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/6297/6153

2020
[239]

Language models are few-shot learners,

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Che...

2020
[240]

Learning to summarize with human feedback,

N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano, “Learning to summarize with human feedback,” in Advances in Neural Information Processing Systems (NeurIPS) ,
[241]

Scaling laws for reward model overoptimization,

L. Gao, J. Schulman, and J. Hilton, “Scaling laws for reward model overoptimization,” arXiv preprint arXiv:2210.10760 , 2022. [Online]. Available: https://arxiv.org/abs/2210.10760

2022 arXiv
[242]

Introducing Claude,

Anthopic, “Introducing Claude,” https://www.anthropic.com/news/ introducing-claude, 2023, accessed: March 1, 2024. [Online]. Avail- able: https://www.anthropic.com/news/introducing-claude

2023
[243]

Model card and evaluations for Claude models,

——, “Model card and evaluations for Claude models,” https: //www-cdn.anthropic.com/bd2a28d2535bfb0494cc8e2a3bf135d2e7523226/ Model-Card-Claude-2.pdf, 2023, accessed: March 1,

2023
[245]

R WKV: Reinventing RNNs for the transformer era,

B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, H. Cao, X. Cheng, M. Chung, M. Grella, K. K. GV, X. He, H. Hou, P. Kazienko, J. Kocon, J. Kong, B. Koptyra, H. Lau, K. S. I. Mantri, F. Mom, A. Saito, X. Tang, B. Wang, J. S. Wind, S. Wozniak, R. Zhang, Z. Zhang, Q. Zh...

2023 arXiv
[246]

Chain of thought prompting elicits reasoning in large language models,

J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou, “Chain of thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems (NeurIPS) , 2022. [Online]. Available: https://arxiv.org/abs/2201.11903.pdf

2022 arXiv
[247]

Available: https://arxiv.org/abs/2009.01325

[Online]. Available: https://arxiv.org/abs/2009.01325

2009 arXiv
[248]

Learning transferable visual models from natural language supervision,

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in Proc. Int’l Conf. on Machine Learning (ICML) , 2021

2021
[249]

Compositional visual generation with composable diffusion models,

N. Liu, S. Li, Y. Du, A. Torralba, and J. B. Tenenbaum, “Compositional visual generation with composable diffusion models,” in Proc. Int’l. Conf. on Computer Vision (ICCV) , 2022

2022
[250]

Objaverse-XL: A universe of 10M+ 3D objects,

M. Deitke, R. Liu, M. Wallingford, H. Ngo, O. Michel, A. Kusupati, A. Fan, C. Laforte, V. S. Voleti, S. Y. Gadre, E. VanderBilt, A. Kembhavi, C. Vondrick, G. Gkioxari, K. Ehsani, L. Schmidt, and A. Farhadi, “Objaverse-XL: A universe of 10M+ 3D objects,” arXiv preprint arXiv:23...

2023 arXiv
[252]

ScanRefer: 3D object localization in RGB-D scans using natural language,

D. Z. Chen, A. X. Chang, and M. Nießner, “ScanRefer: 3D object localization in RGB-D scans using natural language,” arXiv preprint arXiv:1912.08830 , 2019

1912 arXiv
[253]

Available: https://arxiv.org/abs/2307.09288 140

[Online]. Available: https://arxiv.org/abs/2307.09288 140

arXiv
[254]

SAT: 2D semantics assisted training for 3D visual grounding,

Z. Yang, S. Zhang, L. Wang, and J. Luo, “SAT: 2D semantics assisted training for 3D visual grounding,” in Proc. Int’l. Conf. on Computer Vision (ICCV) , 2021

2021
[255]

Learning point- language hierarchical alignment for 3D visual grounding,

J. Chen, W. Luo, R. Song, X. Wei, L. Ma, and W. Zhang, “Learning point- language hierarchical alignment for 3D visual grounding,” in Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , 2022

2022
[256]

ReferIt3D: Neural listeners for fine-grained 3D object identification in real- world scenes,

P. Achlioptas, A. Abdelreheem, F. Xia, M. Elhoseiny, and L. J. Guibas, “ReferIt3D: Neural listeners for fine-grained 3D object identification in real- world scenes,” in Proc. Int’l. Conf. on Computer Vision (ICCV) , 2020

2020
[257]

NS3D: Neuro-symbolic grounding of 3D objects and relations,

J. Hsu, J. Mao, and J. Wu, “NS3D: Neuro-symbolic grounding of 3D objects and relations,” in Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2023

2023
[258]

D3Net: A unified speaker- listener architecture for 3D dense captioning and visual grounding,

D. Z. Chen, Q. Wu, M. Nießner, and A. X. Chang, “D3Net: A unified speaker- listener architecture for 3D dense captioning and visual grounding,” in Proc. Int’l. Conf. on Computer Vision (ICCV) , 2021

2021
[259]

3DJCG: A unified framework for joint dense captioning and visual grounding on 3D point clouds,

D. Cai, L. Zhao, J. Zhang, L. Sheng, and D. Xu, “3DJCG: A unified framework for joint dense captioning and visual grounding on 3D point clouds,” in Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , 2022

2022
[260]

Language conditioned spatial relation reasoning for 3D object grounding,

S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev, “Language conditioned spatial relation reasoning for 3D object grounding,” arXiv preprint arXiv:2211.09646, 2022

2022 arXiv
[261]

Visual programming: Compositional visual reasoning without training,

T. Gupta and A. Kembhavi, “Visual programming: Compositional visual reasoning without training,” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 14 953–14 962, 2022

2023
[262]

Multi-view transformer for 3D visual grounding,

S. Huang, Y. Chen, J. Jia, and L. Wang, “Multi-view transformer for 3D visual grounding,” in Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2022. 141

2022
[263]

SayPlan: Grounding large language models using 3D scene graphs for scalable task planning,

K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. D. Reid, and N. S¨ underhauf, “SayPlan: Grounding large language models using 3D scene graphs for scalable task planning,” arXiv preprint arXiv:2307.06135 , 2023

2023 arXiv
[264]

3D- LLM: Injecting the 3D world into large language models,

Y. Hong, H. Zhen, P. Chen, S. Zheng, Y. Du, Z. Chen, and C. Gan, “3D- LLM: Injecting the 3D world into large language models,” arXiv preprint arXiv:2307.12981, 2023

2023 arXiv
[265]

Attention is all you need,

A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS) , 2017

2017
[266]

LayoutGPT: Compositional visual planning and generation with large language models,

W. Feng, W. Zhu, T.-J. Fu, V. Jampani, A. R. Akula, X. He, S. Basu, X. E. Wang, and W. Y. Wang, “LayoutGPT: Compositional visual planning and generation with large language models,” arXiv preprint:2305.15393, 2023

2023 arXiv
[267]

Co-NavGPT: Multi-robot cooperative visual semantic navigation using large language models,

B. Yu, H. Kasaei, and M. Cao, “Co-NavGPT: Multi-robot cooperative visual semantic navigation using large language models,” arXiv preprint arXiv:2310.07937, 2023

2023 arXiv
[268]

LLM-Grounder: Open-vocabulary 3D visual grounding with large language model as an agent,

J. Yang, X. Chen, S. Qian, N. Madaan, M. Iyengar, D. F. Fouhey, and J. Chai, “LLM-Grounder: Open-vocabulary 3D visual grounding with large language model as an agent,” arXiv preprint arXiv:2309.12311 , 2023

2023 arXiv
[269]

Tree of thoughts: Deliberate problem solving with large language models,

S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” arXiv preprint arXiv:2305.10601 , 2023

2023 arXiv
[270]

Statler: State-maintaining language models for embodied reasoning,

T. Yoneda, J. Fang, P. Li, H. Zhang, T. Jiang, S. Lin, B. Picker, D. Yunis, H. Mei, and M. R. Walter, “Statler: State-maintaining language models for embodied reasoning,” arXiv preprint arXiv:2306.17840 , 2023

2023 arXiv
[271]

Do as I can, not as I say: Grounding language in robotic affordances,

M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. C. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Lev...

2022
[272]

Reflexion: Language agents with verbal reinforcement learning,

N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” Advances in Neural Information Processing Systems, vol. 36, 2024

2024
[273]

PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding,

K. Mo, S. Zhu, A. X. Chang, L. Yi, S. Tripathi, L. J. Guibas, and H. Su, “PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding,” in Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , 2018. 143

2018
[274]

Decomposing NeRF for editing via feature field distillation,

S. Kobayashi, E. Matsumoto, and V. Sitzmann, “Decomposing NeRF for editing via feature field distillation,” arXiv preprint arXiv:2205.15585 , 2022

2022 arXiv
[275]

Faith and fate: Limits of transformers on compo- sitionality,

N. Dziri, X. Lu, M. Sclar, X. L. Li, L. Jian, B. Y. Lin, P. West, C. Bhaga- vatula, R. L. Bras, J. D. Hwang, S. Sanyal, S. Welleck, X. Ren, A. Ettinger, Z. Harchaoui, and Y. Choi, “Faith and fate: Limits of transformers on compo- sitionality,” arXiv preprint arXiv:2305.18654 , 2023

2023 arXiv
[276]

Procedures as a representation for data in a computer pro- gram for understanding natural language,

T. Winograd, “Procedures as a representation for data in a computer pro- gram for understanding natural language,” Ph.D. dissertation, Massachusetts Institute of Technology, 1971

1971
[277]

Robots that use language,

S. Tellex, N. Gopalan, H. Kress-Gazit, and C. Matuszek, “Robots that use language,” Annual Review of Control, Robotics, and Autonomous Systems , vol. 3, pp. 25–55, 2020

2020
[278]

Code as policies: Language model programs for embodied control,

J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. R. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,” arXiv preprint arXiv:2209.07753 , 2022

2022 arXiv
[279]

MDETR - Modulated detection for end-to-end multi-modal understanding,

A. Kamath, M. Singh, Y. LeCun, G. Synnaeve, I. Misra, and N. Carion, “MDETR - Modulated detection for end-to-end multi-modal understanding,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2021

2021
[280]

React: Synergizing reasoning and acting in language models,

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629, 2022

2022 arXiv
[281]

Large language models are zero-shot reasoners,

T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” arXiv preprint arXiv:2205.11916 , 2022

2022 arXiv
[282]

Evalu- ating large language models trained on code,

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, ...

2021 arXiv
[283]

Inferring compact representations for efficient natural language understanding of robot instruc- tions,

S. Patki, A. F. Daniele, M. R. Walter, and T. M. Howard, “Inferring compact representations for efficient natural language understanding of robot instruc- tions,” in Proc. IEEE Int’l Conf. on Robotics and Automation (ICRA) , May 2019

2019
[284]

Inner monologue: Embodied reasoning through planning with language models,

W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. R. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, P. Sermanet, N. Brown, T. Jackson, L. Luu, S. Levine, K. Hausman, and B. Ichter, “Inner monologue: Embodied reasoning through planning with language models,” in Proceed...

2022
[285]

A survey on domain- specific languages in robotics,

A. Nordmann, N. Hochgeschwender, and S. B. Wrede, “A survey on domain- specific languages in robotics,” in Simulation, Modeling, and Programming for Autonomous Robots, 2014

2014
[286]

R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction . Cambridge, MA: MIT Press, 1998

1998
[287]

Koller and N

D. Koller and N. Friedman, Probabilistic graphical models: principles and techniques. MIT press, 2009

2009
[288]

Anthropic introducing 100k Context windows,

“Anthropic introducing 100k Context windows,” https://www.anthropic.com/ index/100k-context-windows, accessed: 2023-05-11

2023
[289]

Group-free 3D object detection via transformers,

Z. Liu, Z. Zhang, Y. Cao, H. Hu, and X. Tong, “Group-free 3D object detection via transformers,” in Proc. Int’l. Conf. on Computer Vision (ICCV) , 2021

2021
[290]

Do long-range language models actually use long-range context?

S. Sun, K. Krishna, A. Mattarella-Micke, and M. Iyyer, “Do long-range language models actually use long-range context?” arXiv preprint arXiv:2109.09115 , 2021

2021 arXiv
[291]

On the planning abilities of large language models (a critical investigation with a proposed benchmark),

K. Valmeekam, S. Sreedharan, M. Marquez, A. Olmo, and S. Kambhampati, “On the planning abilities of large language models (a critical investigation with a proposed benchmark),” arXiv preprint arXiv:2302.06706 , 2023

2023 arXiv
[292]

GPT-4 technical report,

OpenAI, “GPT-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

2023 arXiv
[298]

Lost in the middle: How language models use long contexts,

N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” arXiv preprint arXiv:2307.03172, 2023

2023 arXiv
[2006]

Available: https://pubmed.ncbi.nlm.nih.gov/16584892/

[Online]. Available: https://pubmed.ncbi.nlm.nih.gov/16584892/
[2013]

Available: https://aclanthology.org/D13-1020/

[Online]. Available: https://aclanthology.org/D13-1020/
[2020]

Available: https://arxiv.org/abs/1906.05948

[Online]. Available: https://arxiv.org/abs/1906.05948

1906 arXiv
[2021]

Available: https://arxiv.org/abs/2103.07191

[Online]. Available: https://arxiv.org/abs/2103.07191

arXiv
[2022]

Available: http://dx.doi.org/10.1111/cgf.14505

[Online]. Available: http://dx.doi.org/10.1111/cgf.14505

doi
[2023]

Available: https://arxiv.org/abs/2303.12712.pdf

[Online]. Available: https://arxiv.org/abs/2303.12712.pdf

arXiv
[2024]

Available: https://www-cdn.anthropic.com/ bd2a28d2535bfb0494cc8e2a3bf135d2e7523226/Model-Card-Claude-2.pdf

[Online]. Available: https://www-cdn.anthropic.com/ bd2a28d2535bfb0494cc8e2a3bf135d2e7523226/Model-Card-Claude-2.pdf

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

The double sphere camera model,

V. Usenko, N. Demmel, and D. Cremers, “The double sphere camera model,” in Proceedings of the International Conference on 3D Vision (3DV) , 2018, pp. 552–560

2018

[2] [2]

Visual- inertial mapping with non-linear factor recovery,

V. Usenko, N. Demmel, D. Schubert, J. Stueckler, and D. Cremers, “Visual- inertial mapping with non-linear factor recovery,” IEEE Robotics and Automa- tion Letters, vol. 5, no. 2, pp. 422–429, 2020

2020

[3] [3]

The EuRoC micro aerial vehicle datasets,

M. Burri, J. Nikolic, P. Gohl, T. Schneider, J. Rehder, S. Omari, M. W. Achtelik, and R. Siegwart, “The EuRoC micro aerial vehicle datasets,” International Journal of Robotics Research, vol. 35, no. 10, pp. 1157–1163, 2016

2016

[4] [4]

Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras,

A. Gordon, H. Li, R. Jonschkowski, and A. Angelova, “Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019

2019

[5] [5]

PointGroup: Dual-set point grouping for 3D instance segmentation,

L. Jiang, H. Zhao, S. Shi, S. Liu, C.-W. Fu, and J. Jia, “PointGroup: Dual-set point grouping for 3D instance segmentation,” in Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , 2020

2020

[6] [6]

Mask3D: Mask transformer for 3D semantic instance segmentation,

J. Schult, F. Engelmann, A. Hermans, O. Litany, S. Tang, and B. Leibe, “Mask3D: Mask transformer for 3D semantic instance segmentation,” in Proc. IEEE Int’l Conf. on Robotics and Automation (ICRA) , 2023

2023

[7] [7]

What’s “up

A. Kamath, J. Hessel, and K.-W. Chang, “What’s “up” with vision-language models? investigating their struggle with spatial reasoning,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

2023

[8] [8]

Spa- tialvlm: Endowing vision-language models with spatial reasoning capabilities,

B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia, “Spa- tialvlm: Endowing vision-language models with spatial reasoning capabilities,” 115 in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

2024