Pith. sign in

REVIEW 3 major objections 5 minor 94 references

From reactive to cognitive: brain-inspired spatial intelligence for embodied agents

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BSC-Nav claims that an explicit landmark-and-map memory, modeled on biological spatial cognition, lets embodied agents navigate zero-shot where reactive agents fail.

desk verdict A well-engineered memory architecture that deserves a serious referee, but the headline SOTA numbers rest on a pre-exploration protocol that privileges BSC-Nav over the baselines. read the letter →

arxiv 2508.17198 v1 pith:DZILCR3V submitted 2025-08-24 cs.AI

Shouwei Ruan , Liyuan Wang , Caixin Kang , Qihui Zhu , Songming Liu , Xingxing Wei , Hang Su This is my paper · ORCID
classification cs.AI
keywords spatialcognitionstructuredmemorylandmarkcognitivemapembodiednavigationzero-shotgeneralizationvision-languagemodelsmobilemanipulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that embodied agents fail at general spatial tasks because they process each observation reactively instead of building persistent knowledge, and that a biologically motivated memory architecture closes that gap. BSC-Nav separates spatial knowledge into three stored forms — landmarks (salient objects with coordinates and descriptions), route knowledge (egocentric trajectories), and survey knowledge (an allocentric, map-like voxel grid) — and a working-memory module retrieves the relevant records with the help of multimodal large language models, which combine vision and language. In experiments across object-goal, open-vocabulary, instance-level, and instruction-following navigation, plus embodied question answering, the authors report that this memory-based agent outperforms prior reactive and map-based systems on both success rate and path efficiency, and that the same stack transfers to a physical robot for navigation and mobile manipulation. The claim worth noticing is that explicit structured memory, rather than a larger end-to-end policy, is what makes spatial behavior generalize.

What carries the argument

The load-bearing mechanism is a structured spatial memory built from two continuously updated branches and read by a working-memory module. Landmark memory stores entries of the form (world coordinates, open-vocabulary category, detection confidence, contextual description) and fuses overlapping detections so each instance appears once. Cognitive map memory projects patch-level features from a self-supervised vision encoder into a voxel grid, accumulating route knowledge from egocentric trajectories, and updates each voxel with a surprise-driven rule: a new feature is stored only if its cosine distance to nearby buffered features exceeds a threshold, and it replaces the least surprising entry, keeping memory compact and diverse. At query time the working-memory module performs hierarchical retrieval: for simple goals, a text-only LLM reasons over landmark memory to propose candidate coordinates; for instance-level or ambiguous goals, the LLM enriches the description, a diffusion model renders an imagined image of the target, and those visual features are matched against the voxel grid, clustered, and ranked by a weighted score of existence probability and distance. Low-level movement is handed to heuristic planners, and an MLLM verifies target arrival and generates fine-tuning actions.

What would settle it

Run the leading baselines under the same frontier-exploration protocol before evaluation and give them the same memory budget, or count BSC-Nav's exploration path length in its SPL; if the reported success and efficiency gaps, for instance 78.5% versus 54.5% success rate, largely vanish, the advantage is an artifact of the pre-task exploration grant rather than of the structured memory itself.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that an agent equipped with structured spatial memory can navigate to goals it was never trained on by treating navigation as memory retrieval rather than reactive mapping from pixels to actions. The evidence is a sweep of 8,195 episodes across 62 indoor scenes: 78.5% success on object-goal navigation in one scene collection versus 54.5% for the prior best method, near-doubled success on text-instance navigation, 71.4% success on image-instance navigation, 38.5% success on zero-shot long-horizon instruction following, and 54.6 on an embodied question-answering metric, still 27.5 points below human performance. The same memory stack, deployed on a wheeled robot with a manipulator, completes object-goal, text-instance, and image-instance navigation plus multi-waypoint manipulation in a roughly 200 m² indoor space, which the authors take as evidence that the mechanism is not simulation-bound.

Load-bearing premise

The comparisons assume that BSC-Nav's pre-task frontier-exploration phase, which can cover up to half the traversable area to build its landmark and map memories, is a fair precondition while the baselines are not given a comparable exploration budget.

Editorial extensions

If this is right

  • An embodied agent without task-specific training can navigate to category, open-vocabulary, text-described, and image-specified targets by retrieving from one shared memory.
  • The same policy stack transfers across benchmarks: object-goal, instance-level, instruction-following, and question-answering tasks are solved by changing only the query and the MLLM prompt.
  • Composite distance-plus-confidence ranking makes navigation efficient, since most successes occur at the first candidate coordinate visited.
  • Structured memory supports multi-step mobile manipulation by linking each waypoint to a manipulation primitive under language instruction.
  • Memory-driven exploration improves spatial question answering, especially for questions requiring precise localization, while a sizable gap to human performance remains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the pre-task frontier-exploration phase is counted as part of the agent's inference budget rather than free startup, BSC-Nav's advertised zero-shot advantage may shrink; the paper does not report this accounting.
  • The surprise-driven update should make memory resilient to changed scenes, but the evaluated environments are mostly static, so a direct test with moved furniture or occluded landmarks would show whether the mechanism earns its name.
  • The architecture suggests a modular route to embodied intelligence: pair frozen perception and language components with a small memory controller, rather than training one giant policy that must memorize space in its weights.
  • The paper's closing proposal of a spatial-cognition benchmark could be operationalized by measuring how quickly an agent updates its map after partial reconfiguration and whether it can plan a detour from memory alone.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. BSC-Nav is a modular embodied-navigation framework that instantiates the biological triad of landmark, route, and survey knowledge as structured spatial memory. It constructs a landmark memory from open-vocabulary detections and a voxelized cognitive map from DINOv2 features projected along egocentric trajectories, with a surprise-driven update rule, and uses a working-memory module to retrieve and compose these representations through MLLMs (GPT-4o/GPT-o3) and Stable Diffusion. The paper reports large gains over prior modular and end-to-end baselines across object-goal navigation (OGN), open-vocabulary object navigation (OVON), text-instance navigation (TIN), image-instance navigation (IIN), VLN-CE instruction following, and A-EQA, plus real-robot navigation and manipulation demonstrations. The central claim is that BSC-Nav achieves state-of-the-art efficacy and efficiency while exhibiting strong zero-shot generalization.

Significance. If the reported comparisons were on equal footing, BSC-Nav would be a meaningful step toward memory-centric embodied agents: it is a concrete, interpretable architecture that separates perception, memory construction, and planning, and it is evaluated on an unusually broad set of benchmarks with a real-robot transfer and released code. The biological grounding (landmark/route/survey knowledge, free-energy-inspired update) is a useful framing rather than a mathematical contribution, and the paper is honest about remaining gaps to human performance. However, the significance hinges on whether the evaluation protocol is fair; the current protocol gives BSC-Nav a privileged pre-built map of each test scene, so the headline margins and zero-shot claims are not yet established as claimed.

major comments (3)
  1. [Methods, 'Spatial memory construction before task execution' (pp. 22-23); Metrics, Eq. (14)] The central SOTA claim rests on an unequal comparison. BSC-Nav pre-explores each scene via a frontier-based strategy until an iteration limit of half the traversable area, constructing landmark memory and cognitive map before any task episode begins, while the described baselines (PixNav, DAgRL, PSL, VLFM, MOD-IIN, UniGoal, GOAT, Uni-Navid) start reactively from the episode's initial observation. The SPL metric in Eq. (14) charges only the post-exploration trajectory length, and the agent's starting pose after exploration may differ from the benchmark's canonical episode start, so neither the exploration distance nor the displacement from the original start is penalized. The reported margins (e.g., 78.5% vs. 54.5% SR on HM3D OGN) therefore conflate the architecture's structured memory with the advantage of having already seen up to half the traversable area. To support the paper's claims, the authors should either (i) remove the pre-exploration phase, (ii) charge its full cost to SPL and SR, or (iii) grant the same pre-exploration budget to all baselines; otherwise the comparison should be reframed as navigation with a pre-acquired cognitive map rather than from-scratch reactive-to-cognitive navigation.
  2. [Abstract and 'Results: Generalization to instruction-driven navigation' (VLN-CE, Sec. 3, pp. 8-9)] The term 'zero-shot' is used for results that are not zero-shot under the standard convention. The agent pre-explores the test scene during spatial-memory construction, which is environment-specific adaptation; the VLN-CE 'zero-shot' SR of 38.5% and the OVON 'zero-shot' numbers are obtained after the agent has seen the test environment's layout and landmarks. A zero-shot claim normally excludes any environment-specific adaptation, and several baselines, such as VLFM and UniGoal, are also described as zero-shot. The paper should either remove the 'zero-shot' label, explicitly define it as 'no gradient-based or task-specific supervised training,' or run the evaluation without any test-time pre-exploration.
  3. [Methods, 'Spatial memory construction before task execution' and 'Active embodied question answering' (pp. 22-23, 9-10)] The same pre-exploration bias affects the A-EQA comparison. BSC-Nav is stated to require environmental perception before task execution to build preliminary landmark memory and cognitive map, and this presumably applies to the A-EQA episodes as well. The baselines—a blind LLM, a question-agnostic frontier exploration strategy, and Explore-EQA—are active-exploration methods that build their knowledge from the episode start without a pre-built map. The reported LLM-Match advantage (54.6 vs. the baselines) may thus partly reflect the pre-constructed memory rather than the agent's active-exploration and reasoning policy. The authors should clarify whether BSC-Nav pre-explores for A-EQA, and if so, compare against baselines that receive the same pre-exploration or report the cost of that exploration.
minor comments (5)
  1. [Supplementary Fig. 2] The ablation panel labels 'B', 'Q', and 'B+Q' are not defined in the caption or main text; the reader has to infer they mean landmark-memory-only, cognitive-map-only, and both, respectively. Please add explicit definitions.
  2. [Results, Figs. 3-4] No variance, confidence intervals, or multiple-seed runs are reported for SR/SPL. Given that some reported differences are small (e.g., 40.2% vs. 38.9% SR on OVON seen vs. unseen), the paper would benefit from at least per-episode bootstrap intervals or a statement that the differences are statistically significant.
  3. [Methods, 'Spatial memory construction before task execution' (p. 23)] The sentence 'The iteration count is defined as half of the traversable area' is dimensionally unclear: traversable area is a quantity in m^2, while an iteration count is an integer. Please clarify the exact stopping rule in terms of the number of frontier points or the ratio of explored area to total traversable area.
  4. [Eq. (5) and Methods, 'Landmark memory'] The landmark fusion rule uses a fixed spatial overlap distance of 1.0 m. For large or elongated objects (e.g., sofas, beds), detections of the same object from different viewpoints may be more than 1.0 m apart and thus be treated as distinct landmarks; please justify the threshold or report sensitivity to it.
  5. [References] Uni-Navid is cited twice, as reference [12] and again as [76]; please deduplicate. Also, reference [59] appears to be for a different arXiv ID than the author list suggests; please verify the citation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: BSC-Nav's central claims rest on external benchmark comparisons, and no reported metric reduces by the paper's own equations to a fitted constant or self-citation.

full rationale

The paper's derivation chain is self-contained against external benchmarks and does not exhibit any of the enumerated circular patterns. BSC-Nav's memory modules are defined operationally (landmark tuples in Eq. 1, voxelized cognitive map in Eq. 6, retrieval scoring in Eq. 13), and the headline SR/SPL numbers are measured on public Habitat benchmarks (MP3D, HM3D, VLN-CE R2R) against published baselines, rather than being implied by the framework's equations. Hyperparameters such as lambda=0.5, tau=0.5, confidence thresholds, voxel size, and buffer capacity are stated as defaults or user-set values, not fitted to optimize the reported metrics. The A-EQA evaluation uses the external OpenEQA protocol with text-only GPT-4 as the official LLM-Match judge; this is the benchmark's own metric, and although the agent also uses GPT models, the result is an externally defined correctness score rather than a quantity that reduces to the paper's inputs by construction. The pre-task frontier exploration (up to half the traversable area) is a protocol-comparability concern about whether baselines are evaluated on equal footing, and it may weaken the 'zero-shot' label, but it is not a circular derivation: exploration builds the memory that is then used, and the SPL formula (Eq. 14) measures a real executed trajectory. No load-bearing uniqueness theorem, ansatz, or fitted parameter is imported from the authors' prior work, and no self-citation chain forces the stated conclusions. The paper even reports an honest performance gap versus humans on A-EQA rather than claiming a derivation that must succeed. Overall, no step in the claimed prediction chain is equivalent by definition to its own input.

Assumptions & free parameters 11 free parameters · 6 assumptions · 0 invented entities

No new physical entities are introduced. The free parameters are all hand-set engineering choices for thresholds, voxelization, and retrieval, and they affect the headline numbers without sensitivity analysis. The axioms are standard planners plus domain assumptions about foundation models and the pre-exploration protocol.

free parameters (11)
  • Composite scoring weight lambda = 0.5 (default)
    Eq. 13 blends confidence and normalized distance; the default is hand-set and no sensitivity analysis is reported.
  • Surprise threshold tau = 0.5 (default)
    Eq. 11 decides whether a new DINOv2 feature enters a voxel buffer; hand-set.
  • Voxel size Delta = 0.1 m
    Spatial resolution of the cognitive map; hand-set in methods.
  • Voxel grid dimension N = 1000
    Grid extent in methods; hand-set.
  • Per-voxel buffer capacity K = 10
    Bounded feature buffer per voxel; hand-set.
  • Detection confidence threshold = 0.55
    Filters YOLO-World landmark detections; hand-set.
  • Spatial overlap distance for landmark fusion = 1.0 m
    Determines when two detections are treated as the same landmark; hand-set.
  • Maximum retrieved candidates k = 3
    Number of candidate coordinates from both memory branches; hand-set.
  • Number of Stable Diffusion images per batch = 3
    Used to mitigate stochasticity in imagined prototypes; hand-set.
  • Exploration iteration limit = half of traversable area
    Stops pre-task exploration; arbitrary, affects map quality.
  • Center-distance pooling temperature alpha
    Eq. 12 uses an exponential temperature to weight patch features toward the image center; the value is not reported.
assumptions (6)
  • domain assumption YOLO-World open-vocabulary detections reliably identify salient landmarks in indoor scenes.
    Landmark memory depends on the detector returning correct object classes and bounding boxes (Methods: Landmark memory).
  • domain assumption DINOv2 patch features are sufficiently discriminative to localize and match target objects across viewpoints.
    Cognitive map retrieval equates semantic and instance identity with cosine similarity of DINOv2 features (Methods: Cognitive map).
  • domain assumption GPT-4, GPT-4o, and GPT-o3 produce accurate contextual descriptions, instruction decompositions, and target verification judgments.
    Working memory retrieval, VLN decomposition, and final success verification all outsource reasoning to these closed models (Supplementary prompts).
  • domain assumption Stable Diffusion-generated imagined images of a target preserve enough visual similarity to the real object for DINOv2 matching.
    Association-enhanced retrieval relies on the diffusion model's image aligning with the real instance (Methods: Working memory).
  • standard math The Habitat simulator's greedy mesh shortest-path planner and A*/TEB provide valid low-level motion policies.
    Low-level control is delegated to these standard planners; the central claim is about high-level memory, not control.
  • ad hoc to paper The pre-task frontier exploration builds a representative map without biasing the navigation comparison.
    Evaluation grants BSC-Nav up to half of traversable area as exploration before episodes; this is a protocol choice specific to this paper (Methods: Spatial memory construction).

how reviews work

0 comments
Cite this review

Pith. "Pith review of From reactive to cognitive: brain-inspired spatial intelligence for embodied agents." pith.science (2026). https://pith.science/paper/DZILCR3V

@misc{pith2026250817198,
  author       = {Pith},
  title        = {Pith review of: From reactive to cognitive: brain-inspired spatial intelligence for embodied agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DZILCR3V}},
  note         = {Machine review of arXiv:2508.17198}
}
read the original abstract

Spatial cognition enables adaptive goal-directed behavior by constructing internal models of space. Robust biological systems consolidate spatial knowledge into three interconnected forms: \textit{landmarks} for salient cues, \textit{route knowledge} for movement trajectories, and \textit{survey knowledge} for map-like representations. While recent advances in multi-modal large language models (MLLMs) have enabled visual-language reasoning in embodied agents, these efforts lack structured spatial memory and instead operate reactively, limiting their generalization and adaptability in complex real-world environments. Here we present Brain-inspired Spatial Cognition for Navigation (BSC-Nav), a unified framework for constructing and leveraging structured spatial memory in embodied agents. BSC-Nav builds allocentric cognitive maps from egocentric trajectories and contextual cues, and dynamically retrieves spatial knowledge aligned with semantic goals. Integrated with powerful MLLMs, BSC-Nav achieves state-of-the-art efficacy and efficiency across diverse navigation tasks, demonstrates strong zero-shot generalization, and supports versatile embodied behaviors in the real physical world, offering a scalable and biologically grounded path toward general-purpose spatial intelligence.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

94 extracted references · 58 canonical work pages

  1. [1]

    ����������� ���������� ������ This task [72] comprises two bench- marks, including (i) 2,195 episodes across 34 HM3D scenes involving 6 object categories, and (ii) 2,000 episodes across 10 MP3D scenes with 20 object categories

  2. [2]

    We sampled 1,000 episodes 21 each from the validation-seen and validation-unseen splits

    ��������������� ������ ���������� ������� This task [41] extends to 79 open-vocabulary categories across 10 MP3D scenes to address the lim- ited object categories in object-goal navigation. We sampled 1,000 episodes 21 each from the validation-seen and validation-unseen splits. The seen split includes categories present in training (though not exact insta...

  3. [3]

    ������������� ���������� ������ This task [42] provides natural lan- guage descriptions for 795 instances across 36 HM3D scenes. Descriptions encompass both intrinsic attributes (inherent object properties including shape, color, and material) and extrinsic attributes (surrounding environ- mental contexts), annotated with a strong MLLM termed CogVLM [73]....

  4. [4]

    We sampled 1,000 episodes from the validation split

    �������������� ���������� ������ This task [43] employs single-view rendered images as navigation targets for instances across 34 HM3D scenes. We sampled 1,000 episodes from the validation split. Beyond these foundational navigation tasks, we further evaluate BSC-Nav and baseline methods on higher-level spatially-aware skills:

  5. [5]

    ������������ ����������������� ���������� ������ We employ the VLN-CE Room-to-Room (R2R) benchmark [50], which provides 1,000 nav- igation episodes in Habitat-lab based on human-annotated long-horizon instructions across 11 MP3D scenes

  6. [6]

    Agents are initialized at the first frame of each recorded exploration trajectory and are allowed to actively explore the environment to answer a given spatially grounded question

    ������ �������� �������� ��������� �������� We develop a cus- tom evaluation pipeline within Habitat-lab using OpenEQA [54]. Agents are initialized at the first frame of each recorded exploration trajectory and are allowed to actively explore the environment to answer a given spatially grounded question. We evaluate the 184 test queries that span seven ta...

  7. [7]

    Nature Communications �� (1), 5721 (2021)

    Gupta, A., Savarese, S., Ganguli, S., Fei-Fei, L.: Embodied intelligence via learning and evolution. Nature Communications �� (1), 5721 (2021)

    2021

  8. [8]

    NPJ Science of Learning �(1), 9 (2020)

    Malanchini, M., Rimfeld, K., Shakeshaft, N.G., McMillan, A., Schofield, K.L., Rodic, M., Rossi, V., Kovas, Y., Dale, P.S., Tucker-Drob, E.M., �� ���: Evidence for a unitary structure of spatial cognition beyond general intelligence. NPJ Science of Learning �(1), 9 (2020)

    2020

Show all 94 references
  1. [9]

    Science ��� (6415), 6766 (2018)

    Bellmund, J.L., G¨ ardenfors, P., Moser, E.I., Doeller, C.F.: Navigating cognition: Spatial codes for human thinking. Science ��� (6415), 6766 (2018)

    2018

  2. [10]

    Nature neuroscience �� (11), 1504–1513 (2017)

    Epstein, R.A., Patai, E.Z., Julian, J.B., Spiers, H.J.: The cognitive map in humans: spatial navigation and beyond. Nature neuroscience �� (11), 1504–1513 (2017)

    2017

  3. [11]

    Springer (2007)

    Denis, M., Loomis, J.M.: Perspectives on human spatial cognition: memory, navigation, and environmental learning. Springer (2007)

    2007

  4. [12]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp

    Yang, J., Yang, S., Gupta, A.W., Han, R., Fei-Fei, L., Xie, S.: Thinking in space: How multimodal large language models see, remember, and recall spaces. In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 10632–10643 (2025)

    2025

  5. [13]

    arXiv preprint arXiv:2401.10279 (2024)

    Tucker, S.: A systematic review of geospatial location embedding approaches in large language models: A path to spatial ai systems. arXiv preprint arXiv:2401.10279 (2024)

    2024 arXiv

  6. [14]

    Springer

    Papadimitriou, F.: Spatial Artificial Intelligence. Springer

  7. [15]

    Nature ��� (7705), 429–433 (2018)

    Banino, A., Barry, C., Uria, B., Blundell, C., Lillicrap, T., Mirowski, P., Pritzel, A., Chadwick, M.J., Degris, T., Modayil, J., �� ���: Vector- based navigation using grid-like representations in artificial agents. Nature ��� (7705), 429–433 (2018)

    2018

  8. [16]

    Frontiers in Computational Neuroscience �� , 63 (2020)

    Bermudez-Contreras, E., Clark, B.J., Wilber, A.: The neuroscience of spa- tial navigation and the relationship to artificial intelligence. Frontiers in Computational Neuroscience �� , 63 (2020)

    2020

  9. [17]

    In: 2024 IEEE International Conference on Robotics and Automation (ICRA), pp

    Cai, W., Huang, S., Cheng, G., Long, Y., Gao, P., Sun, C., Dong, H.: Bridging zero-shot object navigation and foundation models through pixel-guided navigation skill. In: 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 5228–5234 (2024). IEEE

    2024

  10. [19]

    arXiv preprint arXiv:2403.08000 (2024)

    Yokoyama, K., et al.: Vlfm: Vision-language frontier maps for zero-shot semantic navigation. arXiv preprint arXiv:2403.08000 (2024)

    2024 arXiv

  11. [20]

    Ramakrishnan, S.K., Wijmans, E., Kraehenbuehl, P., Koltun, V.: Does spatial cognition emerge in frontier models? arXiv preprint arXiv:2410.06468 (2024)

    2024 arXiv

  12. [21]

    Journal of Experimental Psychology: Learning, Memory, and Cognition �� (2), 211 (1989)

    McNamara, T.P., Hardy, J.K., Hirtle, S.C.: Subjective hierarchies in spa- tial memory. Journal of Experimental Psychology: Learning, Memory, and Cognition �� (2), 211 (1989)

    1989

  13. [22]

    In: Informatik’97 Informatik Als Innovation- smotor: 27

    Werner, S., Krieg-Br¨ uckner, B., Mallot, H.A., Schweizer, K., Freksa, C.: Spatial cognition: The role of landmark, route, and survey knowledge in human and robot navigation. In: Informatik’97 Informatik Als Innovation- smotor: 27. Jahrestagung der Gesellschaft F¨ ur Informati...

    1997

  14. [23]

    Advances in Child Development and Behavior �� , 9–55 (1975)

    Siegel, A.W., White, S.H.: The development of spatial representations of large-scale environments. Advances in Child Development and Behavior �� , 9–55 (1975)

    1975

  15. [24]

    In: NeurIPS (2020)

    Chaplot, D.S., �� ���: Object goal navigation using goal-oriented semantic exploration. In: NeurIPS (2020)

    2020

  16. [25]

    In: ICCV (2022)

    Yadav, A., �� ���: Offline reinforcement learning for visual navigation. In: ICCV (2022)

    2022

  17. [26]

    arXiv preprint arXiv:2306.14846 (2023)

    Shah, D., Sridhar, A., Dashora, N., Stachowicz, K., Black, K., Hirose, N., Levine, S.: Vint: A foundation model for visual navigation. arXiv preprint arXiv:2306.14846 (2023)

    2023 arXiv

  18. [27]

    https://cdn.openai.com/papers/ GPTV System Card.pdf (2023)

    OpenAI: GPT-4V(ision) System Card. https://cdn.openai.com/papers/ GPTV System Card.pdf (2023)

    2023

  19. [28]

    arXiv preprint arXiv:2308.12966 (2023)

    Yang, H., et al.: Qwen-vl: A versatile vision-language model. arXiv preprint arXiv:2308.12966 (2023)

    2023 arXiv

  20. [29]

    Springer Cham Heidelberg New York Dordrecht London

    Richter, K.-F., Winter, S.: Landmarks. Springer Cham Heidelberg New York Dordrecht London. doi �� (978-3), 1 (2014)

    2014

  21. [30]

    Experimental Psychology �� (3), 171–181 (2006)

    Jansen-Osmann, P., Fuchs, P.: Wayfinding behavior and spatial knowledge of adults and children in a virtual environment: The role of landmarks. Experimental Psychology �� (3), 171–181 (2006)

    2006

  22. [31]

    Cognitive Brain Research �� (3), 401–411 (2004)

    Wolbers, T., Weiller, C., B¨ uchel, C.: Neural foundations of emerging route 28 knowledge in complex spatial environments. Cognitive Brain Research �� (3), 401–411 (2004)

    2004

  23. [32]

    Journal of Experi- mental Psychology: Learning, Memory, and Cognition�� (5), 1520 (2013)

    Chrastil, E.R., Warren, W.H.: Active and passive spatial learning in human navigation: acquisition of survey knowledge. Journal of Experi- mental Psychology: Learning, Memory, and Cognition�� (5), 1520 (2013)

    2013

  24. [33]

    Annual Review of Psychology �� , 1–29 (2012)

    Baddeley, A.: Working memory: theories, models, and controversies. Annual Review of Psychology �� , 1–29 (2012)

    2012

  25. [34]

    Trends in Cognitive Sciences �(3), 119–126 (2001)

    Awh, E., Jonides, J.: Overlapping mechanisms of attention and spatial working memory. Trends in Cognitive Sciences �(3), 119–126 (2001)

    2001

  26. [35]

    Psychology Press, ??? (2014)

    Logie, R.H.: Visuo-spatial Working Memory. Psychology Press, ??? (2014)

    2014

  27. [36]

    In: CVPR (2023)

    Oquab, M., �� ���: Dinov2: Learning robust visual features without supervision. In: CVPR (2023)

    2023

  28. [37]

    Darcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision Transformers Need Registers (2023)

    2023

  29. [38]

    Friston, K.: The free-energy principle: a unified brain theory? Nature reviews neuroscience �� (2), 127–138 (2010)

    2010

  30. [39]

    Friston, K.: The free-energy principle: a rough guide to the brain? Trends in cognitive sciences �� (7), 293–301 (2009)

    2009

  31. [40]

    https://openai.com/index/hello-gpt-4o/ (2024)

    OpenAI: Hello gpt-4o. https://openai.com/index/hello-gpt-4o/ (2024)

    2024

  32. [41]

    In: Forty-first International Conference on Machine Learning (2024)

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., M¨ uller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., �� ���: Scaling recti- fied flow transformers for high-resolution image synthesis. In: Forty-first International Conference on Machine Learning (2024)

    2024

  33. [42]

    arXiv preprint arXiv:1709.06158 (2017)

    Chang, A., Dai, A., Funkhouser, T., Halber, M., Niessner, M., Savva, M., Song, S., Zeng, A., Zhang, Y.: Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158 (2017)

    2017 arXiv

  34. [43]

    arXiv preprint arXiv:2109.08238 (2021)

    Ramakrishnan, S.K., Gokaslan, A., Wijmans, E., Maksymets, O., Clegg, A., Turner, J., Undersander, E., Galuba, W., Westbury, A., Chang, A.X., et al.: Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv preprint arXiv:2109.08238 (2021)

    2021 arXiv

  35. [44]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp

    Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., �� ���: Habitat: A platform for embodied ai research. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9339–9347 (2019) 29

    2019

  36. [45]

    arXiv preprint arXiv:2006.13171 (2020)

    Batra, D., Gokaslan, A., Kembhavi, A., Maksymets, O., Mottaghi, R., Savva, M., Toshev, A., Wijmans, E.: Objectnav revisited: On evaluation of embodied agents navigating to objects. arXiv preprint arXiv:2006.13171 (2020)

    2020 arXiv

  37. [46]

    IEEE Transactions on Automation Science and Engineering (2024)

    Sun, J., Wu, J., Ji, Z., Lai, Y.-K.: A survey of object goal navigation. IEEE Transactions on Automation Science and Engineering (2024)

    2024

  38. [47]

    In: 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp

    Yokoyama, N., Ramrakhya, R., Das, A., Batra, D., Ha, S.: Hm3d-ovon: A dataset and benchmark for open-vocabulary object goal navigation. In: 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5543–5550 (2024). IEEE

    2024

  39. [48]

    In: European Conference on Computer Vision, pp

    Sun, X., Liu, L., Zhi, H., Qiu, R., Liang, J.: Prioritized semantic learning for zero-shot instance navigation. In: European Conference on Computer Vision, pp. 161–178 (2024). Springer

    2024

  40. [49]

    arXiv preprint arXiv:2211.15876 (2022)

    Krantz, J., Lee, S., Malik, J., Batra, D., Chaplot, D.S.: Instance-specific image goal navigation: Training embodied agents to find object instances. arXiv preprint arXiv:2211.15876 (2022)

    2022 arXiv

  41. [50]

    Science Robotics �(79), 6991 (2023)

    Gervet, T., Chintala, S., Batra, D., Malik, J., Chaplot, D.S.: Navigating to objects in the real world. Science Robotics �(79), 6991 (2023)

    2023

  42. [51]

    arXiv preprint arXiv:1807.06757 (2018)

    Anderson, P., Chang, A., Chaplot, D.S., Dosovitskiy, A., Gupta, S., Koltun, V., Kosecka, J., Malik, J., Mottaghi, R., Savva, M., et al.: On eval- uation of embodied navigation agents. arXiv preprint arXiv:1807.06757 (2018)

    2018 arXiv

  43. [52]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp

    Krantz, J., Gervet, T., Yadav, K., Wang, A., Paxton, C., Mottaghi, R., Batra, D., Malik, J., Lee, S., Chaplot, D.S.: Navigating to objects specified by images. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10916–10925 (2023)

    2023

  44. [53]

    arXiv preprint arXiv:2503.10630 (2025)

    Yin, H., et al.: Unigoal: Towards universal zero-shot goal-oriented navi- gation. arXiv preprint arXiv:2503.10630 (2025)

    2025 arXiv

  45. [54]

    arXiv preprint arXiv:2311.06430 (2023)

    Chang, M., Gervet, T., Khanna, M., Yenamandra, S., Shah, D., Min, S.Y., Shah, K., Paxton, C., Gupta, S., Batra, D., Mottaghi, R., Malik, J., Chaplot, D.S.: Goat: Go to any thing. arXiv preprint arXiv:2311.06430 (2023)

    2023 arXiv

  46. [55]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018) 30

    Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., S¨ underhauf, N., Reid, I., Gould, S., van den Hengel, A.: Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environ- ments. In: Proceedings of the IEEE Conference on Computer...

    2018

  47. [56]

    In: European Conference on Computer Vision (ECCV) (2020)

    Krantz, J., Wijmans, E., Majundar, A., Batra, D., Lee, S.: Beyond the nav-graph: Vision and language navigation in continuous environments. In: European Conference on Computer Vision (ECCV) (2020)

    2020

  48. [57]

    https://openai.com/index/introducing-o3-and-o4-mini/ (2025)

    OpenAI: Introducing openai o3 and o4-mini. https://openai.com/index/introducing-o3-and-o4-mini/ (2025)

    2025

  49. [58]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp

    Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., Batra, D.: Embodied question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–10 (2018)

    2018

  50. [59]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Wijmans, E., Datta, S., Maksymets, O., Das, A., Gkioxari, G., Lee, S., Essa, I., Parikh, D., Batra, D.: Embodied question answering in photore- alistic environments with point cloud perception. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

    2019

  51. [60]

    In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Majumdar, A., Ajay, A., Zhang, X., Putta, P., Yenamandra, S., Henaff, M., Silwal, S., Mcvay, P., Maksymets, O., Arnaud, S., �� ���: Openeqa: Embodied question answering in the era of foundation models. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    2024

  52. [61]

    arXiv preprint arXiv:2403.15941 (2024)

    Ren, A.Z., Clark, J., Dixit, A., Itkina, M., Majumdar, A., Sadigh, D.: Explore until confident: Efficient exploration for embodied question answering. arXiv preprint arXiv:2403.15941 (2024)

    2024 arXiv

  53. [62]

    arXiv preprint arXiv:2406.09246 (2024)

    Kim, M.J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al.: Openvla: An open- source vision-language-action model. arXiv preprint arXiv:2406.09246 (2024)

    2024 arXiv

  54. [63]

    arXiv preprint arXiv:2410.07864 (2024)

    Liu, S., Wu, L., Li, B., Tan, H., Chen, H., Wang, Z., Xu, K., Su, H., Zhu, J.: Rdt-1b: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864 (2024)

    2024 arXiv

  55. [64]

    arXiv preprint arXiv:2410.24164 (2024)

    Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al.:��� 0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164 (2024)

    2024 arXiv

  56. [65]

    arXiv preprint arXiv:2307.01848 (2023)

    Wu, Z., Wang, Z., Xu, X., Lu, J., Yan, H.: Embodied task planning with large language models. arXiv preprint arXiv:2307.01848 (2023)

    2023 arXiv

  57. [66]

    Hippocampus �� (11), 1161–1170 (2023) 31

    Hilton, C., Wiener, J.: Route sequence knowledge supports the formation of cognitive maps. Hippocampus �� (11), 1161–1170 (2023) 31

    2023

  58. [67]

    Nature Communications �� (1), 1597 (2023)

    Zador, A., Escola, S., Richards, B., ¨Olveczky, B., Bengio, Y., Boahen, K., Botvinick, M., Chklovskii, D., Churchland, A., Clopath, C., �� ���: Catalyzing next-generation artificial intelligence through neuroai. Nature Communications �� (1), 1597 (2023)

    2023

  59. [68]

    IEEE Transactions on Intelligent Transportation Systems �� (7), 6907–6921 (2021)

    Zou, Q., Sun, Q., Chen, L., Nie, B., Li, Q.: A comparative analysis of lidar slam-based indoor navigation for autonomous vehicles. IEEE Transactions on Intelligent Transportation Systems �� (7), 6907–6921 (2021)

    2021

  60. [69]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Cheng, T., Song, L., Ge, Y., Liu, W., Wang, X., Shan, Y.: Yolo- world: Real-time open-vocabulary object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16901–16911 (2024)

    2024

  61. [70]

    arXiv preprint arXiv:2302.07241 (2023)

    Jatavallabhula, K.M., Kuwajerwala, A., Gu, Q., Omama, M., Chen, T., Maalouf, A., Li, S., Iyer, G., Saryazdi, S., Keetha, N., et al.: Conceptfu- sion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.07241 (2023)

    2023 arXiv

  62. [71]

    In: 2023 IEEE International Conference on Robotics and Automation (ICRA), pp

    Huang, C., Mees, O., Zeng, A., Burgard, W.: Visual language maps for robot navigation. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 10608–10615 (2023). IEEE

    2023

  63. [72]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High- resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695 (2022)

    2022

  64. [73]

    Wiley interdisciplinary reviews: data mining and knowledge discovery �(3), 231–240 (2011)

    Kriegel, H.-P., Kr¨ oger, P., Sander, J., Zimek, A.: Density-based cluster- ing. Wiley interdisciplinary reviews: data mining and knowledge discovery �(3), 231–240 (2011)

    2011

  65. [74]

    IEEE transactions on Systems Science and Cybernetics �(2), 100–107 (1968)

    Hart, P.E., Nilsson, N.J., Raphael, B.: A formal basis for the heuris- tic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics �(2), 100–107 (1968)

    1968

  66. [75]

    In: ROBOTIK 2012; 7th German Conference on Robotics, pp

    R¨ osmann, C., Feiten, W., W¨ osch, T., Hoffmann, F., Bertram, T.: Trajec- tory modification considering dynamic constraints of autonomous robots. In: ROBOTIK 2012; 7th German Conference on Robotics, pp. 1–6 (2012). VDE

    2012

  67. [76]

    In: 2013 European Conference on Mobile Robots, pp

    R¨ osmann, C., Feiten, W., W¨ osch, T., Hoffmann, F., Bertram, T.: Effi- cient trajectory optimization using a sparse model. In: 2013 European Conference on Mobile Robots, pp. 138–143 (2013). IEEE

    2013

  68. [77]

    arXiv preprint arXiv:2310.13724 (2023)

    Puig, X., Undersander, E., Szot, A., Cote, M.D., Yang, T.-Y., Partsey, R., 32 Desai, R., Clegg, A.W., Hlavac, M., Min, S.Y., et al.: Habitat 3.0: A co- habitat for humans, avatars and robots. arXiv preprint arXiv:2310.13724 (2023)

    2023 arXiv

  69. [78]

    https://aihabitat.org/challenge/2023/ (2023)

    Yadav, K., Krantz, J., Ramrakhya, R., Ramakrishnan, S.K., Yang, J., Wang, A., Turner, J., Gokaslan, A., Berges, V.-P., Mootaghi, R., Maksymets, O., Chang, A.X., Savva, M., Clegg, A., Chaplot, D.S., Batra, D.: Habitat Challenge 2023. https://aihabitat.org/challenge/2023/ (2023)

    2023

  70. [79]

    Advances in Neural Information Processing Systems �� , 121475–121499 (2024)

    Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., XiXuan, S., �� ���: Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems �� , 121475–121499 (2024)

    2024

  71. [80]

    In: Proceed- ings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp

    Ross, S., Gordon, G., Bagnell, D.: A reduction of imitation learning and structured prediction to no-regret online learning. In: Proceed- ings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 627–635 (2011). JMLR Workshop and Conference...

    2011

  72. [81]

    arXiv preprint arXiv:2402.15852 (2024)

    Zhang, J., Wang, K., Xu, R., Zhou, G., Hong, Y., Fang, X., Wu, Q., Zhang, Z., Wang, H.: Navid: Video-based vlm plans the next step for vision-and-language navigation. arXiv preprint arXiv:2402.15852 (2024)

    2024 arXiv

  73. [82]

    arXiv preprint arXiv:2412.06224 (2024)

    Zhang, J., et al.: Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks. arXiv preprint arXiv:2412.06224 (2024)

    2024 arXiv

  74. [83]

    arXiv preprint arXiv:2412.04453 (2024)

    Cheng, A.-C., Ji, Y., Yang, Z., Gongye, Z., Zou, X., Kautz, J., Bıyık, E., Yin, H., Liu, S., Wang, X.: Navila: Legged robot vision-language-action model for navigation. arXiv preprint arXiv:2412.04453 (2024)

    2024 arXiv

  75. [84]

    A marble island in a kitchen

    Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In: International Conference on Machine Learning, pp. 19730–19742 (2023). PMLR 33 � ������ �������� BSC-Nav deeply integrates the perc...

    2023

  76. [85]

    centered

    Look at their loc coordinates. If multiple instances with the same label have very close or nearly identical coordinates, treat them as the same object. 2) Determine which set of coordinates (if there are multiple distinct sets) is the most reliable representation of the objec...

  77. [86]

    Move to the �stairs at the end of the hallway �

  78. [87]

    Move to the �bed in the bedroom �

  79. [88]

    Move to the �closet�

  80. [89]

    Go up the stairs and go between the couch and the table

    Move to the �toilet in the bathroom � ������� �� � ���� ������� “Go to the wooden stairs. Go up the stairs and go between the couch and the table. Walk into the house through the sliding glass door. Go to the television. Go to the refrigerator. Go to the front of the toaster a...

  81. [90]

    Move to the �wooden stairs�

  82. [91]

    Move to the �area between a couch and a table �

  83. [92]

    Move to the �sliding glass door �

  84. [93]

    Move to the �television�

  85. [94]

    Move to the �refrigerator�

  86. [95]

    We need to go around and check

    Move to the �toaster� �� ������ ������������� 3.1 Please planning the following text prompt into sub-goals and respond strictly in the specified format. 3.2 Do not include any other information. Text prompt: �text prompt� ������ ��� ����������� ��� ������ ��������� You will ac...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.