Pith. sign in

REVIEW 5 major objections 6 minor 3 cited by

OctoNav: Towards Generalist Embodied Navigation

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a single vision-language-action model can follow free-form instructions that compound several navigation capabilities at once, and that a benchmark with think-before-action reasoning traces plus staged reinforcement…

desk verdict A serious benchmark-and-method paper whose headline margin can't be trusted until the automatic instruction-trajectory grounding is audited. read the letter →

arxiv 2506.09839 v1 pith:F6UC2VOJ submitted 2025-06-11 cs.CV cs.AIcs.RO

Chen Gao , Liankai Jin , Xingyu Peng , Jiazhao Zhang , Yue Deng , Annan Li , He Wang , Si Liu This is my paper
classification cs.CVcs.AIcs.RO
keywords generalistembodiednavigationvision-language-actionmodelfree-formmulti-modalinstructionsmulti-capabilitybenchmarkthink-before-actionreasoningreinforcementlearningforsim-to-realtransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a single vision-language-action model can follow free-form instructions that string together several different navigation capabilities in one command: point goals, image goals, object goals, and step-by-step language navigation. To support that claim, it introduces a benchmark with over 45,000 instruction-trajectory pairs in 400+ continuous indoor scenes, plus a dataset of reasoning traces that explain each action. It then trains a model that emits low-level actions directly from 2D video, using a three-stage pipeline of imitation, reasoning fine-tuning with reinforcement learning, and online RL. On the benchmark's test split, the model reaches a 19.40% success rate, more than double the best fine-tuned baseline, and a real-robot demo suggests the behavior transfers to a physical quadruped. The message is that generalist navigation can be treated as a single instruction-following problem rather than a list of separate tasks, with reasoning-before-action as the ingredient that makes the mixture work.

What carries the argument

The load-bearing mechanism is the Hybrid Training Paradigm operating on the OctoNav-Bench data. The benchmark's automatic annotation pipeline samples trajectories in continuous scenes, instantiates multi-capability instruction templates with grounded placeholders (coordinates, reference images, object categories), and then uses a reasoning model to write a <Think>...</Think> chain-of-thought for each action, producing the TBA-CoT dataset. Training runs in stages: Action-SFT then TBA-SFT for imitation, Nav-GRPO (group relative policy optimization with a stepped reward of 1 for exact action-plus-magnitude, 0.5 for correct action with wrong magnitude, and 0 otherwise) to sharpen reasoning, and online advantage-actor-critic reinforcement learning with a distance-based reward in the simulator. Each stage is designed to add one capability: following, thinking, thinking quality, and efficiency.

What would settle it

Take a random sample of 200 test trajectories from OctoNav-Bench, have human annotators verify every sub-goal placeholder against the visual observation at its waypoint, and re-evaluate OctoNav-R1 only on trajectories where all groundings are correct; if the success rate falls to the baseline level on that subset, or if the audit finds a substantial mismatch rate, the claimed superiority would not be a reliable measure of instruction-following.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that OctoNav-R1, a vision-language-action (VLA) model built on a video-based multimodal LLM, can process instructions that freely mix modalities (text, reference images, coordinates) and capabilities (point, object, image, instance-image, and vision-language navigation) and output atomic actions like 'move forward 75cm' or 'turn right 30 degrees' in continuous environments. The paper attributes the result to the Hybrid Training Paradigm: Action-SFT teaches instruction following; TBA-SFT teaches the model to emit thinking before acting; Nav-GRPO reinforces the thinking with a stepped reward that gives partial credit for a correct action with the wrong magnitude; and online RL further optimizes success and path efficiency. Ablations show overall success rising from 5.80% for the base model to 8.80% after Action-SFT, 14.40% after TBA-SFT, 17.00% after Nav-GRPO, and 19.40% after online RL. The final model surpasses all prior methods on every capability in the benchmark, and on a real quadruped it completes multi-stage instructions without real-world fine-tuning.

Load-bearing premise

The whole comparison rests on the assumption that each automatically generated instruction's reference images, coordinates, and object labels really point at distinct waypoints along the paired trajectory; if the grounding is noisy, the success metric measures alignment with the annotation rather than genuine instruction following.

Editorial extensions

If this is right

  • If the central claim holds, benchmarks for embodied navigation no longer need separate splits per task; a single benchmark with mixed free-form instructions and per-capability metrics can measure generalist ability.
  • The staged training result implies that reasoning traces alone provide a large jump in success even before reinforcement learning, suggesting explicit thinking data is a strong supervision signal for navigation.
  • The stepped GRPO reward, which gives partial credit for a correct action with the wrong magnitude, could be applied to other continuous-control VLA settings where exact magnitudes are hard to match.
  • The sim-to-real demo suggests a model trained purely on 2D RGB frames in simulation can control a real robot without fine-tuning, opening a path for navigation policies that avoid explicit mapping or depth input.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported 19.40% overall success rate is still low in absolute terms; if the benchmark is adopted widely, the main bottleneck may shift from instruction parsing to low-level exploration and long-horizon memory.
  • The thinking-frequency ablation shows that performance is similar whether the model thinks every 10, 20, or 40 steps, suggesting the benefit comes from occasional re-planning rather than continuous deliberation; an adaptive 'think only when progress stalls' policy might capture most of the gain at lower cost.
  • Because overall success requires all sub-tasks to succeed in order, the per-capability success numbers likely hide compound failures: an early error masks later capabilities, so per-capability success conditioned on reaching that sub-goal would more honestly separate instruction-order effects from capability skill.
  • The automatic grounding of placeholders is the key risk in the benchmark: if a reference image or coordinate is attached to the wrong waypoint, the success metric measures alignment with the annotation rather than genuine instruction following, so a small human-audited subset with corrected labels would make the comparison fairer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces OctoNav-Bench, a large-scale Habitat-based benchmark with 45k+ instruction-trajectory pairs over 400+ scenes from HM3D, MP3D, Gibson, and ProcTHOR, where free-form instructions arbitrarily combine ObjNav, PointNav, ImgNav, Ins-ImgNav, and VLN, and a 10k+ Think-Before-Action Chain-of-Thought (TBA-CoT) dataset. On top of this, the authors propose OctoNav-R1, a LLaMA-VID/Vicuna-based vision-language-action model, and a Hybrid Training Paradigm (HTP) consisting of Action-SFT, TBA-SFT, Nav-GRPO, and online RL. The main empirical claim is that OctoNav-R1 achieves 19.40% overall SR on OctoNav-Bench versus 9.20% for the strongest baseline (Uni-NaVid fine-tuned on the same benchmark), with ablations showing each HTP stage contributes to the gain. A qualitative real-robot deployment is also reported.

Significance. If the benchmark and the reported gains hold up, the paper is a meaningful step toward generalist embodied navigation: it unifies five navigation capabilities in continuous environments, provides free-form multi-modal instructions, and demonstrates a DeepSeek-R1-style thinking-before-action training pipeline for VLA models. The strengths are the scale of the annotation effort, the consistency of the ablations, the large gap over the reported baselines, and the inclusion of a real-robot demonstration. However, the empirical value is currently conditional on benchmark validity and reproducibility, because the benchmark is not released, the grounding of automatically generated instructions is not quantitatively verified, and all results are single-run point estimates without error bars.

major comments (5)
  1. [§3.1, Appendix A.2.5] The load-bearing premise that automatically grounded placeholders correspond to real waypoints is not quantitatively verified. The text describes GPT-based filtering and manual review but reports no rejection rate, inter-annotator agreement, or geometric/visual consistency checks for the ImgNav reference image, ObjNav category, PointNav coordinate, or VLN segment. Because the same pipeline builds the test instructions, systematic grounding errors would directly inflate all success metrics in Tables 2-4. Please report quantitative grounding audits per capability and make the benchmark available for independent audit.
  2. [Tables 2-4, Appendix D] All main results are point estimates from a single run. Several ablations differ by amounts that are likely within run-to-run noise (e.g., Table 4c: per-20-step SR 19.40 vs per-40-step 18.80; Table 4a: Strict 16.20 vs Stepped 17.00). Please provide multiple seeds, error bars, and significance tests for the main tables, or at least for Tables 2-4 and the key ablations.
  3. [Appendix A.2.6, §4.1] The TBA-CoT dataset is generated by embedding the ground-truth action in the DeepSeek-R1 prompt ('reference action embedding strategy'), so the '<Think>' traces are post-hoc rationalizations conditioned on the answer. This does not invalidate TBA-SFT as an imitation signal, but it weakens the paper's claim that the model learns an independent thinking-before-action process and biases the reasoning traces toward the policy's ground truth. Please either generate a comparison set of reasoning traces without the reference action and ablate, or clearly reframe the claim as supervised learning on answer-conditioned rationalizations.
  4. [Appendix C.3, Table 2] The fairness of baseline comparisons is not established. The modifications to NaviLLM, NavGPT-2, NaVid, and Uni-NaVid are described at a high level, with no reported tuning budgets, training steps, data amounts, or seed variation; NavGPT-2 is only given its first training stage. Without evidence that the baselines are trained to comparable capacity, the 19.40% vs 9.20% gap may partly reflect underfitting of the baselines. Please report full training configurations and, ideally, evaluate each baseline with the same compute budget.
  5. [§5.2, §5.3] The generalist claim rests entirely on OctoNav-Bench; there is no evaluation on established held-out benchmarks such as R2R-CE, GOAT-Bench, or Habitat ObjectNav, and the real-robot section is qualitative. Please add quantitative cross-benchmark results or clearly scope the claims to OctoNav-Bench.
minor comments (6)
  1. [§5.2] The text says NaviLLM and NavGPT-2 are marked '*' in Tab 1, but the asterisks appear in Table 2; please fix the cross-reference.
  2. [Appendix A.5] The success-distance thresholds (0.36m for PointNav/ImgNav, 1m for ObjNav/Ins-ImgNav, 3m for VLN) are stated without sensitivity analysis; please report how results vary with these thresholds or justify them more explicitly from prior work.
  3. [Appendix A.5] The overall OSR in Eq. (18) only checks the final sub-task indicator OSi,numi, which makes 'oracle' success for composite instructions harder to interpret; please clarify this in the metric description.
  4. [Abstract and Figure 4] The spelling 'Nav-GPRO' is used in the abstract and in Figure 4, while 'Nav-GRPO' is used in the text; please unify the notation.
  5. [Appendix B.3, Appendix C.3] There are typos: 'visulizations' in Appendix B.3 and 'OctaNav-Bench' in Appendix C.3; please proofread.
  6. [Figure 1] The label '10k+TBA-CoT' is ambiguous; §5.1 states the TBA-CoT dataset contains 10k+ instruction-think-action pairs, so please make the figure caption consistent with that definition.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark, training pipeline, and ablations are self-contained; the answer-conditioned TBA-CoT annotations are training data, not fitted predictions.

full rationale

I inspected the claimed derivation chain: OctoNav-Bench is constructed from an automatic annotation pipeline that instantiates instruction placeholders with trajectory-derived elements, and OctoNav-R1 is trained on that benchmark with SFT, Nav-GRPO, and online RL. The central empirical claim—that OctoNav-R1 reaches 19.40% SR versus 9.20% for the strongest baseline—is evaluated on held-out scenes and against external or fine-tuned baselines; it does not reduce to a fitted parameter or a self-referential definition. The nearest potential circularity is the TBA-CoT construction: Section 3.2 states 'we also feed the ground-truth action of the current step, making LLM generate a detailed reasoning trace according to the reference action,' and Appendix A.2.6 describes a 'reference action embedding strategy.' This means the reasoning traces are post-hoc rationalizations conditioned on the known action. However, this is training-data generation, not an evaluation prediction: the model must still produce actions on unseen instructions and scenes, and the success metrics are computed against trajectory waypoints, not against the generated thoughts. The paper even labels these as 'pseudo reasoning-thoughts.' No load-bearing self-citation chain, imported uniqueness theorem, or ansatz-smuggled-by-citation was found. The validity concern about placeholder grounding is a benchmark-quality risk, not a circularity, because it does not make the reported SR equal to an input by construction. Therefore the score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central empirical results rest on hand-set benchmark thresholds (success distances, trajectory filters, coverage thresholds) and hand-tuned reward values and thinking frequency. The axioms are largely domain assumptions about the reliability of automated data generation, the validity of the CoT supervision, and the fairness of baseline adaptations. No new physical or conceptual entities are postulated.

free parameters (6)
  • Per-capability success distance thresholds = 0.36m (PointNav, ImgNav), 1m (ObjNav, Ins-ImgNav), 3m (VLN)
    Set by hand in appendix A.5; directly determines success metrics and heavily influences all reported SR/SPL numbers.
  • Trajectory distance bounds per capability count = [3k, 10k] meters
    Chosen in appendix A.2.2 to filter sampled trajectories; controls task difficulty and dataset composition.
  • Nav-GRPO stepped reward values = 1.0 for exact action+magnitude, 0.5 for correct action wrong magnitude, 0 otherwise
    Hand-selected in Eq. 4; ablation (Table 4a) shows the choice matters, making it a tuned parameter.
  • Thinking frequency = one think step per 20 action steps
    Selected after ablation (Table 4c); performance varies with frequency, so this is a tuned parameter.
  • Ins-ImgNav frame coverage threshold = 20%
    Used in appendix A.2.1 to select instance-image goals; changes which images are used as goals.
  • Online RL move-forward distance d' = 25 cm
    Introduced in Section 4.3 to give non-zero rewards for turning actions; affects reward shaping.
assumptions (5)
  • domain assumption Automated pipeline produces correctly grounded instruction-trajectory pairs after quality checks.
    Section 3.1 and appendix A.2.5 rely on automatic and manual checks to ensure reference images, categories, and coordinates truly match the trajectory. If mismatches remain, the benchmark measures noise.
  • domain assumption The TBA-CoT traces generated by Qwen-VL and DeepSeek-R1 are valid reasoning for navigation decisions.
    Appendix A.2.6: CoT is generated by feeding the ground-truth action as a reference, so the reasoning is action-conditioned. The paper assumes this distillation still yields useful thinking at inference.
  • domain assumption Ordered sub-goal success metric correctly captures multi-task completion.
    Appendix A.5 defines success as visiting sub-goal areas in order. The paper assumes this matches human notions of following compound instructions.
  • domain assumption Habitat simulation is a reliable proxy for real-world navigation, so sim2real transfer is meaningful.
    The real-world deployment (appendix B) is a small demo; the main results rely on Habitat, and the paper assumes transfer without real-world fine-tuning.
  • ad hoc to paper The baseline modifications (NaviLLM*, NavGPT-2*, NaVid†, Uni-NaVid†) are fair and do not disadvantage prior methods.
    Section 5.2 and appendix C.3 describe modifications; the fairness of these adaptations is assumed and not independently validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OctoNav: Towards Generalist Embodied Navigation." pith.science (2026). https://pith.science/paper/F6UC2VOJ

@misc{pith2026250609839,
  author       = {Pith},
  title        = {Pith review of: OctoNav: Towards Generalist Embodied Navigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F6UC2VOJ}},
  note         = {Machine review of arXiv:2506.09839}
}
read the original abstract

Embodied navigation stands as a foundation pillar within the broader pursuit of embodied AI. However, previous navigation research is divided into different tasks/capabilities, e.g., ObjNav, ImgNav and VLN, where they differ in task objectives and modalities, making datasets and methods are designed individually. In this work, we take steps toward generalist navigation agents, which can follow free-form instructions that include arbitrary compounds of multi-modal and multi-capability. To achieve this, we propose a large-scale benchmark and corresponding method, termed OctoNav-Bench and OctoNav-R1. Specifically, OctoNav-Bench features continuous environments and is constructed via a designed annotation pipeline. We thoroughly craft instruction-trajectory pairs, where instructions are diverse in free-form with arbitrary modality and capability. Also, we construct a Think-Before-Action (TBA-CoT) dataset within OctoNav-Bench to provide the thinking process behind actions. For OctoNav-R1, we build it upon MLLMs and adapt it to a VLA-type model, which can produce low-level actions solely based on 2D visual observations. Moreover, we design a Hybrid Training Paradigm (HTP) that consists of three stages, i.e., Action-/TBA-SFT, Nav-GPRO, and Online RL stages. Each stage contains specifically designed learning policies and rewards. Importantly, for TBA-SFT and Nav-GRPO designs, we are inspired by the OpenAI-o1 and DeepSeek-R1, which show impressive reasoning ability via thinking-before-answer. Thus, we aim to investigate how to achieve thinking-before-action in the embodied navigation field, to improve model's reasoning ability toward generalists. Specifically, we propose TBA-SFT to utilize the TBA-CoT dataset to fine-tune the model as a cold-start phrase and then leverage Nav-GPRO to improve its thinking ability. Finally, OctoNav-R1 shows superior performance compared with previous methods.

Figures

Figures reproduced from arXiv: 2506.09839 by the authors.

Figure 1
Figure 1. On the left, we present the large-scale OctoNav-Bench, which contains diverse instruction [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The automatic construction pipeline of OctoNav-Bench. (I) Template Generation. We generate diverse instruction templates, where multiple capabilities are involved and specific elements are represented via placeholders. (II) Trajectory Generation and Instruction Instantiation. We extract elements along the sampled trajectory and instantiate the instruction by grounding the placeholders with corresponding elements. (I… view at source ↗
Figure 3
Figure 3. The automatic construction method of TBA-CoT. For the trajectories in OctoNav-Bench, we leverage Qwen-VL and DeepSeek-R1 to produce the thinking thoughts behind the action steps [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (22 more)
Figure 4
Figure 4. Figure 4: Overview of the HTP for training OctoNav-R1. The model takes multi-model instruction and visual observation as inputs, and produces textual answers, where model architecture details are in appendix C.1. HTP contains three training stages, which are described in Sec. 4.…
Figure 5
Figure 5. Figure 5: Visualization of TBA in a trajectory. TBA-SFT. After the TBA-SFT phase, the overall SR is improved by 5.60%, demonstrating the effectiveness of the thinking process. Moreover, the model can better handle multi-capability with the thinking ability, as the performance is…
Figure 6
Figure 6. Figure 6: Construction pipeline of instance image goals for Ins-ImgNav [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Example Prompt for Template Generation. A.2.2 Trajectory Generation During trajectory generation, we apply several constraints to ensure diversity and reality of trajectories. Firstly, the starting and ending points of trajectories are randomly sampled within the navig…
Figure 8
Figure 8. Figure 8: Prompt for Image Information Extraction. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Prompt for Historical Information Aggregation. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Prompt for TBA Reasoning. TBA Reasoning. The deep reasoning stage integrates current observations, historical trajectories, navigation instructions, and optional image goals into a structured prompt template for reasoning model to process. To align model outputs with …
Figure 11
Figure 11. Figure 11: Sankey diagram of capability distribution within instructions. [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Distribution of trajectory and category. [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Visualization of the word cloud. be longer in word count, even if fewer in number. As a result, the total textual proportion of VLN remains balanced within the dataset. Bar Charts. In [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Distribution of the task. The x-axis represents the number of words in the instruction, [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: We use Unitree GO2, and mount RealSense D455, a portable Wi-Fi and a LiDAR-L1. [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Visualization in real world. B Appendix: Real-World Deployment B.1 Robot Setup OctoNav-R1 is tested on a robotic dog (Unitree GO2) mounted with a RealSense D455 camera on its head (We provide a visualization at [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: The model architecture of OctoNav-R1. Best viewed in color. [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 18
Figure 18. Figure 18: Prompt for Training. Then each query token embedding is inserted to the head of the corresponding image embeddings. The embedding of the placeholder (e.g., {ImageNav} and {InstanceImageNav}) is replaced by the goal image embedding E′ g , as shown in [PITH_FULL_IMAGE:…
Figure 19
Figure 19. Figure 19: Visualization on OctoNav-Bench. Instruction: Your current position is (-3.28, -2.79, 0.18) and your current orientation is (-0.38, 0.93, 0). Adjust your path with image {ImageNav} until the corresponding viewpoint appears. Find the chair and then head straight to the …
Figure 20
Figure 20. Figure 20: Visualization on OctoNav-Bench. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_20.png]
Figure 21
Figure 21. Figure 21: Visualization on OctoNav-Bench. Instruction: Your current position is (9.60, -0.83, 0.02) and your current orientation is (0.88, -0.48, 0). Steer based on image {ImageNav} until environmental features correspond. Follow the bay windows then take a left. go slightly ri…
Figure 22
Figure 22. Figure 22: Visualization on OctoNav-Bench. In [PITH_FULL_IMAGE:figures/full_fig_p028_22.png]
Figure 23
Figure 23. Figure 23: Visualization of think-before-actioin in a trajectory. [PITH_FULL_IMAGE:figures/full_fig_p029_23.png]
Figure 24
Figure 24. Figure 24: Visualization of think-before-actioin in a trajectory. [PITH_FULL_IMAGE:figures/full_fig_p029_24.png]
Figure 25
Figure 25. Figure 25: Visualization of think-before-actioin in a trajectory. [PITH_FULL_IMAGE:figures/full_fig_p029_25.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Joint On-and-Off Policy Learning for Vision-and-Language Navigation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    JOP-VLN combines DAgger imitation learning with GRPO reinforcement learning, using high-entropy trajectory filtering and error-correction prioritization, achieving 69.9% SR on R2R Val-Unseen.

  2. Structured Observation Language for Efficient and Generalizable Vision-Language Navigation

    cs.CV 2026-03 reject novelty 6.0 of 10

    SOL-Nav encodes RGB-D observations as grid-organized text and uses a 0.6B text-embedding model with four classification heads to predict navigation action blocks, reporting SOTA/comparable R2R-CE/RxR-CE results with 1...

  3. Nav-R1: Reasoning and Navigation in Embodied Scenes

    cs.RO 2025-09 reject novelty 4.0 of 10

    Nav-R1 uses a 110K synthetic CoT dataset, GRPO with three rewards, and a fast-in-slow system to set new SOTA on R2R-CE, RxR-CE, and HM3D-OVON.

Reference graph

Works this paper leans on

79 extracted references · 39 canonical work pages · cited by 3 Pith papers

  1. [1]

    Object goal navigation using goal-oriented semantic exploration.Advances in Neural Information Processing Systems, 33:4247–4258, 2020

    Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinov. Object goal navigation using goal-oriented semantic exploration.Advances in Neural Information Processing Systems, 33:4247–4258, 2020

    2020

  2. [2]

    Room-object entity prompting and reasoning for embodied referring expression.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(2):994–1010, 2023

    Chen Gao, Si Liu, Jinyu Chen, Luting Wang, Qi Wu, Bo Li, and Qi Tian. Room-object entity prompting and reasoning for embodied referring expression.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(2):994–1010, 2023

    work page 2023

  3. [3]

    Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance? volume 5, pages 6670–6677, 2020

    Abhishek Kadian*, Joanne Truong*, Aaron Gokaslan, Alexander Clegg, Erik Wijmans, Stefan Lee, Manolis Savva, Sonia Chernova, and Dhruv Batra. Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance? volume 5, pages 6670–6677, 2020

    work page 2020

  4. [4]

    The surprising effectiveness of visual odometry techniques for embodied pointgoal navigation

    Xiaoming Zhao, Harsh Agrawal, Dhruv Batra, and Alexander G Schwing. The surprising effectiveness of visual odometry techniques for embodied pointgoal navigation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 16127–16136, 2021

    work page 2021

  5. [5]

    Is mapping necessary for realistic pointgoal navigation? InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17232–17241, 2022

    Ruslan Partsey, Erik Wijmans, Naoki Yokoyama, Oles Dobosevych, Dhruv Batra, and Oleksandr Maksymets. Is mapping necessary for realistic pointgoal navigation? InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17232–17241, 2022

    work page 2022

  6. [6]

    Target-driven visual navigation in indoor scenes using deep reinforcement learning

    Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In2017 IEEE international conference on robotics and automation (ICRA), pages 3357–3364. IEEE, 2017

    work page 2017

  7. [7]

    Fgprompt: Fine-grained goal prompting for image-goal navigation.Advances in Neural Information Processing Systems, 36:12054– 12073, 2023

    Xinyu Sun, Peihao Chen, Jugang Fan, Jian Chen, Thomas Li, and Mingkui Tan. Fgprompt: Fine-grained goal prompting for image-goal navigation.Advances in Neural Information Processing Systems, 36:12054– 12073, 2023

    work page 2023

  8. [8]

    Habitat challenge 2023.https://aihabitat.org/challenge/2023/, 2023

    Karmesh Yadav, Jacob Krantz, Ram Ramrakhya, Santhosh Kumar Ramakrishnan, Jimmy Yang, Austin Wang, John Turner, Aaron Gokaslan, Vincent-Pierre Berges, Roozbeh Mootaghi, Oleksandr Maksymets, Angel X Chang, Manolis Savva, Alexander Clegg, Devendra Singh Chaplot, and Dhruv Batra. Habitat challenge 2023.https://aihabitat.org/challenge/2023/, 2023

    work page 2023

Show all 79 references
  1. [9]

    Navigating to objects specified by images

    Jacob Krantz, Theophile Gervet, Karmesh Yadav, Austin Wang, Chris Paxton, Roozbeh Mottaghi, Dhruv Batra, Jitendra Malik, Stefan Lee, and Devendra Singh Chaplot. Navigating to objects specified by images. InProceedings of the IEEE/CVF International Conference on Computer Vision...

    2023

  2. [10]

    Instance-specific image goal navigation: Training embodied agents to find object instances.arXiv preprint arXiv:2211.15876, 2022

    Jacob Krantz, Stefan Lee, Jitendra Malik, Dhruv Batra, and Devendra Singh Chaplot. Instance-specific image goal navigation: Training embodied agents to find object instances.arXiv preprint arXiv:2211.15876, 2022

    2022 arXiv

  3. [11]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. InProceedings of the IEEE conference ...

    2018

  4. [12]

    Beyond the nav-graph: Vision-and-language navigation in continuous environments

    Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision-and-language navigation in continuous environments. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVIII 16, pages ...

    2020

  5. [13]

    Reverie: Remote embodied visual referring expression in real indoor environments

    Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages ...

    2020

  6. [14]

    Goat-bench: A benchmark for multi-modal lifelong navigation

    Mukul Khanna, Ram Ramrakhya, Gunjan Chhablani, Sriram Yenamandra, Theophile Gervet, Matthew Chang, Zsolt Kira, Devendra Singh Chaplot, Dhruv Batra, and Roozbeh Mottaghi. Goat-bench: A benchmark for multi-modal lifelong navigation. InProceedings of the IEEE/CVF Conference on Co...

    2024

  7. [15]

    Towards long-horizon vision-language navigation: Platform, benchmark and method.arXiv preprint arXiv:2412.09082, 2024

    Xinshuai Song, Weixing Chen, Yang Liu, Vincent Chan, Guanbin Li, and Liang Lin. Towards long-horizon vision-language navigation: Platform, benchmark and method.arXiv preprint arXiv:2412.09082, 2024

    2024 arXiv

  8. [16]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  9. [17]

    Instructnav: Zero-shot system for generic instruction navigation in unexplored environment

    Yuxing Long, Wenzhe Cai, Hongcheng Wang, Guanqi Zhan, and Hao Dong. Instructnav: Zero-shot system for generic instruction navigation in unexplored environment. In8th Annual Conference on Robot Learning, 2024

    2024

  10. [18]

    Navcot: Boosting llm-based vision-and-language navigation via learning disentangled reasoning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    Bingqian Lin, Yunshuang Nie, Ziming Wei, Jiaqi Chen, Shikui Ma, Jianhua Han, Hang Xu, Xiaojun Chang, and Xiaodan Liang. Navcot: Boosting llm-based vision-and-language navigation via learning disentangled reasoning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  11. [19]

    Enhancing large language models with rag for visual language navigation in continuous environments.Electronics, 14(5):909, 2025

    Xiaoan Bao, Zhiqiang Lv, and Biao Wu. Enhancing large language models with rag for visual language navigation in continuous environments.Electronics, 14(5):909, 2025

    2025

  12. [20]

    Llm-guided zero-shot visual object navigation with building semantic map

    Jin Shi, Satoshi Yagi, Satoshi Yamamori, and Jun Morimoto. Llm-guided zero-shot visual object navigation with building semantic map. In2025 IEEE/SICE International Symposium on System Integration (SII), pages 1274–1279. IEEE, 2025

    2025

  13. [21]

    Flexvln: Flexible adaptation for diverse vision-and-language navigation tasks.arXiv preprint arXiv:2503.13966, 2025

    Siqi Zhang, Yanyuan Qiao, Qunbo Wang, Longteng Guo, Zhihua Wei, and Jing Liu. Flexvln: Flexible adaptation for diverse vision-and-language navigation tasks.arXiv preprint arXiv:2503.13966, 2025

    2025 arXiv

  14. [22]

    Unigoal: Towards universal zero-shot goal-oriented navigation.arXiv preprint arXiv:2503.10630, 2025

    Hang Yin, Xiuwei Xu, Lingqing Zhao, Ziwei Wang, Jie Zhou, and Jiwen Lu. Unigoal: Towards universal zero-shot goal-oriented navigation.arXiv preprint arXiv:2503.10630, 2025

    2025 arXiv

  15. [23]

    Navila: Legged robot vision-language-action model for navigation.arXiv preprint arXiv:2412.04453, 2024

    An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Bıyık, Hongxu Yin, Sifei Liu, and Xiaolong Wang. Navila: Legged robot vision-language-action model for navigation.arXiv preprint arXiv:2412.04453, 2024

    2024 arXiv

  16. [24]

    Xia, Mehdi S

    Danny Driess, F. Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Ho Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussai...

    2023

  17. [25]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. InConference on Robot Learning, pages 2165–2183. PMLR, 2023

    2023

  18. [26]

    Vision-language foundation models as effective robot imitators

    Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators. In ICLR, 2024

    2024

  19. [27]

    Navid: Video-based vlm plans the next step for vision-and-language navigation

    Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and He Wang. Navid: Video-based vlm plans the next step for vision-and-language navigation. In Robotics: Science and Systems, 2024

    2024

  20. [28]

    Navgpt: Explicit reasoning in vision-and-language navigation with large language models

    Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-language navigation with large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7641–7649, 2024. 11

    2024

  21. [29]

    Navgpt-2: Unleashing navigational reasoning capability for large vision-language models

    Gengze Zhou, Yicong Hong, Zun Wang, Xin Eric Wang, and Qi Wu. Navgpt-2: Unleashing navigational reasoning capability for large vision-language models. InEuropean Conference on Computer Vision, pages 260–278. Springer, 2024

    2024

  22. [30]

    Jiaqi Chen, Bingqian Lin, Ran Xu, Zhenhua Chai, Xiaodan Liang, and Kwan-Yee K. Wong. Mapgpt: Map-guided prompting with adaptive path planning for vision-and-language navigation. InAnnual Meeting of the Association for Computational Linguistics, 2024

    2024

  23. [31]

    Topv-nav: Unlocking the top-view spatial reasoning potential of mllm for zero-shot object navigation.arXiv preprint arXiv:2411.16425, 2024

    Linqing Zhong, Chen Gao, Zihan Ding, Yue Liao, and Si Liu. Topv-nav: Unlocking the top-view spatial reasoning potential of mllm for zero-shot object navigation.arXiv preprint arXiv:2411.16425, 2024

    2024 arXiv

  24. [32]

    Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks

    Jiazhao Zhang, Kunyu Wang, Shaoan Wang, Minghan Li, Haoran Liu, Songlin Wei, Zhongyuan Wang, Zhizheng Zhang, and He Wang. Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks. InRobotics: Science and Systems, 2025

    2025

  25. [33]

    Towards learning a generalist model for embodied navigation

    Duo Zheng, Shijia Huang, Lin Zhao, Yiwu Zhong, and Liwei Wang. Towards learning a generalist model for embodied navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13624–13634, 2024

    2024

  26. [34]

    Towards versatile embodied navigation

    Hanqing Wang, Wei Liang, Luc V Gool, and Wenguan Wang. Towards versatile embodied navigation. Advances in neural information processing systems, 35:36858–36874, 2022

    2022

  27. [35]

    Re- act: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. Re- act: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  28. [36]

    Topologies of reasoning: Demysti- fying chains, trees, and graphs of thoughts.CoRR, 2024

    Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwasniewski, Jürgen Müller, Lukas Gianinazzi, et al. Topologies of reasoning: Demysti- fying chains, trees, and graphs of thoughts.CoRR, 2024

    2024

  29. [37]

    Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems, 36:11809–11822, 2023

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems, 36:11809–11822, 2023

    2023

  30. [38]

    Large language model guided tree-of-thought.arXiv preprint arXiv:2305.08291, 2023

    Jieyi Long. Large language model guided tree-of-thought.arXiv preprint arXiv:2305.08291, 2023

    2023 arXiv

  31. [39]

    Graph of thoughts: Solving elaborate problems with large language models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. InProceedings of the AAAI Confer...

    2024

  32. [40]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  33. [41]

    Chain of thought empowers transformers to solve inherently serial problems

    Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transformers to solve inherently serial problems. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  34. [42]

    Autoregressive+ chain of thought= recurrent: Recurrence’s role in language models’ computability and a revisit of recurrent transformer

    Xiang Zhang, Muhammad Abdul-Mageed, and Laks VS Lakshmanan. Autoregressive+ chain of thought= recurrent: Recurrence’s role in language models’ computability and a revisit of recurrent transformer. arXiv preprint arXiv:2409.09239, 2024

    2024 arXiv

  35. [43]

    Towards revealing the mystery behind chain of thought: a theoretical perspective.Advances in Neural Information Processing Systems, 36:70757–70798, 2023

    Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: a theoretical perspective.Advances in Neural Information Processing Systems, 36:70757–70798, 2023

    2023

  36. [44]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  37. [45]

    Cppo: Accelerating the training of group relative policy optimization-based reasoning models.arXiv preprint arXiv:2503.22342, 2025

    Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. Cppo: Accelerating the training of group relative policy optimization-based reasoning models.arXiv preprint arXiv:2503.22342, 2025

    2025

  38. [46]

    Group robust preference optimization in reward-free rlhf.Advances in Neural Information Processing Systems, 37:37100–37137, 2024

    Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Am- mar, and Ilija Bogunovic. Group robust preference optimization in reward-free rlhf.Advances in Neural Information Processing Systems, 37:37100–37137, 2024

    2024

  39. [47]

    Adaptive group policy optimization: Towards stable training and token-efficient reasoning.arXiv preprint arXiv:2503.15952, 2025

    Chen Li, Nazhou Liu, and Kai Yang. Adaptive group policy optimization: Towards stable training and token-efficient reasoning.arXiv preprint arXiv:2503.15952, 2025. 12

    2025

  40. [48]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749, 2025

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749, 2025

    2025 arXiv

  41. [49]

    Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    2025 arXiv

  42. [50]

    Tinyllava-video-r1: Towards smaller lmms for video reasoning.arXiv preprint arXiv:2504.09641, 2025

    Xingjian Zhang, Siwei Wen, Wenjun Wu, and Lei Huang. Tinyllava-video-r1: Towards smaller lmms for video reasoning.arXiv preprint arXiv:2504.09641, 2025

    2025 arXiv

  43. [51]

    Matterport3d: Learning from rgb-d data in indoor environments

    Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In International Conference on 3D Vision (3DV), 2017

    2017

  44. [52]

    Habitat- matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai

    Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat- matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. In...

    2021

  45. [53]

    Gibson env: Real-world perception for embodied agents

    Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 9068–9079, 2018

    2018

  46. [54]

    Procthor: Large-scale embodied ai using procedural generation.Advances in Neural Information Processing Systems, 35:5982–5994, 2022

    Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation.Advances in Neural Information Processing Systems, 35:5982–5994, 2022

    2022

  47. [55]

    Soon: Scenario oriented object navigation with graph-based exploration

    Fengda Zhu, Xiwen Liang, Yi Zhu, Qizhi Yu, Xiaojun Chang, and Xiaodan Liang. Soon: Scenario oriented object navigation with graph-based exploration. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12689–12699, 2021

    2021

  48. [56]

    Hm3d-ovon: A dataset and benchmark for open-vocabulary object goal navigation

    Naoki Yokoyama, Ram Ramrakhya, Abhishek Das, Dhruv Batra, and Sehoon Ha. Hm3d-ovon: A dataset and benchmark for open-vocabulary object goal navigation. In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5543–5550. IEEE, 2024

    2024

  49. [57]

    Iterative vision-and-language navigation

    Jacob Krantz, Shurjo Banerjee, Wang Zhu, Jason Corso, Peter Anderson, Stefan Lee, and Jesse Thomason. Iterative vision-and-language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14921–14930, 2023

    2023

  50. [58]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. InEuropean Conference on Computer Vision, pages 323–340. Springer, 2024

    2024

  51. [59]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  52. [60]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5971–5984, 2024

    2024

  53. [61]

    Llava-next: A strong zero-shot video understanding model

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model. https://llava-vl.github. io/blog/2024-04-30-llava-next-video/, 2024

    2024

  54. [62]

    Habitat: A Platform for Embodied AI Research

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. InProceedings of the IEEE/CVF International Conferenc...

    2019

  55. [63]

    Room-and-object aware knowledge reasoning for remote embodied referring expression

    Chen Gao, Jinyu Chen, Si Liu, Luting Wang, Qiong Zhang, and Qi Wu. Room-and-object aware knowledge reasoning for remote embodied referring expression. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3064–3073, 2021

    2021

  56. [64]

    Bird’s-eye-view scene graph for vision-language navigation

    Rui Liu, Xiaohan Wang, Wenguan Wang, and Yi Yang. Bird’s-eye-view scene graph for vision-language navigation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 10968– 10980, 2023

    2023

  57. [65]

    Vision-language navigation with energy-based policy

    Rui Liu, Wenguan Wang, and Yi Yang. Vision-language navigation with energy-based policy. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 13

    2024

  58. [66]

    V olumetric environment representation for vision-language navigation

    Rui Liu, Wenguan Wang, and Yi Yang. V olumetric environment representation for vision-language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16317–16328, 2024

    2024

  59. [67]

    Adaptive zone-aware hierarchical planner for vision-language navigation

    Chen Gao, Xingyu Peng, Mi Yan, He Wang, Lirong Yang, Haibing Ren, Hongsheng Li, and Si Liu. Adaptive zone-aware hierarchical planner for vision-language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14911–14920, 2023

    2023

  60. [68]

    Reinforced structured state-evolution for vision-language navigation

    Jinyu Chen, Chen Gao, Erli Meng, Qiong Zhang, and Si Liu. Reinforced structured state-evolution for vision-language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15450–15459, 2022

    2022

  61. [69]

    Target-driven structured transformer planner for vision-language navigation

    Yusheng Zhao, Jinyu Chen, Chen Gao, Wenguan Wang, Lirong Yang, Haibing Ren, Huaxia Xia, and Si Liu. Target-driven structured transformer planner for vision-language navigation. InProceedings of the 30th ACM international conference on multimedia, pages 4194–4203, 2022

    2022

  62. [70]

    Controllable navigation instruction generation with chain of thought prompting

    Xianghao Kong, Jinyu Chen, Wenguan Wang, Hang Su, Xiaolin Hu, Yi Yang, and Si Liu. Controllable navigation instruction generation with chain of thought prompting. InEuropean Conference on Computer Vision, pages 37–54. Springer, 2024

    2024

  63. [71]

    Habitat 2.0: Training home assistants to rearrange their habitat

    Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir V ondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, J...

    2021

  64. [72]

    Habitat 3.0: A co-habitat for humans, avatars and robots.arXiv preprint arXiv:2310.13724, 2023

    Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, et al. Habitat 3.0: A co-habitat for humans, avatars and robots.arXiv preprint arXiv:2310.13724, 2023

    arXiv

  65. [73]

    On evaluation of embodied navigation agents.arXiv preprint arXiv:1807.06757, 2018

    Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents.arXiv preprint arXiv:1807.06757, 2018

    2018 arXiv

  66. [74]

    Objectnav revisited: On evaluation of embodied agents navigating to objects.arXiv preprint arXiv:2006.13171, 2020

    Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects.arXiv preprint arXiv:2006.13171, 2020

    2006 arXiv

  67. [75]

    Eva-clip: Improved training techniques for clip at scale.arXiv preprint arXiv:2303.15389, 2023

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale.arXiv preprint arXiv:2303.15389, 2023

    2023 arXiv

  68. [76]

    Bert: Pre-training of deep bidi- rectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human lang...

    2019

  69. [77]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chat- bot impressing gpt-4 with 90%* chatgpt quality. https://lmsys.org/blog/2023-03-3...

    2023

  70. [78]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  71. [79]

    You arrive at [room name], where you observe [iconic objects]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning.arXiv preprint arXiv:2305.06500, 2023. 14 A Appendix: More Deta...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.