Pith. sign in

REVIEW 4 major objections 13 minor 2 cited by

Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigation

T0 review · 4 major / 13 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that imagining future scenes in language rather than rendering them as images lets a 1.5B-parameter navigation agent beat larger vision-based and LLM baselines on the R2R benchmark.

desk verdict Solid engineering contribution with a coherent dual-branch design, but the 'future imagination' claim is weaker than advertised and the training objective is underspecified. read the letter →

arxiv 2505.20897 v2 pith:MF2ZQSQA submitted 2025-05-27 cs.CV cs.AIcs.CLcs.RO

Pingrui Zhang , Yifei Su , Pengyuan Wu , Dong An , Li Zhang , Zhigang Wang , Dong Wang , Yan Ding
show 2 more authors
Bin Zhao Xuelong Li
This is my paper
classification cs.CVcs.AIcs.CLcs.RO
keywords vision-and-languagenavigationtextimaginationlargelanguagemodelsstateestimationQ-Formertuninggraph-basedpolicypartialobservabilityR2Rbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a navigation agent does not need to render future scenes pixel by pixel; it can imagine the key upcoming objects and rooms in words. It introduces Adaptive Text Dreamer (ATD), a dual-branch system built on a frozen large language model, where one branch estimates how far along the instruction the agent has progressed and the other writes short descriptions of what lies in each candidate direction. The imagined text is filtered by the state estimate and injected into a graph-based navigation policy. On the R2R benchmark the authors report state-of-the-art success rates with only 1.5B parameters, outperforming the 7B NaviLLM and the 1.5B NavGPT2. If true, this suggests that selective language-based imagination can replace expensive visual future synthesis in instruction-following navigation.

What carries the argument

The load-bearing mechanism is the paired left-right brain of frozen LLMs, each adapted by fine-tuning only the Q-Former on top of frozen InstructBLIP. The left brain is trained with cross-entropy loss to reproduce GPT-4V state-estimation reasoning, and the right brain is trained to reproduce Qwen2.5-VL captions of candidate-node panoramic stitches, giving it a text dreamer target. State Grounded Cross-Attention (SGCA) computes $A=\mathrm{SoftMax}(\mathrm{Sim}_{\cos}(Q_S,K_I))$ with state-estimation embeddings as queries and imagination embeddings as keys and values, then feeds the attended imagination into the graph policy through multi-head cross-attention. The navigation policy is a DUET-style graph transformer trained with behavior cloning and pseudo-interactive-demonstrator losses.

What would settle it

Run ATD on R2R with candidate-node images withheld from both the current observation and the LLM prompts, keeping only the current panorama; if the success-rate gain over the no-imagination baseline disappears, the effect is selective summarization of visible candidates, not imagination of unseen future scenes. Alternatively, feed the actual candidate captions into the policy in place of the imagination branch at inference; if SR does not change, the imaginative generation itself is not the source of the gain.

Watch

Extended reading notes

Core claim

ATD is a dual-branch self-guided imagination policy: a left-brain State Estimation LLM reasons about which part of the instruction has been completed and what the agent currently sees, while a right-brain Imagination LLM generates candidate-direction descriptions of future key environmental semantics. The right brain is constrained by the left brain through State Grounded Cross-Attention (SGCA), and the resulting latent is fused into the node embeddings of a graph-based navigation expert. On the R2R benchmark the paper reports gains of 8.0% and 12.0% in Success Rate and 5.0% and 11.0% in SPL over its NavGPT2-style baseline on val seen and val unseen, respectively, with a 1.5B-parameter model beating the fully fine-tuned 7B NaviLLM. The paper frames this as evidence that abstract textual imagination is more reliable and efficient than visual synthesis for handling partial observability.

Load-bearing premise

The right brain is trained on captions of candidate viewpoints that are one step ahead and are stitched from images reachable from the current position; the method assumes that learning from these captions teaches the agent to imagine genuinely unseen scenes rather than to summarize what is already visible.

Editorial extensions

If this is right

  • Language-form imagination can substitute for pixel-level future rendering in discrete VLN, cutting the cost of imagination while improving success.
  • Keeping the LLM frozen and tuning only the Q-Former is enough to activate navigation-specific reasoning, so a 1.5B model can outperform a fully fine-tuned 7B action generator.
  • The state-estimation branch acts as an adaptive filter: SGCA attention visualizations show completed instruction steps being suppressed as navigation proceeds.
  • The trained policy transfers zero-shot to REVERIE and R4R with consistent gains over the DUET baseline, indicating the imagined text helps rather than overfits to R2R.
  • Synthetic PREVALENT data remains necessary to avoid premature validation-loss plateaus, so the reported gains depend on that pre-training data as well as on the imagination branches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the imagination ground truth is captions of one-step-ahead candidate nodes, a charitable reading is that ATD learns selective summarization; a strict reading is that the partial-observability motivation is not tested. One could withhold all candidate images from input and see whether gains survive.
  • The same dual-branch recipe could be extended to multi-step-ahead imagination targets; the authors note the current data only covers one step ahead, so the ceiling of text dreaming is likely understated.
  • The SGCA design is a state-conditioned latent filter; any LLM-policy hybrid that suffers from stale or irrelevant text could reuse this cross-attention grounding.
  • A simple testable extension: at inference, replace the imagination branch's generated text with the true candidate captions; if performance is unchanged, the contribution is in fusion and grounding rather than in generation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 13 minor

Summary. The paper proposes ATD, a dual-branch language-imagination system for vision-and-language navigation. A left 'state estimation' branch uses a frozen InstructBLIP/FlanT5-XL with a tunable Q-Former to describe the current navigation state, while a right 'imagination' branch is trained on captions of candidate-node views to predict future scene semantics in text form. The two branches interact through a State-Grounded Cross-Attention (SGCA) layer, and the resulting embedding is injected into a DUET-style graph transformer policy. Experiments on R2R report state-of-the-art results among LLM-based methods with 1.5B parameters, together with ablations of the two branches, the SGCA layer count, and zero-shot cross-dataset evaluations on REVERIE and R4R.

Significance. If the claims hold, the paper is a meaningful step: it shows that language-form 'imagination' can replace expensive pixel-level future-scene generation in VLN, and that a parameter-efficient dual-branch design can outperform larger LLM-based navigators. The paper is clearly organized, the architecture is described in enough detail to be re-implemented in outline, and the ablations are useful: they show that both branches contribute and that the SGCA direction (state grounding imagination) beats reversed and parallel variants. The honest limitation statement in Sec. 9 that the imagination data covers only one step ahead is a point in favor of transparency. However, the central 'future imagination' claim rests on a data-collection description that is internally inconsistent, and the quantitative support lacks variance estimates and uses a weak ablation baseline; these issues need to be resolved before the headline claims can be fully accepted.

major comments (4)
  1. [Sec. 3.1, Eq. (5); Supplementary Sec. 6, Fig. 6] The training objective for the Imagination LLM is defined at the level of N per-candidate captions, but the collection procedure describes a single stitched-panorama caption. Eq. (5) writes L_rightbrain = -Σ_t Σ_i {C^i_candidate_t} log(I_t), which requires N separate caption sequences for the N candidate nodes at each step. Supplementary Sec. 6 instead says that images from all candidate nodes are stitched into one panoramic view and that Qwen2.5-VL is used to generate 'a caption' for that panoramic image, and Fig. 6 shows a single caption. If the actual target is one stitched caption, the sum over i is undefined (or N collapses to 1); if there are N per-candidate captions, the supplementary description and Fig. 6 are wrong. This ambiguity is load-bearing because the paper's core claim is that the right brain learns to predict future key semantics. The authors should specify the exact protocol, including whether candidate-node images are obtained by visiting those nodes and whether those images are visible from the current node, and then align Eq. (5), the prompt in Fig. 7, and the collection description with that protocol.
  2. [Sec. 4.5, Table 4; Abstract] The attribution of the reported gains to the imagination branch is not established because the ablation baseline is not a strong matched model. The text states that removing both SELLM and IMLLM yields 'DUET without the local branch and without BERT pretraining', while the abstract reports gains of 8%/12% SR and 5%/11% SPL. Such a weakened baseline confounds the effect of the new branches with the removal of standard DUET components. The most informative comparison in the paper is against NavGPT2 (Table 1), where the gains are 2-4 points in SR and 3-4 points in SPL on the validation/test splits. Please add an ablation that removes only the right-brain (Imagination) branch from the full ATD, or starts from a matched single-branch NavGPT2-style policy, so that the contribution of the text dreamer can be isolated.
  3. [Tables 1, 3, 4, 5, 7; Sec. 4.5] All reported numbers are single runs without error bars or multiple seeds. Several conclusions rest on very small differences: for example, in Table 5 the val-unseen SR values for 3 and 4 SGCA layers are 74.93 and 74.63, and the text nonetheless concludes that 3 or 4 layers are best; Table 3 shows SR differences of 1-2 points between FlanT5-XL and FlanT5-XXL. Given the 2-4 point gaps that separate ATD from NavGPT2 on some splits, the claim of state-of-the-art performance needs to be accompanied by variance estimates. Please report means and standard deviations over at least three seeds for the main results and for the key ablations.
  4. [Sec. 4.2; Supplementary Sec. 7.2] The parameter-efficiency claim should be made precise. The abstract states that ATD uses 1.5B parameters, but Supplementary Sec. 7.2 says 'the parameter size of our model is 1.5B because only the encoder was used during policy training, resulting in half of the parameters of the Flant5-XL model.' InstructBLIP with FlanT5-XL also includes a ViT, a Q-Former, and the decoder side of the LM, so the total number of weights in the full ATD system at inference is not simply 1.5B. Please specify the exact total and trainable parameter counts for the full ATD system (both VLM branches plus the graph policy) and compare those numbers consistently with NavGPT2 and NaviLLM.
minor comments (13)
  1. [Sec. 3.2, Eq. (9)] The text says 'with V_vis_t as the query and V_ATD_t as both the key and value', but Eq. (9) writes V_fusion_t = MCA(V_ATD_t, V_vis_t), which reverses the order under the standard MCA(query, key, value) convention. Please correct either the text or the equation.
  2. [Table 2 and Table 6] The zero-shot REVERIE numbers are inconsistent across tables: Table 2 lists OSR/SR/SPL of 33/27/23 for ATD(F lanT5-XXL), while Table 6 lists 32.52/27.04/22.51 for the same model. Please align the values and state the split and whether PREVALENT data was used.
  3. [Abstract; Sec. 4.5] The abstract's 'gains in SR of 8.0% and 12.0%' and 'SPL of 5.0% and 11.0%' are taken from the weak ablation baseline in Table 4, not from the comparison against the previous state of the art. Please clarify in the abstract that these gains are relative to a DUET-based baseline without the proposed branches.
  4. [Sec. 3.1, Eq. (5)] The notation {C^i_candidate_t} in Eq. (5) is informal and could be read as a set; please write the cross-entropy over token sequences explicitly and specify the masking over different caption lengths.
  5. [Sec. 3.1 and Fig. 7] The Imagination System Prompt asks the model to 'choose the one that best matches the instructions', but Eq. (5) and the caption-collection description do not explain how this choice is supervised or how it is used in the SGCA layer. Please clarify whether the choice token is part of the imagination output I_t.
  6. [Sec. 3.2 and Sec. 4.2] Please state explicitly which components are frozen and which are trainable during the second-stage policy training: are the Q-Formers and LLMs from the two branches frozen when the graph policy is trained?
  7. [Sec. 4.4 and Sec. 4.5] The attention visualization in Fig. 3 is qualitative, and several conclusions in Sec. 4.5 are based on differences of under one point (e.g., Table 5). Please either provide a quantitative measure of attention quality or soften the corresponding claims.
  8. [Sec. 4.3] The claim 'state-of-the-art performance with fewer parameters' should be qualified: in Table 1, BEVBert achieves comparable or higher val-unseen SR/SPL (75/64 vs 75/63) while using additional depth information, so the 'state-of-the-art' claim is best restricted to LLM-based methods or to methods without extra depth input.
  9. [Sec. 2 and Sec. 5] The conclusion states that ATD achieves 'lower computational costs' than visual-imagination methods, but no runtime, GPU memory, or inference-time measurements are reported. Please add a quantitative efficiency comparison.
  10. [Table 1 and References] There is a typo in the Table 1 caption ('ooutperforms'), and references [89] and [90] appear to be the same paper; please correct both.
  11. [Supplementary Sec. 7.2] The statement that PREVALENT data is 'vital' to prevent overfitting is important, but the paper does not report results without PREVALENT for the final ATD model. Please include these numbers so the reader can see the sensitivity to synthetic data.
  12. [Sec. 4.2] Please report the number of training episodes/samples used for the GPT-4V state-estimation labels and the Qwen2.5-VL imagination captions, as well as any filtering criteria, since these are the core training data for the two branches.
  13. [Sec. 4.5] The hyperparameter λ in the loss L = λL_BC + L_PID and the number of SGCA layers are free parameters; please state whether these were selected on the validation splits and include an ablation over λ.

Circularity Check

1 steps flagged · score 6.0 of 10

The right-brain 'imagination' is trained to reproduce captions of the candidate views already contained in the current observation, making the claimed future-scene prediction a fitted summarization of the input.

  1. fitted input called prediction [Sec. 3.1 (Task Setup; Eq. 4-5) and Supplementary Sec. 6 (Fig. 6)]
    "To obtain the Imagination LLM, long and detailed captions {C^i_candidate_t}^N_{i=1} for N candidate nodes at each sampled location in the trajectory are collected using Qwen2.5-VL-7B [5] as the ground truth for the text dreamer. ... At each time step t, the agent perceives its surrounding environment by observing a set of RGB views O_t ≜⟨o_i, a_i⟩^N_{i=1} for each connected navigable node candidate. ... for each current node, we first collect images from all candidate nodes and stitch them together into a panoramic view."

    By Eq. (5), L_rightbrain = -Σ_t Σ_i {C^i_candidate_t} log(I_t): the imagination output I_t is trained to reproduce captions of candidate nodes. Those candidate nodes are the very views o_i that the agent already observes at time t in O_t, and the supplementary collection procedure builds the ground-truth caption from images of all candidate nodes at the current node. Thus the 'future key environmental semantics' are not future: they are captions of the candidate views present in the current observation, so the model is fitted to output a selective summarization of its input. The central claim that ATD 'imaginatively predicts future scenes' is therefore, by construction, a renaming of current-observation captioning.

full rationale

The benchmark evaluation itself is not circular: ATD is tested on held-out R2R splits, and the two LLM branches are trained on external GPT-4V and Qwen2.5-VL captions, so the reported SR/SPL gains are not produced by fitting to the test set. There is also no load-bearing self-citation: references to the authors' own prior work (e.g., the MOMA-Kitchen citation) are peripheral. The circularity concern is localized to the central novelty claim of 'language-based future imagination': the right-brain Imagination LLM's ground truth is generated from the candidate node views that are part of the current observation, and its loss trains I_t to match those captions. Calling this 'imagining future key environmental semantics' renames a fitted captioning/summarization function as a predictive imagination mechanism. Because this is the paper's core conceptual contribution and the motivation for the dual-branch design, the central claim partially reduces to a distillation of the input, warranting a score of 6 rather than a fully independent derivation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two distilled supervision pipelines (GPT-4V state text and Qwen2.5-VL candidate captions) and on a DUET-style graph policy. No new physical entities are introduced. The loss trade-off lambda is a reported-but-unspecified free parameter, and the SGCA layer count is selected by ablation.

free parameters (2)
  • Loss trade-off lambda
    The total loss is defined as L = lambda * L_BC + L_PID in Sec. 3.2, but the paper never reports the value of lambda or a sensitivity analysis, so it is a chosen-but-unspecified hyperparameter.
  • Number of SGCA layers = 4
    Table 5 varies SGCA layers from 1 to 4 and selects 3 or 4; this is an architecture choice selected by validation performance rather than a fitted physical parameter.
assumptions (4)
  • domain assumption Candidate-node captioning by Qwen2.5-VL provides valid ground truth for what the agent should imagine about future directions.
    Used in Sec. 3.1 and Fig. 6; if these captions do not represent task-relevant future semantics, the right brain learns a different objective.
  • domain assumption The left brain's state estimate, distilled from GPT-4V outputs, accurately identifies completed and remaining instruction parts.
    Invoked to filter imagination in Sec. 3.2; errors here propagate into SGCA and into the policy inputs.
  • domain assumption Frozen InstructBLIP with only the Q-former fine-tuned can be activated to perform VLN-specific state estimation and imagination.
    Central training strategy in Sec. 3.1 and Sec. 4.2; no comparison with full fine-tuning is reported.
  • domain assumption The discrete R2R graph, with candidate nodes and precomputed views, faithfully models the partial observability the paper motivates.
    Used in the task setup and all experiments; the method is not evaluated in continuous environments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigation." pith.science (2026). https://pith.science/paper/MF2ZQSQA

@misc{pith2026250520897,
  author       = {Pith},
  title        = {Pith review of: Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MF2ZQSQA}},
  note         = {Machine review of arXiv:2505.20897}
}
read the original abstract

Vision-and-Language Navigation (VLN) requires the agent to navigate by following natural instructions under partial observability, making it difficult to align perception with language. Recent methods mitigate this by imagining future scenes, yet they rely on vision-based synthesis, leading to high computational cost and redundant details. To this end, we propose to adaptively imagine key environmental semantics via \textit{language} form, enabling a more reliable and efficient strategy. Specifically, we introduce a novel Adaptive Text Dreamer (ATD), a dual-branch self-guided imagination policy built upon a large language model (LLM). ATD is designed with a human-like left-right brain architecture, where the left brain focuses on logical integration, and the right brain is responsible for imaginative prediction of future scenes. To achieve this, we fine-tune only the Q-former within both brains to efficiently activate domain-specific knowledge in the LLM, enabling dynamic updates of logical reasoning and imagination during navigation. Furthermore, we introduce a cross-interaction mechanism to regularize the imagined outputs and inject them into a navigation expert module, allowing ATD to jointly exploit both the reasoning capacity of the LLM and the expertise of the navigation model. We conduct extensive experiments on the R2R benchmark, where ATD achieves state-of-the-art performance with fewer parameters. The code is \href{https://github.com/zhangpingrui/Adaptive-Text-Dreamer}{here}.

Figures

Figures reproduced from arXiv: 2505.20897 by the authors.

Figure 1
Figure 1. Given the task “Walk down the stairs and walk towards the red sofa. Turn left and enter [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the Adaptive Text Dreamer (ATD) architecture. The dual-branch left–right vision-language model (VLM) structure consists of a left-brain branch that performs state estimation using a large language model (LLM), and a right-brain branch that generates future candidate imaginations constrained by latent embeddings derived from the state estimation. The imagined latent representations are then integrated int… view at source ↗
Figure 4
Figure 4. Comparison of convergence speeds. The red vertical lines highlight the points where ATD surpasses NavGPT2’s maximum SPL and SR values. capability of GPT-4 and directly prompting the LLM to make decisions for the next action. Later, some methods [52, 66, 108] fine-tune LLMs as VLN experts, while NavGPT2 [110] connects the LLM and VLN specialist through a latent space. Main Results [PITH_FULL_IMAGE:figures/full_fig_p… view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: State Estimation generation prompt. 2 1 3 Qwen2.5-VL-7B-Instruct Imagination Intruction Collection Prompt {image} Describe the picture in detail, especially notice the furniture in the picture. Caption of Candidate Node [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Imagination Ground Truth Collection. 7 Additional Implementation Details 7.1 Evaluation Metrics In this section, we present a comprehensive overview of the evaluation metrics employed in our study. Consistent with prior research, we utilize five key metrics: Trajectory…
Figure 7
Figure 7. Figure 7: State Estimation System Prompt and Imagination System Prompt. • OSR. Measures whether any viewpoint along the predicted path falls within the goal set, reflecting the best possible success if the agent stopped at the closest point to the goal. Soracle =  1 if ∃vi ∈ G …
Figure 8
Figure 8. Figure 8: Visualization of attention matrix for every layer. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hume: Introducing System-2 Thinking in Visual-Language-Action Model

    cs.RO 2025-05 conditional novelty 6.0 of 10

    A dual-system vision-language-action model that improves robot control by ranking multiple sampled action chunks with a learned value function before fast execution.

  2. Towards Dual-Brain Minimal Sufficient Representation for Vision-Language Navigation

    cs.CV 2026-07 reject novelty 4.0 of 10

    A CP-decomposed, instruction-conditioned latent bottleneck (CompactNav) improves VLN-CE success rate by about 2% over prior state of the art on two benchmarks.

Reference graph

Works this paper leans on

111 extracted references · 42 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 3, 5, 10

    arXiv 2023

  2. [2]

    Bevbert: Multimodal map pre-training for language-guided navigation.arXiv preprint arXiv:2212.04385, 2022

    Dong An, Yuankai Qi, Yangguang Li, Yan Huang, Liang Wang, Tieniu Tan, and Jing Shao. Bevbert: Multimodal map pre-training for language-guided navigation.arXiv preprint arXiv:2212.04385, 2022. 3, 7, 8, 13

    arXiv 2022

  3. [3]

    On evaluation of embodied navigation agents.arXiv preprint arXiv:1807.06757, 2018

    Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents.arXiv preprint arXiv:1807.06757, 2018. 7

    arXiv 2018

  4. [4]

    Reid, Stephen Gould, and Anton van den Hengel

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually- grounded navigation instructions in real environments.2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3674–3683, 2017. 1, 2, 3, 7, 12

    2018

  5. [5]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 5

    arXiv 2025

  6. [6]

    Navigation world models.arXiv preprint arXiv:2412.03572, 2024

    Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models.arXiv preprint arXiv:2412.03572, 2024. 3

    arXiv 2024

  7. [7]

    Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields

    Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. InProceedings of the IEEE/CVF international conference on computer vision, pages 5855–5864, 2021. 3

    2021

  8. [8]

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky.π0: A visio...

    2024

Show all 111 references
  1. [9]

    Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems.arXiv preprint arXiv:2503.06669, 2025

    Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang, Shu Jiang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems.arXiv preprint arXiv:2503.06669, 2025. 2

    2025 arXiv

  2. [10]

    Select2plan: Training-free icl-based planning through vqa and memory retrieval.arXiv preprint arXiv:2411.04006, 2024

    Davide Buoso, Luke Robinson, Giuseppe Averta, Philip Torr, Tim Franzmeyer, and Daniele De Mar- tini. Select2plan: Training-free icl-based planning through vqa and memory retrieval.arXiv preprint arXiv:2411.04006, 2024. 3

    2024 arXiv

  3. [11]

    pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction

    David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19457–19467, 2024. 3

    2024

  4. [12]

    Touchdown: Natural language navigation and spatial reasoning in visual street environments

    Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12538–12547, 2019. 3

    2019

  5. [13]

    Affordances- oriented planning using foundation models for continuous vision-language navigation

    Jiaqi Chen, Bingqian Lin, Xinmin Liu, Lin Ma, Xiaodan Liang, and Kwan-Yee K Wong. Affordances- oriented planning using foundation models for continuous vision-language navigation. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23568–23576, 2025. 3

    2025

  6. [14]

    Mapgpt: Map-guided prompting for unified vision-and-language navigation.arXiv e-prints, pages arXiv–2401,

    Jiaqi Chen, Bingqian Lin, Ran Xu, Zhenhua Chai, Xiaodan Liang, and Kwan-Yee K Wong. Mapgpt: Map-guided prompting for unified vision-and-language navigation.arXiv e-prints, pages arXiv–2401,

  7. [15]

    Constraint-aware zero-shot vision-language navigation in continuous environments.arXiv preprint arXiv:2412.10137, 2024

    Kehan Chen, Dong An, Yan Huang, Rongtao Xu, Yifei Su, Yonggen Ling, Ian Reid, and Liang Wang. Constraint-aware zero-shot vision-language navigation in continuous environments.arXiv preprint arXiv:2412.10137, 2024. 3

    2024 arXiv

  8. [16]

    a2 nav: Action-aware zero-shot robot navigation by exploiting vision-and-language ability of foundation models.arXiv preprint arXiv:2308.07997, 2023

    Peihao Chen, Xinyu Sun, Hongyan Zhi, Runhao Zeng, Thomas H Li, Gaowen Liu, Mingkui Tan, and Chuang Gan. a2 nav: Action-aware zero-shot robot navigation by exploiting vision-and-language ability of foundation models.arXiv preprint arXiv:2308.07997, 2023. 3

    2023 arXiv

  9. [17]

    History aware multimodal trans- former for vision-and-language navigation.ArXiv, abs/2110.13309, 2021

    Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal trans- former for vision-and-language navigation.ArXiv, abs/2110.13309, 2021. 1, 3, 7, 13 16

    2021 arXiv

  10. [18]

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Think global, act local: Dual-scale graph transformer for vision-and-language navigation.2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16516–16526, 2022. 1,...

    2022

  11. [19]

    Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images

    Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. In European Conference on Computer Vision, pages 370–386. Springer, 2024. 3

    2024

  12. [20]

    Navila: Legged robot vision-language-action model for navigation.arXiv preprint arXiv:2412.04453, 2024

    An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Bıyık, Hongxu Yin, Sifei Liu, and Xiaolong Wang. Navila: Legged robot vision-language-action model for navigation.arXiv preprint arXiv:2412.04453, 2024. 3

    2024 arXiv

  13. [21]

    Unitedvln: Generalizable gaussian splatting for continuous vision-language navigation

    Guangzhao Dai, Jian Zhao, Yuantao Chen, Yusen Qin, Hao Zhao, Guosen Xie, Yazhou Yao, Xiangbo Shu, and Xuelong Li. Unitedvln: Generalizable gaussian splatting for continuous vision-language navigation. arXiv preprint arXiv:2411.16053, 2024. 3

    2024 arXiv

  14. [22]

    Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. 2, 4, 7

    2023

  15. [23]

    Embodied question answering

    Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 1–10, 2018. 3

    2018

  16. [24]

    Evolving graphical planner: Contextual global planning for vision-and-language navigation.ArXiv, abs/2007.05655, 2020

    Zhiwei Deng, Karthik Narasimhan, and Olga Russakovsky. Evolving graphical planner: Contextual global planning for vision-and-language navigation.ArXiv, abs/2007.05655, 2020. 3

    2007 arXiv

  17. [25]

    Navigation instruction generation with bev perception and large language models

    Sheng Fan, Rui Liu, Wenguan Wang, and Yi Yang. Navigation instruction generation with bev perception and large language models. InEuropean Conference on Computer Vision, pages 368–387. Springer, 2024. 3

    2024

  18. [26]

    Aerial vision-and- dialog navigation.arXiv preprint arXiv:2205.12219, 2022

    Yue Fan, Winson Chen, Tongzhou Jiang, Chun Zhou, Yi Zhang, and Xin Eric Wang. Aerial vision-and- dialog navigation.arXiv preprint arXiv:2205.12219, 2022. 3

    2022 arXiv

  19. [27]

    Speaker-follower models for vision-and-language navigation.Advances in neural information processing systems, 31, 2018

    Daniel Fried, Ronghang Hu, V olkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for vision-and-language navigation.Advances in neural information processing systems, 31...

    2018

  20. [28]

    Room-and-object aware knowledge reasoning for remote embodied referring expression

    Chen Gao, Jinyu Chen, Si Liu, Luting Wang, Qiong Zhang, and Qi Wu. Room-and-object aware knowledge reasoning for remote embodied referring expression. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3064–3073, 2021. 3

    2021

  21. [29]

    Airbert: In-domain pretraining for vision-and-language navigation

    Pierre-Louis Guhur, Makarand Tapaswi, Shizhe Chen, Ivan Laptev, and Cordelia Schmid. Airbert: In-domain pretraining for vision-and-language navigation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 1634–1643, 2021. 7, 13

    2021

  22. [30]

    Deep learning for real-time atari game play using offline monte-carlo tree search planning.Advances in neural information processing systems, 27, 2014

    Xiaoxiao Guo, Satinder Singh, Honglak Lee, Richard L Lewis, and Xiaoshi Wang. Deep learning for real-time atari game play using offline monte-carlo tree search planning.Advances in neural information processing systems, 27, 2014. 3

    2014

  23. [31]

    Towards learning a generic agent for vision-and-language navigation via pre-training

    Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning a generic agent for vision-and-language navigation via pre-training. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13137–13146, 2020. 3, 7, 12, 13

    2020

  24. [32]

    Vln bert: A recurrent vision-and-language bert for navigation

    Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould. Vln bert: A recurrent vision-and-language bert for navigation. InProceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 1643–1653, 2021. 1, 3, 7, 13

    2021

  25. [33]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR, 2019. 3

    2019

  26. [34]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022. 3

    2022

  27. [35]

    An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song- Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023. 3 17

    2023 arXiv

  28. [36]

    Visual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. InEuropean conference on computer vision, pages 709–727. Springer, 2022. 3

    2022

  29. [37]

    A new path: Scaling vision-and-language navigation with synthetic instructions and imitation learning

    Aishwarya Kamath, Peter Anderson, Su Wang, Jing Yu Koh, Alexander Ku, Austin Waters, Yinfei Yang, Jason Baldridge, and Zarana Parekh. A new path: Scaling vision-and-language navigation with synthetic instructions and imitation learning. InProceedings of the IEEE/CVF Conference...

    2023

  30. [38]

    Tactical rewind: Self-correction via backtracking in vision-and-language navigation

    Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Siddhartha Srinivasa. Tactical rewind: Self-correction via backtracking in vision-and-language navigation. InProceedings of the IEEE/CVF conference on computer vision and p...

    2019

  31. [39]

    3d gaussian splatting for real-time radiance field rendering.ACM Trans

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering.ACM Trans. Graph., 42(4):139–1, 2023. 3

    2023

  32. [40]

    Bandit based monte-carlo planning

    Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. InEuropean conference on machine learning, pages 282–293. Springer, 2006. 3

    2006

  33. [41]

    Pathdreamer: A world model for indoor navigation

    Jing Yu Koh, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson. Pathdreamer: A world model for indoor navigation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 14738–14748, 2021. 3

    2021

  34. [42]

    Waypoint models for instruction-guided navigation in continuous environments

    Jacob Krantz, Aaron Gokaslan, Dhruv Batra, Stefan Lee, and Oleksandr Maksymets. Waypoint models for instruction-guided navigation in continuous environments. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15162–15171, 2021. 3

    2021

  35. [43]

    Beyond the nav-graph: Vision-and-language navigation in continuous environments

    Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision-and-language navigation in continuous environments. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVIII 16, pages ...

    2020

  36. [44]

    Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding.arXiv preprint arXiv:2010.07954, 2020

    Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding.arXiv preprint arXiv:2010.07954, 2020. 3

    2010 arXiv

  37. [45]

    Citynav: Language-goal aerial navigation dataset with geographic information.arXiv preprint arXiv:2406.14240, 2024

    Jungdae Lee, Taiki Miyanishi, Shuhei Kurita, Koya Sakamoto, Daichi Azuma, Yutaka Matsuo, and Nakamasa Inoue. Citynav: Language-goal aerial navigation dataset with geographic information.arXiv preprint arXiv:2406.14240, 2024. 3

    2024 arXiv

  38. [46]

    Jialu Li and Mohit Bansal. Improving vision-and-language navigation by generating future-view image semantics.2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10803–10812, 2023. 3

    2023

  39. [47]

    Panogen: Text-conditioned panoramic environment generation for vision-and- language navigation

    Jialu Li and Mohit Bansal. Panogen: Text-conditioned panoramic environment generation for vision-and- language navigation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 21878...

    2023

  40. [48]

    Envedit: Environment editing for vision-and-language navigation

    Jialu Li, Hao Tan, and Mohit Bansal. Envedit: Environment editing for vision-and-language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15407– 15417, 2022. 3

    2022

  41. [49]

    Walking with mind: Mental imagery enhanced embodied qa

    Juncheng Li, Siliang Tang, Fei Wu, and Yueting Zhuang. Walking with mind: Mental imagery enhanced embodied qa. InProceedings of the 27th ACM International Conference on Multimedia, pages 1211–1219,

  42. [50]

    Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023. 2, 4

    2023

  43. [51]

    Kerm: Knowledge enhanced reasoning for vision-and-language navigation.2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2583–2592, 2023

    Xiangyang Li, Zihan Wang, Jiahao Yang, Yaowei Wang, and Shuqiang Jiang. Kerm: Knowledge enhanced reasoning for vision-and-language navigation.2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2583–2592, 2023. 1, 3

    2023

  44. [52]

    Navcot: Boosting llm-based vision-and-language navigation via learning disentangled reasoning.arXiv preprint arXiv:2403.07376, 2024

    Bingqian Lin, Yunshuang Nie, Ziming Wei, Jiaqi Chen, Shikui Ma, Jianhua Han, Hang Xu, Xiaojun Chang, and Xiaodan Liang. Navcot: Boosting llm-based vision-and-language navigation via learning disentangled reasoning.arXiv preprint arXiv:2403.07376, 2024. 7, 8, 13 18

    2024 arXiv

  45. [53]

    Correctable landmark discovery via large models for vision-language navigation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    Bingqian Lin, Yunshuang Nie, Ziming Wei, Yi Zhu, Hang Xu, Shikui Ma, Jianzhuang Liu, and Xiao- dan Liang. Correctable landmark discovery via large models for vision-language navigation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 4

    2024

  46. [54]

    Learning to model the world with language.arXiv preprint arXiv:2308.01399, 2023

    Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan. Learning to model the world with language.arXiv preprint arXiv:2308.01399, 2023. 2

    2023 arXiv

  47. [55]

    V olumetric environment representation for vision-language navigation.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16317–16328, 2024

    Rui Liu, Wenguan Wang, and Yi Yang. V olumetric environment representation for vision-language navigation.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16317–16328, 2024. 3

    2024

  48. [56]

    Instructnav: Zero-shot system for generic instruction navigation in unexplored environment.arXiv preprint arXiv:2406.04882,

    Yuxing Long, Wenzhe Cai, Hongcheng Wang, Guanqi Zhan, and Hao Dong. Instructnav: Zero-shot system for generic instruction navigation in unexplored environment.arXiv preprint arXiv:2406.04882,

    arXiv

  49. [57]

    Discuss before moving: Visual language navigation via multi-expert discussions

    Yuxing Long, Xiaoqi Li, Wenzhe Cai, and Hao Dong. Discuss before moving: Visual language navigation via multi-expert discussions. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 17380–17387. IEEE, 2024. 3, 7, 13

    2024

  50. [58]

    Self-monitoring navigation agent via auxiliary progress estimation.arXiv preprint arXiv:1901.03035,

    Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. Self-monitoring navigation agent via auxiliary progress estimation.arXiv preprint arXiv:1901.03035,

    1901 arXiv

  51. [59]

    The regretful agent: Heuristic-aided navigation through progress estimation

    Chih-Yao Ma, Zuxuan Wu, Ghassan AlRegib, Caiming Xiong, and Zsolt Kira. The regretful agent: Heuristic-aided navigation through progress estimation. InProceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 6732–6740, 2019. 3

    2019

  52. [60]

    Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 1:3, 2020

    Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 1:3, 2020. 3

    2005 arXiv

  53. [61]

    Nerf: Representing scenes as neural radiance fields for view synthesis.Communications of the ACM, 65(1):99–106, 2021

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis.Communications of the ACM, 65(1):99–106, 2021. 3

    2021

  54. [62]

    Instant neural graphics primitives with a multiresolution hash encoding.ACM transactions on graphics (TOG), 41(4):1–15, 2022

    Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding.ACM transactions on graphics (TOG), 41(4):1–15, 2022. 3

    2022

  55. [64]

    Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning.arXiv preprint arXiv:1909.01871, 2019

    Khanh Nguyen and Hal Daumé III. Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning.arXiv preprint arXiv:1909.01871, 2019. 3

    1909 arXiv

  56. [65]

    Wmnav: Integrating vision- language models into world models for object goal navigation.arXiv preprint arXiv:2503.02247, 2025

    Dujun Nie, Xianda Guo, Yiqun Duan, Ruijun Zhang, and Long Chen. Wmnav: Integrating vision- language models into world models for object goal navigation.arXiv preprint arXiv:2503.02247, 2025. 3

    2025 arXiv

  57. [66]

    Langnav: Language as a perceptual representation for navigation.arXiv preprint arXiv:2310.07889, 2023

    Bowen Pan, Rameswar Panda, SouYoung Jin, Rogerio Feris, Aude Oliva, Phillip Isola, and Yoon Kim. Langnav: Language as a perceptual representation for navigation.arXiv preprint arXiv:2310.07889, 2023. 7, 8, 14

    2023 arXiv

  58. [67]

    Planning from imagination: Episodic simulation and episodic memory for vision-and-language navigation

    Yiyuan Pan, Yunzhe Xu, Zhe Liu, and Hesheng Wang. Planning from imagination: Episodic simulation and episodic memory for vision-and-language navigation. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 6345–6353, 2025. 3

    2025

  59. [68]

    Episodic transformer for vision-and-language navigation

    Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision-and-language navigation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15942–15952, 2021. 3

    2021

  60. [69]

    Reverie: Remote embodied visual referring expression in real indoor environments

    Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages...

  61. [70]

    Hop: History-and-order aware pre-training for vision-and-language navigation

    Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop: History-and-order aware pre-training for vision-and-language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15418–15427, 2022. 3 19

    2022

  62. [71]

    Hop+: History-enhanced and order-aware pre-training for vision-and-language navigation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):8524–8537, 2023

    Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop+: History-enhanced and order-aware pre-training for vision-and-language navigation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):8524–8537, 2023. 3, 7, 13

    2023

  63. [72]

    March in chat: Interactive prompting for remote embodied referring expression

    Yanyuan Qiao, Yuankai Qi, Zheng Yu, Jing Liu, and Qi Wu. March in chat: Interactive prompting for remote embodied referring expression. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15758–15767, 2023. 3

    2023

  64. [73]

    Saynav: Grounding large language models for dynamic planning to navigation in new environments

    Abhinav Rajvanshi, Karan Sikka, Xiao Lin, Bhoram Lee, Han-Pang Chiu, and Alvaro Velasquez. Saynav: Grounding large language models for dynamic planning to navigation in new environments. InProceed- ings of the International Conference on Automated Planning and Scheduling, volu...

  65. [74]

    A reduction of imitation learning and structured prediction to no-regret online learning

    Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. InProceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627–635. JMLR Workshop and Conf...

    2011

  66. [75]

    A reduction of imitation learning and structured prediction to no-regret online learning

    Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. InProceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627–635. JMLR Workshop and Conf...

    2011

  67. [76]

    Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action

    Dhruv Shah, Bła˙zej Osi´nski, Sergey Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. InConference on robot learning, pages 492–504. PMLR, 2023. 3

    2023

  68. [77]

    Foresightnav: Learning scene imagination for efficient exploration.arXiv preprint arXiv:2504.16062,

    Hardik Shah, Jiaxu Xing, Nico Messikommer, Boyang Sun, Marc Pollefeys, and Davide Scaramuzza. Foresightnav: Learning scene imagination for efficient exploration.arXiv preprint arXiv:2504.16062,

    arXiv

  69. [78]

    Alfred: A benchmark for interpreting grounded instructions for everyday tasks

    Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. InProceedings of the IEEE/CVF conference on computer vision and pattern r...

    2020

  70. [79]

    Sadler, Wei-Lun Chao, and Yu Su

    Chan Hee Song, Jiaman Wu, Clay Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. Llm- planner: Few-shot grounded planning for embodied agents with large language models.2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2986–2997, 2022. 3

    2023

  71. [80]

    Learning to navigate unseen environments: Back translation with environmental dropout.arXiv preprint arXiv:1904.04195, 2019

    Hao Tan, Licheng Yu, and Mohit Bansal. Learning to navigate unseen environments: Back translation with environmental dropout.arXiv preprint arXiv:1904.04195, 2019. 3, 7, 12

    1904 arXiv

  72. [81]

    Vision-and-dialog navigation

    Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation. InConference on Robot Learning, pages 394–406. PMLR, 2020. 3

    2020

  73. [82]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017. 6

    2017

  74. [83]

    Dreamwalker: Mental planning for continuous vision-language navigation.2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 10839–10849, 2023

    Hanqing Wang, Wei Liang, Luc Van Gool, and Wenguan Wang. Dreamwalker: Mental planning for continuous vision-language navigation.2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 10839–10849, 2023. 2, 3

    2023

  75. [84]

    Counterfactual cycle- consistent learning for instruction following and generation in vision-language navigation

    Hanqing Wang, Wei Liang, Jianbing Shen, Luc Van Gool, and Wenguan Wang. Counterfactual cycle- consistent learning for instruction following and generation in vision-language navigation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages ...

    2022

  76. [85]

    Soft expert reward learning for vision-and-language navigation

    Hu Wang, Qi Wu, and Chunhua Shen. Soft expert reward learning for vision-and-language navigation. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16, pages 126–141. Springer, 2020. 3

    2020

  77. [86]

    Panogen++: Domain-adapted text-guided panoramic environment generation for vision-and-language navigation.Neural Networks, 187:107320, 2025

    Sen Wang, Dongliang Zhou, Liang Xie, Chao Xu, Ye Yan, and Erwei Yin. Panogen++: Domain-adapted text-guided panoramic environment generation for vision-and-language navigation.Neural Networks, 187:107320, 2025. 3

    2025

  78. [87]

    Mos: Unleashing parameter efficiency of low-rank adaptation with mixture of shards.arXiv preprint arXiv:2410.00938, 2024

    Sheng Wang, Liheng Chen, Pengan Chen, Jingwei Dong, Boyang Xue, Jiyue Jiang, Lingpeng Kong, and Chuan Wu. Mos: Unleashing parameter efficiency of low-rank adaptation with mixture of shards.arXiv preprint arXiv:2410.00938, 2024. 3 20

    arXiv

  79. [88]

    Lana: A language-capable navigator for instruction following and generation

    Xiaohan Wang, Wenguan Wang, Jiayi Shao, and Yi Yang. Lana: A language-capable navigator for instruction following and generation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19048–19058, 2023. 3

    2023

  80. [89]

    Reinforced cross-modal matching and self-supervised imita- tion learning for vision-language navigation

    Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imita- tion learning for vision-language navigation. InProceedings of the IEEE/CVF conference on compute...

    2019

  81. [90]

    Reinforced cross-modal matching and self-supervised imita- tion learning for vision-language navigation

    Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imita- tion learning for vision-language navigation. InProceedings of the IEEE/CVF Conference on Compute...

    2019

  82. [91]

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, Junjie Hu, Ming Jiang, and Shuqiang Jiang. Looka- head exploration with neural radiance representation for continuous vision-language navigation.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1...

    2024

  83. [92]

    Gridmm: Grid memory map for vision-and-language navigation

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang. Gridmm: Grid memory map for vision-and-language navigation. InProceedings of the IEEE/CVF International conference on computer vision, pages 15625–15636, 2023. 1, 3

    2023

  84. [93]

    Scaling data generation in vision-and-language navigation.2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975–11986, 2023

    Zun Wang, Jialu Li, Yicong Hong, Yi Wang, Qi Wu, Mohit Bansal, Stephen Gould, Hao Tan, and Yu Qiao. Scaling data generation in vision-and-language navigation.2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975–11986, 2023. 3

    2023

  85. [94]

    Unseen from seen: Rewriting observation-instruction using foundation models for augmenting vision-language navigation.arXiv preprint arXiv:2503.18065, 2025

    Ziming Wei, Bingqian Lin, Yunshuang Nie, Jiaqi Chen, Shikui Ma, Hang Xu, and Xiaodan Liang. Unseen from seen: Rewriting observation-instruction using foundation models for augmenting vision-language navigation.arXiv preprint arXiv:2503.18065, 2025. 3

    2025

  86. [95]

    From foresight to forethought: Vlm-in-the-loop policy steering via latent alignment.arXiv preprint arXiv:2502.01828, 2025

    Yilin Wu, Ran Tian, Gokul Swamy, and Andrea Bajcsy. From foresight to forethought: Vlm-in-the-loop policy steering via latent alignment.arXiv preprint arXiv:2502.01828, 2025. 2

    2025 arXiv

  87. [96]

    Flame: Learning to navigate with multimodal llm in urban environments

    Yunzhe Xu, Yiyuan Pan, Zhe Liu, and Hesheng Wang. Flame: Learning to navigate with multimodal llm in urban environments. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 9005–9013, 2025. 4

    2025

  88. [97]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024. 10

    2024 arXiv

  89. [98]

    Mc-gpt: Empowering vision-and-language navigation with memory map and reasoning chains.arXiv preprint arXiv:2405.10620, 2024

    Zhaohuan Zhan, Lisha Yu, Sijie Yu, and Guang Tan. Mc-gpt: Empowering vision-and-language navigation with memory map and reasoning chains.arXiv preprint arXiv:2405.10620, 2024. 3

    2024 arXiv

  90. [99]

    Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks.arXiv preprint arXiv:2412.06224, 2024

    Jiazhao Zhang, Kunyu Wang, Shaoan Wang, Minghan Li, Haoran Liu, Songlin Wei, Zhongyuan Wang, Zhizheng Zhang, and He Wang. Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks.arXiv preprint arXiv:2412.06224, 2024. 3

    2024 arXiv

  91. [100]

    Navid: Video-based vlm plans the next step for vision-and-language navigation.arXiv preprint arXiv:2402.15852, 2024

    Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and He Wang. Navid: Video-based vlm plans the next step for vision-and-language navigation.arXiv preprint arXiv:2402.15852, 2024. 3

    2024 arXiv

  92. [101]

    Mapnav: A novel memory representation via annotated semantic maps for vlm-based vision-and-language navigation.arXiv preprint arXiv:2502.13451,

    Lingfeng Zhang, Xiaoshuai Hao, Qinwen Xu, Qiang Zhang, Xinyao Zhang, Pengwei Wang, Jing Zhang, Zhongyuan Wang, Shanghang Zhang, and Renjing Xu. Mapnav: A novel memory representation via annotated semantic maps for vlm-based vision-and-language navigation.arXiv preprint arXiv:2...

    arXiv

  93. [102]

    Moma-kitchen: A 100k+ benchmark for affordance-grounded last-mile navigation in mobile manipulation.arXiv preprint arXiv:2503.11081, 2025

    Pingrui Zhang, Xianqiang Gao, Yuhan Wu, Kehui Liu, Dong Wang, Zhigang Wang, Bin Zhao, Yan Ding, and Xuelong Li. Moma-kitchen: A 100k+ benchmark for affordance-grounded last-mile navigation in mobile manipulation.arXiv preprint arXiv:2503.11081, 2025. 3

    2025 arXiv

  94. [103]

    Flexvln: Flexible adaptation for diverse vision-and-language navigation tasks.arXiv preprint arXiv:2503.13966, 2025

    Siqi Zhang, Yanyuan Qiao, Qunbo Wang, Longteng Guo, Zhihua Wei, and Jing Liu. Flexvln: Flexible adaptation for diverse vision-and-language navigation tasks.arXiv preprint arXiv:2503.13966, 2025. 3

    2025 arXiv

  95. [104]

    Imagine before go: Self-supervised generative map for object goal navigation

    Sixian Zhang, Xinyao Yu, Xinhang Song, Xiaohan Wang, and Shuqiang Jiang. Imagine before go: Self-supervised generative map for object goal navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16414–16425, 2024. 3 21

    2024

  96. [105]

    Question-guided knowledge graph re-scoring and injection for knowledge graph question answering.arXiv preprint arXiv:2410.01401,

    Yu Zhang, Kehai Chen, Xuefeng Bai, Quanjiang Guo, Min Zhang, et al. Question-guided knowledge graph re-scoring and injection for knowledge graph question answering.arXiv preprint arXiv:2410.01401,

    arXiv

  97. [106]

    On the evaluation of vision-and-language navigation instructions.arXiv preprint arXiv:2101.10504, 2021

    Ming Zhao, Peter Anderson, Vihan Jain, Su Wang, Alexander Ku, Jason Baldridge, and Eugene Ie. On the evaluation of vision-and-language navigation instructions.arXiv preprint arXiv:2101.10504, 2021. 3

    2021 arXiv

  98. [107]

    Imaginenav: Prompting vision-language models as embodied navigator through scene imagination.arXiv preprint arXiv:2410.09874, 2024

    Xinxin Zhao, Wenzhe Cai, Likun Tang, and Teng Wang. Imaginenav: Prompting vision-language models as embodied navigator through scene imagination.arXiv preprint arXiv:2410.09874, 2024. 3

    2024 arXiv

  99. [108]

    Towards learning a generalist model for embodied navigation

    Duo Zheng, Shijia Huang, Lin Zhao, Yiwu Zhong, and Liwei Wang. Towards learning a generalist model for embodied navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13624–13634, 2024. 3, 7, 8, 14

    2024

  100. [109]

    P3nav: A unified framework for embodied navigation integrating perception, planning, and prediction.arXiv preprint arXiv:2503.18525, 2025

    Yufeng Zhong, Chengjian Feng, Feng Yan, Fanfan Liu, Liming Zheng, and Lin Ma. P3nav: A unified framework for embodied navigation integrating perception, planning, and prediction.arXiv preprint arXiv:2503.18525, 2025. 3

    2025 arXiv

  101. [110]

    Navgpt-2: Unleashing navigational reasoning capability for large vision-language models

    Gengze Zhou, Yicong Hong, Zun Wang, Xin Eric Wang, and Qi Wu. Navgpt-2: Unleashing navigational reasoning capability for large vision-language models. InEuropean Conference on Computer Vision,

  102. [111]

    Navgpt: Explicit reasoning in vision-and-language navigation with large language models

    Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-language navigation with large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7641–7649, 2024. 3, 7, 13

    2024

  103. [112]

    Vision-language navigation with self-supervised auxiliary reasoning tasks

    Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. Vision-language navigation with self-supervised auxiliary reasoning tasks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10012–10022, 2020. 3 22

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.