Pith. sign in

REVIEW 4 major objections 4 minor 86 references

NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environments

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read NavMorph claims that a self-evolving latent world model with a scene-context memory raises VLN-CE success and path efficiency, reporting monocular R2R-CE unseen SR of 47.91 versus 43.77 for the base agent.

desk verdict A solid world-model-plus-memory contribution to VLN-CE; the headline tables mix an online-adapting model with static baselines, but the core result survives even under the fair static comparison. read the letter →

arxiv 2506.23468 v2 pith:JNMJEG35 submitted 2025-06-30 cs.CV

Xuan Yao , Junyu Gao , Changsheng Xu This is my paper · ORCID
classification cs.CV
keywords vision-and-languagenavigationcontinuousenvironmentsworldmodelslatentdynamicsonlineadaptationscenememoryRSSMwaypoint
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NavMorph argues that a vision-and-language navigation agent should carry a compact model of how its environment changes as it acts, and should keep updating that model while it navigates. To test this, the paper builds a world-model layer on top of existing VLN-CE agents: a Recurrent State-Space Model-style latent dynamics network gives the agent foresight about future states, and a Contextual Evolution Memory stores scene features that are retrieved and refreshed online. On R2R-CE Val Unseen in the monocular setting, success rate rises from 43.77% with the base agent to 47.91%, and SPL rises from 29.39 to 33.22; on RxR-CE Val Unseen, SR rises from 26.66% to 30.76%. The central claim is that these gains come from predicting futures in latent space plus memory-based self-evolution, not merely from adding action information to the existing model.

What carries the argument

The central object is a latent dynamics world model of the RSSM family, customized for waypoint navigation: a deterministic recurrent state $h_t$, a stochastic state $s_t$, a visual encoder/decoder pair $e_\phi/d_\theta$, and a shared CEM-enhanced recurrence $f$. The named component is Contextual Evolution Memory (CEM), a bank of $N_m=1000$ scene-contextual vectors with top-$K$ cosine retrieval and exponential forward updates ($\alpha=\beta=0.7$ in the monocular setting). The Foresight Action Planner uses these states to predict $T_p=2$ future latent states, decodes them into visual embeddings rather than pixels, and guides the policy by weighting candidate waypoints closer to the imagined trajectory. The full model is trained with a variational lower bound on the data log-likelihood plus an imitation-learning term, with NDTW-regularized reconstruction and action prediction losses enforcing temporal consistency between predicted and actual trajectories.

What would settle it

Run the released model on R2R-CE Val Unseen with CEM forward updates disabled during testing but all weights identical. The paper reports this variant at 45.08 SR; an independent run that shows no drop from 47.91 would falsify the claim that online memory accumulation contributes. Conversely, resetting CEM before every episode and seeing no loss in SR would falsify the claim that cross-episode memory reuse matters.

Watch

Extended reading notes

Core claim

The paper's central claim is that a world model with online self-evolution is a better backbone for VLN-CE than the static policies used by most baselines. NavMorph treats navigation as latent-state inference: an encoder turns RGB-D observations into visual embeddings, a recurrent model maintains a deterministic history, and a stochastic state captures uncertainty; the predictive network then rolls the latent state forward $T_p$ steps to imagine future embeddings and actions, and the navigation policy scores candidate waypoints by their proximity to the imagined path. The Contextual Evolution Memory augments the recurrent state by retrieving its top-$K$ scene features by cosine similarity and blending them with an enhancement factor, and during both training and testing the retrieved entries are updated by a forward move $v_k \leftarrow (1-\beta)v_k + \beta h_t$, so the model accumulates scene context without backpropagation at test time. The paper reports consistent gains over both monocular and panoramic baselines, and its ablations attribute part of the gain to the world-model losses (visual reconstruction, action prediction, posterior-prior matching) and part to the online self-evolution.

Load-bearing premise

The headline comparisons allow the Contextual Evolution Memory to store and reuse scene features from the test episodes during online evaluation, while most baseline numbers in the main tables come from static models; if test-time learning is not permitted, the paper's no-self-evolution variant is the fair comparison and the reported margins shrink.

Editorial extensions

If this is right

  • The same learned weights can keep improving in a new building during an episode, because CEM accumulates scene context without needing ground-truth actions.
  • Because memory updates are forward passes rather than gradient steps, online adaptation adds little time: average per-instruction cost is 21.22s versus 20.53s for the base model, while the LSTM-based gradient test-time adaptation variant takes 44.56s.
  • Foresight is computed in latent space and used to re-rank candidate waypoints, so NavMorph can be layered on existing monocular and panoramic waypoint agents rather than replacing the waypoint predictor.
  • The ablation gap (47.91 vs 45.08 SR without self-evolution) quantifies the online-memory contribution, while the world-model losses alone still exceed the base model (45.08 vs 43.77).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If an evaluation protocol resets or freezes CEM at episode boundaries, the headline margins over static baselines would not hold; the comparable numbers would be the paper's no-self-evolution variant.
  • Because CEM stores scene-contextual vectors with top-K retrieval, ablating memory content by clustering seen scenes could test whether the gains come from recognizing scene identity rather than from generic visual priors.
  • The paper shows imagined paths sometimes cut through obstacles; adding a collision penalty to the world-model losses could convert foresight gains into larger SPL improvements.
  • The same latent-dynamics-plus-memory recipe could transfer to other embodied tasks with low-level actions, such as object navigation or point-goal navigation, where online adaptation matters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes NavMorph, a world-model framework for Vision-and-Language Navigation in Continuous Environments (VLN-CE). NavMorph combines an RSSM-style latent dynamics model (World-aware Navigator) with a Foresight Action Planner that predicts future visual embeddings and actions in latent space, and a Contextual Evolution Memory (CEM) that stores and iteratively updates scene-contextual features. Training maximizes a variational lower bound with reconstruction, action-prediction, and KL terms, together with an imitation objective and an NDTW-based regularization. The method is evaluated on R2R-CE and RxR-CE in both monocular and panoramic settings, reporting gains over VLN-3DFF, ETPNav, and HNR, with ablations over loss components, CEM design, memory size, predictive steps, and test-time-adaptation baselines.

Significance. If the reported results are reliable, NavMorph is a meaningful advance for VLN-CE: it demonstrates that a latent world model with foresight planning can be integrated into existing monocular and panoramic navigation policies and improve success and path efficiency. The paper has several strengths: the ablation study is fairly complete, the supplement includes variance estimates for the TTA comparison, the discussion is candid about the online memory mechanism, and the authors promise code release. In particular, Supplementary Table 8 shows that the world model without online evolution still improves SR from 43.77 to 46.87 on R2R-CE Val Unseen, so the core architectural contribution appears credible. The main unresolved issue is the evaluation protocol for cross-episode CEM updates, which affects how the headline SOTA comparisons should be interpreted.

major comments (4)
  1. [§3.4, §4.1, Tables 1–2] The main SOTA comparisons are asymmetric because NavMorph updates CEM across test episodes during online evaluation (Eq. (3), §3.4, batch size 1 in §4.1), while most baselines in Tables 1–2 are static models. The paper's own Supplementary Table 8 shows that the online-updated variant reaches SR 47.91 while the no-online-evolution variant 'NavMorph w/o SE*' reaches 46.87 and the base model reaches 43.77; thus the headline 4.14-point gain over VLN-3DFF* is a blend of ~3.1 points of architectural improvement and ~1.0 point of cross-episode test-time memory accumulation. The authors should clearly state the protocol in the main tables, identify which baselines were also run with online adaptation, and report the static variant as the primary point of comparison if the benchmark is intended to measure static generalization.
  2. [Table 1, DREAMW rows] The DREAMW rows contain impossible numbers: OSR cannot be lower than SR, yet the table reports OSR 49 and SR 59 on Val Unseen, OSR 59 and SR 66 on Val Seen, and OSR 49 and SR 57 on Test Unseen. This indicates that the baseline table was not carefully verified against the original publication, and it undercuts confidence in the surrounding comparisons. The authors should correct these entries and re-verify all other baseline numbers in Tables 1, 2, 6, and 7.
  3. [§3.3, Eq. (6), Tables 1–2] NDTW is used both as a training regularization term (Eq. (6), applied to the reconstruction and action-prediction losses) and as an evaluation metric reported in NDTW and SDTW. Consequently, the reported NDTW/SDTW gains are partially driven by optimizing the evaluation criterion itself, and these metrics should not be presented as neutral evidence of better path fidelity. The paper should state this overlap explicitly and ideally provide an ablation that removes the NDTW regularization term to quantify its effect on the reported metrics.
  4. [Tables 1–2 and Supplementary Table 10] The main SOTA tables do not report variance or the number of seeds for NavMorph or the baselines, even though the gains over ETPNav and HNR are often small (1–3 percentage points). Supplementary Table 10 reports mean ± standard deviation only for the TTA comparison; the same rigor should be applied to the headline results before the paper can claim consistent improvements over state-of-the-art methods.
minor comments (4)
  1. [§4.3 and Table 3] The two no-evolution variants 'NavMorph w/o SE' (Table 3, SR 45.08) and 'NavMorph w/o SE*' (Supplementary Table 8, SR 46.87) should be defined in the main text and clearly distinguished, since they differ by 1.8 SR points and the discussion refers to both.
  2. [Eq. (5) and Supplementary Eq. (13)] The notation in the future prediction terms is inconsistent: Eq. (5) writes p(x_{T+j}|h_T,s_T) without hats and without explicit sampling over future latent states, while the supplementary derivation in Eq. (13) places the latent transition terms inside the log-likelihood and then drops them; the derivation should be rewritten so that the ELBO steps are formally clean.
  3. [§3.1] The task description says the agent predicts actions by selecting a candidate sub-goal point, but the action is later defined as a delta-position Δposition_t; these two notions should be reconciled.
  4. [§4.3 and Table 4 note] There are several typographical errors, including 'memroy' in §4.3, 'Supplymentary' in §4.3, 'agnet' in Table 4, and ' )is' in the supplementary evaluation-metrics section; a careful proofread is needed.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the variational world-model derivation is self-contained and the NDTW-in-loss / test-time CEM updates are disclosed mechanisms rather than circular reductions.

full rationale

The paper's derivation chain is self-contained. The world model loss LW (Eq. 5) is derived from a standard variational lower bound in the supplementary (Eqs. 7-16), with no target result fed back as an input. The NDTW term in Eq. 6 appears both in the training loss and in the reported NDTW/SDTW metrics, but the training loss compares predicted sequences against ground-truth sequences from training episodes; the evaluation metrics are computed on held-out episodes, so there is no by-construction equivalence. The Contextual Evolution Memory (CEM) is updated online during testing via Eq. (3), which the paper explicitly discloses in Section 4.3 as 'accumulates scene-specific information from test environments,' and it validates the effect of this mechanism with the w/o SE and w/o SE* ablations (SR 43.77 base, 46.87 w/o SE*, 47.91 full on R2R-CE Val Unseen). This is a test-time adaptation protocol, not a circular derivation. The only minor self-citation is the reliance on the authors' own FSTTA [19] to justify the online VLN evaluation setting (batch size 1, Section 4.1) and the fairness of cross-episode memory accumulation; FSTTA is a published ICML paper and the paper provides an internal static-model comparison, so this citation is not load-bearing for the method's core architecture. The impossible DREAMW row in Table 1 (OSR 49 < SR 59) is a data-integrity concern, not circularity. Overall, no prediction or derivation reduces to its own inputs by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central claim rests on a standard variational world-model loss plus several hand-set hyperparameters (N_m, K, alpha, beta, gamma, T_p). CEM is a new component without independent theoretical backing. The main non-standard assumption is that online test-time memory updates are an acceptable evaluation protocol.

free parameters (5)
  • Memory size N_m = 1000
    Chosen via ablation on R2R-CE Val Unseen (Fig. 4); optimal SR at N_m=1000.
  • Top-K retrieval K = 16 (monocular), 10 (panoramic)
    Set per camera setting; no sensitivity analysis reported.
  • Update factors alpha, beta = 0.7 (monocular), 0.9 (panoramic)
    Chosen by hand; ablations not provided.
  • Predictive steps T_p = 2
    Selected via ablation on R2R-CE Val Unseen (Table 9); larger T_p hurts performance.
  • KL weight gamma = 1e-3
    Heuristically adjusted to balance loss terms (Supplementary Section 8).
assumptions (5)
  • standard math Variational lower bound via Jensen's inequality justifies the world-model loss (Supplementary Section 6).
    Used to derive L_W; standard ELBO argument.
  • domain assumption Latent dynamics factorize as in Eq. (7): x_t and a_t are conditionally independent given (h_t, s_t), and future predictions depend only on h_T, s_T in Eq. (5).
    Underlies the RSSM; if long-range dependencies are needed beyond this Markov structure, foresight predictions lose information.
  • domain assumption Teacher-action supervision (DAgger-style interactive demonstrator) is available for the imitation loss L_IL.
    Section 3.3; standard for VLN-CE but requires an oracle.
  • ad hoc to paper NDTW is a suitable, differentiable training objective (Eq. 6) and its implementation matches the evaluation NDTW.
    No implementation details or gradient derivation are given; also used as an evaluation metric, which can inflate NDTW/SDTW gains.
  • domain assumption Online test-time adaptation with batch size 1 and cross-episodic memory is a valid evaluation protocol.
    Section 4.1 and Discussion; based on prior work [19], but makes comparisons to static baselines not apples-to-apples.
invented entities (1)
  • Contextual Evolution Memory (CEM)
    purpose: Storage and online refinement of scene-contextual features; retrieved top-K to enrich deterministic latent states h_t.
    Introduced as a new module with forward EMA updates; no external falsifiable prediction, only internal ablations. Its benefit is measured on the paper's own benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environments." pith.science (2026). https://pith.science/paper/JNMJEG35

@misc{pith2026250623468,
  author       = {Pith},
  title        = {Pith review of: NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JNMJEG35}},
  note         = {Machine review of arXiv:2506.23468}
}
read the original abstract

Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires agents to execute sequential navigation actions in complex environments guided by natural language instructions. Current approaches often struggle with generalizing to novel environments and adapting to ongoing changes during navigation. Inspired by human cognition, we present NavMorph, a self-evolving world model framework that enhances environmental understanding and decision-making in VLN-CE tasks. NavMorph employs compact latent representations to model environmental dynamics, equipping agents with foresight for adaptive planning and policy refinement. By integrating a novel Contextual Evolution Memory, NavMorph leverages scene-contextual information to support effective navigation while maintaining online adaptability. Extensive experiments demonstrate that our method achieves notable performance improvements on popular VLN-CE benchmarks. Code is available at https://github.com/Feliciaxyao/NavMorph.

Figures

Figures reproduced from arXiv: 2506.23468 by the authors.

Figure 1
Figure 1. Self-Evolving World Model for Online VLN-CE tasks. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall framework of the proposed NavMorph, a self-evolving world model for VLN-CE tasks. World-aware Navigator (Infer [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Detailed architecture of our world model. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Experimental Results of Different Memory Sizes [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results of NavMorph on the R2R-CE dataset are presented, showcasing a comparison between ground truth paths [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

86 extracted references · 62 canonical work pages

  1. [1]

    Bevbert: Topo-metric map pre-training for language-guided navigation

    Dong An, Yuankai Qi, Yangguang Li, Yan Huang, Liang Wang, Tieniu Tan, and Jing Shao. Bevbert: Topo-metric map pre-training for language-guided navigation. arXiv preprint arXiv:2212.04385, 2022. 2, 4

    arXiv 2022

  2. [2]

    1st place so- lutions for rxr-habitat vision-and-language navigation com- petition

    Dong An, Zun Wang, Yangguang Li, Yi Wang, Yicong Hong, Yan Huang, Liang Wang, and Jing Shao. 1st place so- lutions for rxr-habitat vision-and-language navigation com- petition. arXiv preprint arXiv:2206.11610, 2022. 7, 5

    arXiv 2022

  3. [3]

    Etpnav: Evolving topo- logical planning for vision-language navigation in continu- ous environments

    Dong An, Hanqing Wang, Wenguan Wang, Zun Wang, Yan Huang, Keji He, and Liang Wang. Etpnav: Evolving topo- logical planning for vision-language navigation in continu- ous environments. IEEE TPAMI, 2024. 2, 4, 6, 7, 3, 5

    2024

  4. [4]

    On evaluation of embodied navigation agents

    Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757, 2018. 6, 2

    arXiv 2018

  5. [5]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko S ¨underhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, pages 3674–3683, 2018. 1, 2, 6, 7, 3, 4, 5

    2018

  6. [6]

    Sim- to-real transfer for vision-and-language navigation

    Peter Anderson, Ayush Shrivastava, Joanne Truong, Arjun Majumdar, Devi Parikh, Dhruv Batra, and Stefan Lee. Sim- to-real transfer for vision-and-language navigation. InCoRL, pages 671–681, 2021. 2

    2021

  7. [7]

    Robot-assisted navigation for visually impaired through adaptive impedance and path planning

    Pietro Balatti, Idil Ozdamar, Doganay Sirintuna, Luca For- tini, Mattia Leonori, Juan M Gandarias, and Arash Ajoudani. Robot-assisted navigation for visually impaired through adaptive impedance and path planning. In ICRA, pages 2310–2316, 2024. 1

    2024

  8. [8]

    The robotslang benchmark: Dialog-guided robot localization and navigation

    Shurjo Banerjee, Jesse Thomason, and Jason Corso. The robotslang benchmark: Dialog-guided robot localization and navigation. In CoRL, pages 1384–1393, 2021. 2

    2021

Show all 86 references
  1. [9]

    Navigation world models

    Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. ArXiv, 2024. 1, 2, 5

    2024

  2. [10]

    Matterport3d: Learning from rgb-d data in indoor environments

    Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158, 2017. 3

    2017 arXiv

  3. [11]

    Reinforced structured state-evolution for vision-language navigation

    Jinyu Chen, Chen Gao, Erli Meng, Qiong Zhang, and Si Liu. Reinforced structured state-evolution for vision-language navigation. In CVPR, pages 15429–15438, 2022. 4

    2022

  4. [12]

    Affordances-oriented planning using foundation models for continuous vision-language naviga- tion

    Jiaqi Chen, Bingqian Lin, Xinmin Liu, Xiaodan Liang, and Kwan-Yee K Wong. Affordances-oriented planning using foundation models for continuous vision-language naviga- tion. arXiv preprint arXiv:2407.05890, 2024. 7, 5

    2024 arXiv

  5. [13]

    Topological planning with transformers for vision-and-language navigation

    Kevin Chen, Junshen Chen, Jo Chuang, Marynel V’azquez, and Silvio Savarese. Topological planning with transformers for vision-and-language navigation. In CVPR, pages 11271– 11281, 2020. 4

    2020

  6. [14]

    Li, Mingkui Tan, and Chuang Gan

    Peihao Chen, Dongyu Ji, Kun-Li Channing Lin, Run- hao Zeng, Thomas H. Li, Mingkui Tan, and Chuang Gan. Weakly-supervised multi-granularity map learning for vision-and-language navigation. In NeurIPS, pages 38149– 38161, 2022. 7, 4, 5

    2022

  7. [15]

    A2 nav: Action-aware zero-shot robot navigation by exploit- ing vision-and-language ability of foundation models

    Peihao Chen, Xinyu Sun, Hongyan Zhi, Runhao Zeng, Thomas H Li, Gaowen Liu, Mingkui Tan, and Chuang Gan. A2 nav: Action-aware zero-shot robot navigation by exploit- ing vision-and-language ability of foundation models. arXiv preprint arXiv:2308.07997, 2023. 7, 5

    2023 arXiv

  8. [16]

    Think global, act local: Dual-scale graph transformer for vision-and-language navi- gation

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Think global, act local: Dual-scale graph transformer for vision-and-language navi- gation. In CVPR, pages 16537–16547, 2022. 3, 4

    2022

  9. [17]

    Unitedvln: Generalizable gaussian splatting for contin- uous vision-language navigation

    Guangzhao Dai, Jian Zhao, Yuantao Chen, Yusen Qin, Hao Zhao, Guosen Xie, Yazhou Yao, Xiangbo Shu, and Xuelong Li. Unitedvln: Generalizable gaussian splatting for contin- uous vision-language navigation. ArXiv, abs/2411.16053,

    arXiv

  10. [18]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 3

    2010 arXiv

  11. [19]

    Fast-slow test- time adaptation for online vision-and-language navigation

    Junyu Gao, Xuan Yao, and Changsheng Xu. Fast-slow test- time adaptation for online vision-and-language navigation. In ICML, pages 14902–14919, 2024. 1, 6, 7, 8, 4, 5

    2024

  12. [20]

    Cross-modal map learning for vision and language navigation

    Georgios Georgakis, Karl Schmeckpeper, Karan Wanchoo, Soham Dan, Eleni Miltsakaki, Dan Roth, and Kostas Dani- ilidis. Cross-modal map learning for vision and language navigation. In CVPR, pages 15439–15449, 2022. 7, 3, 4, 5

    2022

  13. [21]

    Note: Robust continual test- time adaptation against temporal correlation

    Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. Note: Robust continual test- time adaptation against temporal correlation. In NeurIPS, pages 27253–27266, 2022. 6

    2022

  14. [22]

    Recurrent world models facilitate policy evolution

    David Ha and J ¨urgen Schmidhuber. Recurrent world models facilitate policy evolution. In NeurIPS, 2018. 2

    2018

  15. [23]

    Learning latent dynamics for planning from pixels

    Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Ville- gas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In ICML, pages 2555–2565, 2019. 2, 3

    2019

  16. [24]

    Dream to control: Learning behaviors by la- tent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Moham- mad Norouzi. Dream to control: Learning behaviors by la- tent imagination. In ICLR, 2020. 2

    2020

  17. [25]

    Mastering diverse domains through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023. 2

    2023 arXiv

  18. [26]

    Reasoning with language model is planning with world model

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023. 1

    2023 arXiv

  19. [27]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016. 3

    2016

  20. [28]

    Frequency-enhanced data augmentation for vision-and-language navigation

    Keji He, Chenyang Si, Zhihe Lu, Yan Huang, Liang Wang, and Xinchao Wang. Frequency-enhanced data augmentation for vision-and-language navigation. In NeurIPS, 2024. 2

    2024

  21. [29]

    Bridg- ing the gap between learning in discrete and continuous en- vironments for vision-and-language navigation

    Yicong Hong, Zun Wang, Qi Wu, and Stephen Gould. Bridg- ing the gap between learning in discrete and continuous en- vironments for vision-and-language navigation. In CVPR, pages 15418–15428, 2022. 2, 3, 6, 7, 4, 5

    2022

  22. [30]

    Learning navigational visual representations with semantic map super- vision

    Yicong Hong, Yang Zhou, Ruiyi Zhang, Franck Dernon- court, Trung Bui, Stephen Gould, and Hao Tan. Learning navigational visual representations with semantic map super- vision. In Proceedings of the IEEE/CVF international con- ference on computer vision, pages 3032–3044, 2023. 4

    2023

  23. [31]

    Learning navigational visual representations with semantic map super- vision

    Yicong Hong, Yang Zhou, Ruiyi Zhang, Franck Dernon- court, Trung Bui, Stephen Gould, and Hao Tan. Learning navigational visual representations with semantic map super- vision. In ICCV, pages 3055–3067, 2023. 2

    2023

  24. [32]

    Model-based imitation learning for urban driving

    Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zachary Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton. Model-based imitation learning for urban driving. In NeurIPS, pages 20703–20716, 2022. 2

    2022

  25. [33]

    General evaluation for instruction con- ditioned navigation using dynamic time warping

    Gabriel Ilharco, Vihan Jain, Alexander Ku, Eugene Ie, and Jason Baldridge. General evaluation for instruction con- ditioned navigation using dynamic time warping. arXiv preprint arXiv:1907.05446, 2019. 5, 6, 2

    1907 arXiv

  26. [34]

    Semantically-aware spatio-temporal rea- soning agent for vision-and-language navigation in continu- ous environments

    Muhammad Zubair Irshad, Niluthpol Chowdhury Mithun, Zachary Seymour, Han-Pang Chiu, Supun Samarasekera, and Rakesh Kumar. Semantically-aware spatio-temporal rea- soning agent for vision-and-language navigation in continu- ous environments. In ICPR, pages 4065–4071, 2021. 4

    2021

  27. [35]

    Semantically-aware spatio-temporal rea- soning agent for vision-and-language navigation in continu- ous environments

    Muhammad Zubair Irshad, Niluthpol Chowdhury Mithun, Zachary Seymour, Han-Pang Chiu, Supun Samarasekera, and Rakesh Kumar. Semantically-aware spatio-temporal rea- soning agent for vision-and-language navigation in continu- ous environments. In ICPR, pages 4065–4071, 2022. 2

    2022

  28. [36]

    Sur les fonctions convexes et les in ´egalit´es entre les valeurs moyennes

    Johan Ludwig William Valdemar Jensen. Sur les fonctions convexes et les in ´egalit´es entre les valeurs moyennes. Acta mathematica, 30(1):175–193, 1906. 1

    1906

  29. [37]

    A new path: Scaling vision- and-language navigation with synthetic instructions and im- itation learning

    Aishwarya Kamath, Peter Anderson, Su Wang, Jing Yu Koh, Alexander Ku, Austin Waters, Yinfei Yang, Jason Baldridge, and Zarana Parekh. A new path: Scaling vision- and-language navigation with synthetic instructions and im- itation learning. In CVPR, pages 10813–10823, 2023. 2

    2023

  30. [38]

    Enhancing blind visitor’s autonomy in a science museum using an autonomous navigation robot

    Seita Kayukawa, Daisuke Sato, Masayuki Murata, Tatsuya Ishihara, Hironobu Takagi, Shigeo Morishima, and Chieko Asakawa. Enhancing blind visitor’s autonomy in a science museum using an autonomous navigation robot. In Proceed- ings of the 2023 CHI Conference on Human Factors in ...

    2023

  31. [39]

    Pathdreamer: A world model for indoor navigation

    Jing Yu Koh, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson. Pathdreamer: A world model for indoor navigation. In ICCV, pages 14738–14748, 2021. 1, 2, 4, 5

    2021

  32. [40]

    Sim-2-sim transfer for vision- and-language navigation in continuous environments

    Jacob Krantz and Stefan Lee. Sim-2-sim transfer for vision- and-language navigation in continuous environments. In ECCV, pages 588–603, 2022. 2, 6, 7, 4

    2022

  33. [41]

    Beyond the nav-graph: Vision-and- language navigation in continuous environments

    Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Ba- tra, and Stefan Lee. Beyond the nav-graph: Vision-and- language navigation in continuous environments. In ECCV, pages 104–120, 2020. 2, 3, 6

    2020

  34. [42]

    Waypoint models for instruction- guided navigation in continuous environments

    Jacob Krantz, Aaron Gokaslan, Dhruv Batra, Stefan Lee, and Oleksandr Maksymets. Waypoint models for instruction- guided navigation in continuous environments. In ICCV, pages 15162–15171, 2021. 2, 3

    2021

  35. [43]

    Room-across-room: Multilingual vision- and-language navigation with dense spatiotemporal ground- ing

    Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-room: Multilingual vision- and-language navigation with dense spatiotemporal ground- ing. In EMNLP, pages 4392–4412, 2020. 6, 3

    2020

  36. [44]

    Memory- maze: Scenario driven benchmark and visual language nav- igation model for guiding blind people

    Masaki Kuribayashi, Kohei Uehara, Allan Wang, Daisuke Sato, Simon Chu, and Shigeo Morishima. Memory- maze: Scenario driven benchmark and visual language nav- igation model for guiding blind people. arXiv preprint arXiv:2405.07060, 2024. 1

    2024

  37. [45]

    A path towards autonomous machine intelli- gence version 0.9

    Yann LeCun. A path towards autonomous machine intelli- gence version 0.9. 2, 2022-06-27. Open Review, 62(1):1–62,

    2022

  38. [46]

    Improving vision-and-language navigation by generating future-view image semantics

    Jialu Li and Mohit Bansal. Improving vision-and-language navigation by generating future-view image semantics. In CVPR, pages 10803–10812, 2023. 5

    2023

  39. [47]

    Panogen: Text-conditioned panoramic environment generation for vision-and-language navigation

    Jialu Li and Mohit Bansal. Panogen: Text-conditioned panoramic environment generation for vision-and-language navigation. In NeurIPS, 2024. 2

    2024

  40. [48]

    Vida: Homeostatic visual domain adapter for continual test time adaptation

    Jiaming Liu, Senqiao Yang, Peidong Jia, Ming Lu, Yandong Guo, Wei Xue, and Shanghang Zhang. Vida: Homeostatic visual domain adapter for continual test time adaptation. In ICLR, 2024. 6

    2024

  41. [49]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 364, 2019. 3

    1907 arXiv

  42. [50]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2017. 6

    2017

  43. [51]

    Openeqa: Embodied question answering in the era of foun- dation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foun- dation models. In CVPR, pages 16488–16498, 2024. 1

    2024

  44. [52]

    Driveworld: 4d pre-trained scene understanding via world models for autonomous driving

    Chen Min, Dawei Zhao, Liang Xiao, Jian Zhao, Xinli Xu, Zheng Zhu, Lei Jin, Jianshu Li, Yulan Guo, Junliang Xing, et al. Driveworld: 4d pre-trained scene understanding via world models for autonomous driving. In CVPR, pages 15522–15533, 2024. 2

    2024

  45. [53]

    Embodiedgpt: Vision-language pre-training via embodied chain of thought

    Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. In NeurIPS, 2024. 1

    2024

  46. [54]

    Towards stable test-time adaptation in dynamic wild world

    Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In ICLR, 2023. 6

    2023

  47. [55]

    Dreamingv2: Re- inforcement learning with discrete world models without re- construction

    Masashi Okada and Tadahiro Taniguchi. Dreamingv2: Re- inforcement learning with discrete world models without re- construction. In IROS, pages 985–991, 2022. 2

    2022

  48. [56]

    Agronav: Autonomous navigation framework for agricul- tural robots and vehicles using semantic segmentation and semantic line detection

    Shivam K Panda, Yongkyu Lee, and M Khalid Jawed. Agronav: Autonomous navigation framework for agricul- tural robots and vehicles using semantic segmentation and semantic line detection. In CVPR, pages 6272–6281, 2023. 1

    2023

  49. [57]

    Visual language navi- gation: A survey and open challenges

    Sang-Min Park and Young-Gab Kim. Visual language navi- gation: A survey and open challenges. Artificial Intelligence Review, 56(1):365–427, 2023. 1

    2023

  50. [58]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019. 6

    2019

  51. [59]

    Recore: Regularized contrastive represen- tation learning of world model

    Rudra PK Poudel, Harit Pandya, Stephan Liwicki, and Roberto Cipolla. Recore: Regularized contrastive represen- tation learning of world model. In CVPR, pages 22904– 22913, 2024. 2

    2024

  52. [60]

    Reverie: Remote embodied visual referring expres- sion in real indoor environments

    Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expres- sion in real indoor environments. In CVPR, pages 9982– 9991, 2020. 2

    2020

  53. [61]

    Hop+: History-enhanced and order- aware pre-training for vision-and-language navigation.IEEE TPAMI, 45(7):8524–8537, 2023

    Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop+: History-enhanced and order- aware pre-training for vision-and-language navigation.IEEE TPAMI, 45(7):8524–8537, 2023. 2

    2023

  54. [62]

    Sonia Raychaudhuri, Saim Wani, Shivansh Patel, Unnat Jain, and Angel X. Chang. Language-aligned waypoint (law) su- pervision for vision-and-language navigation in continuous environments. In EMNLP, pages 4018–4028, 2021. 7, 4, 5

    2021

  55. [63]

    A re- duction of imitation learning and structured prediction to no- regret online learning

    St ´ephane Ross, Geoffrey Gordon, and Drew Bagnell. A re- duction of imitation learning and structured prediction to no- regret online learning. In AISTATS, pages 627–635, 2011. 6

    2011

  56. [64]

    Habitat: A plat- form for embodied ai research

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A plat- form for embodied ai research. In ICCV, pages 9339–9347,

  57. [65]

    Lxmert: Learning cross- modality encoder representations from transformers

    Hao Tan and Mohit Bansal. Lxmert: Learning cross- modality encoder representations from transformers. arXiv preprint arXiv:1908.07490, 2019. 3

    1908 arXiv

  58. [66]

    A framework towards ambient assisted living en- hanced by service robots

    Petros Toupas, Georgios Tsamis, Andreas Kargakos, Dim- itrios Giakoumis, Konstantinos V otis, and Dimitrios Tzo- varas. A framework towards ambient assisted living en- hanced by service robots. In Proceedings of the 16th In- ternational Conference on PErvasive Technologies Rel...

    2023

  59. [67]

    Using interoper- ability between mobile robot and knx technology for occu- pancy monitoring in smart home care

    Jan Vanus, Radim Hercik, and Petr Bilik. Using interoper- ability between mobile robot and knx technology for occu- pancy monitoring in smart home care. Sensors, 23(21):8953,

  60. [68]

    Tent: Fully test-time adaptation by entropy minimization

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Ol- shausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021. 6

    2021

  61. [69]

    Dreamwalker: Mental planning for continuous vision-language navigation

    Hanqing Wang, Wei Liang, Luc Van Gool, and Wenguan Wang. Dreamwalker: Mental planning for continuous vision-language navigation. In ICCV, pages 10873–10883,

  62. [70]

    Embodiedscan: A holistic multi- modal 3d perception suite towards embodied ai

    Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al. Embodiedscan: A holistic multi- modal 3d perception suite towards embodied ai. In CVPR, pages 19757–19767, 2024. 1

    2024

  63. [71]

    Graph-based environment representation for vision-and- language navigation in continuous environments

    Ting Wang, Zongkai Wu, Feiyu Yao, and Donglin Wang. Graph-based environment representation for vision-and- language navigation in continuous environments. InICASSP, pages 8331–8335, 2024. 4

    2024

  64. [72]

    Driving into the future: Multiview visual forecasting and planning with world model for au- tonomous driving

    Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for au- tonomous driving. In CVPR, pages 14749–14759, 2024. 2

    2024

  65. [73]

    Scaling data generation in vision-and-language navigation

    Zun Wang, Jialu Li, Yicong Hong, Yi Wang, Qi Wu, Mohit Bansal, Stephen Gould, Hao Tan, and Yu Qiao. Scaling data generation in vision-and-language navigation. In Proceed- ings of the IEEE/CVF international conference on computer vision, pages 12009–12020, 2023. 4

    2023

  66. [74]

    Gridmm: Grid memory map for vision-and- language navigation

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang. Gridmm: Grid memory map for vision-and- language navigation. In ICCV, pages 15625–15636, 2023. 7, 3, 4

    2023

  67. [75]

    Sim-to-real transfer via 3d feature fields for vision- and-language navigation

    Zihan Wang, Xiangyang Li, Jiahao Yang, and Shuqiang Jiang. Sim-to-real transfer via 3d feature fields for vision- and-language navigation. In CoRL, 2024. 2, 6, 7, 3, 4, 5

    2024

  68. [76]

    Lookahead exploration with neural radiance representation for continuous vision- language navigation

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, Junjie Hu, Ming Jiang, and Shuqiang Jiang. Lookahead exploration with neural radiance representation for continuous vision- language navigation. In CVPR, pages 13753–13762, 2024. 6, 7, 4, 5

    2024

  69. [77]

    Daydreamer: World models for physical robot learning

    Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. In CoRL, pages 2226–2240, 2023. 2

    2023

  70. [78]

    Vision-language navigation: a survey and tax- onomy

    Wansen Wu, Tao Chang, Xinmeng Li, Quanjun Yin, and Yue Hu. Vision-language navigation: a survey and tax- onomy. Neural Computing and Applications , 36(7):3291– 3316, 2024. 1

    2024

  71. [79]

    Holodeck: Language guided gen- eration of 3d embodied ai environments

    Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Kr- ishna, Lingjie Liu, et al. Holodeck: Language guided gen- eration of 3d embodied ai environments. In CVPR, pages 16227–16237, 2024. 1

    2024

  72. [80]

    Navid: Video-based vlm plans the next step for vision-and-language navigation

    Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and Wang He. Navid: Video-based vlm plans the next step for vision-and-language navigation. In RSS, 2024. 2, 7, 4, 5

    2024

  73. [81]

    Storm: Efficient stochastic transformer based world models for reinforcement learning

    Weipu Zhang, Gang Wang, Jian Sun, Yetian Yuan, and Gao Huang. Storm: Efficient stochastic transformer based world models for reinforcement learning. In NeurIPS, 2024. 2 NavM rph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environments Suppleme...

    2024

  74. [82]

    There- fore, we need to infer the latent variables s1:T

    Lower Bound Derivation Following the predictive network described in § 3 of our main paper, the joint probability distribution for our pro- posed world model can be factorized as: p(h1:T , s1:T , x1:T +Tp a1:T +Tp ) = TY t=1 p(ht, st|ht−1, st−1, at−1)p(xt, at|ht, st) TpY j=1 p...

  75. [83]

    • NE (Navigation Error) measures the average distance (in meter) between the agent’s final position in the pre- dicted trajectory and the target in the ground truth

    Evaluation Metrics for VLN-CE agents We follow previous approaches [4, 5, 33] and adopt the stan- dard metrics for evaluating VLN-CE agents: • TL (Trajectory length) measures the average length of the predicted navigation trajectories. • NE (Navigation Error) measures the aver...

  76. [84]

    In conventional panoramic VLN-CE frameworks [3, 74], the agent perceives its sur- roundings through multi-view RGB-D panoramas captured at 30-degree intervals at each timestep t

    Implementation Details The Baseline Framework. In conventional panoramic VLN-CE frameworks [3, 74], the agent perceives its sur- roundings through multi-view RGB-D panoramas captured at 30-degree intervals at each timestep t. These panoramic observations are processed by a tra...

  77. [85]

    Walk out of the room with the bed on your right. Go through the doorway on your left, entering the bathroom

    Complementary Experiments 9.1. Full Results In our main paper, we provide representative comparison results on the R2R-CE [5, 41] and RxR-CE [41, 43] bench- marks due to space constraints. Here, we present the com- plete results across the ‘validation seen’, ‘validation un- se...

  78. [86]

    Since our world model encodes high-level features instead of raw images, direct visualization of latent states remains non-trivial

    Qualitative Analysis To evaluate the predictive performance of NavMorph, we conduct qualitative analysis by comparing trajectories gen- erated through our Foresight Action Planner with executed paths and ground truth sequences. Since our world model encodes high-level features...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.