Pith. sign in

REVIEW 4 major objections 5 minor 5 cited by

A 0.6B language model that reads grid-text descriptions of RGB-D views matches or beats much larger multimodal systems on vision-language navigation benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 05:38 UTC pith:XIN4SSIK

load-bearing objection A clever language-only VLN idea whose central benchmark claim is undercut by an undisclosed semantic-segmentation channel and by internal numerical contradictions. the 4 major comments →

arxiv 2603.27577 v3 pith:XIN4SSIK submitted 2026-03-29 cs.CV cs.RO

Structured Observation Language for Efficient and Generalizable Vision-Language Navigation

Daojie Peng , Fulong Ma , Jun Ma This is my paper
classification cs.CV cs.RO
keywords Vision-Language NavigationStructured Observation LanguageLarge Language ModelsRGB-D perceptionAction chunk predictionGeneralizable NavigationRobot Action Prediction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to show that an embodied navigation agent does not need to process raw pixels at all: egocentric RGB-D frames can be compressed into a grid of short text strings describing depth, dominant semantic category, and color per cell, and this text, concatenated with the instruction, is enough for a 0.6B-parameter pre-trained language model to predict navigation actions. If true, it would mean that a large fraction of vision-language navigation can be done as pure language reasoning, with a model more than ten times smaller than current multimodal baselines, no waypoint predictor, and no extra training data. The authors report state-of-the-art results on the R2R-CE validation-unseen split and competitive results on RxR-CE, plus a real-world deployment on a quadruped robot. A sympathetic reader would care because the approach directly targets the cost and generalization bottlenecks of VLN: it avoids scratch visual encoders and is insensitive to lighting and texture changes.

Core claim

The central claim: an egocentric RGB-D frame can be compressed into a grid of short text strings—each cell holding its average depth, dominant semantic class, and HSV-derived color name—and this text, arranged over multi-resolution grids with history and paired with the instruction, is enough for a 0.6B pre-trained language model to predict four-step action chunks. Reported on R2R-CE val-unseen: NE 5.11, OS 72.9, SR 53.6, SPL 49.2, best among methods without a waypoint predictor or extra training data; RxR-CE results are competitive. Ablations identify depth as the most critical input (SR drops from 53.6 to 21.6 without it), and the paper adds a real-world quadruped deployment at about 0.8s

What carries the argument

The load-bearing object is the structured observation language prompt: a time-ordered sequence of observation matrices, each cell of which is a string concatenation of average depth, dominant semantic class, and HSV-mapped color name. The current observation uses a 6x6 grid, short-term history uses 4x4 grids, and long-term history uses 2x2 grids; these are concatenated with a system description and the instruction into one text sequence. The argument runs through this representation: because the input is pure text, the navigation policy can be implemented entirely in a pre-trained language model with lightweight linear classification heads, and the model's commonsense reasoning is reused dir

Load-bearing premise

The benchmark tables likely rely on semantic segmentation maps that the simulator can supply directly from the dataset ground truth; if so, the claimed efficiency and generalization have not been tested against real perception error.

What would settle it

Run SOL-Nav on R2R-CE val-unseen with a learned segmenter in place of dataset-provided semantic maps; a significant drop from the reported SR/SPL 53.6/49.2 would show the headline depends on the oracle perception channel. Alternatively, check the evaluation code to see whether the semantic input in Tables 1-2 is ground-truth or predicted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the results hold, a 0.6B pure-language policy is sufficient for strong VLN-CE performance, making visual encoders and multimodal fusion optional rather than necessary.
  • Because structured text discards lighting, texture, and raw appearance, the policy should transfer to unseen environments with less sim-to-real gap than pixel-based policies.
  • Training cost drops: the model is fine-tuned with LoRA on 10 epochs with standard benchmark data, no auxiliary waypoint predictor, and no extra datasets.
  • Real-world deployment becomes practical: inference at about 0.8s on an edge device with a quadruped robot, using a fine-tuned segmentation model and 50 real navigation samples.
  • Ablations imply that depth information is the most critical channel; any practical instantiation must preserve reliable depth, while color and fine-grained shape are less essential.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The benchmark comparison is probably not perception-fair as reported: the paper notes segmentation maps 'can be obtained directly from the datasets,' which in the simulator means oracle semantics; if Tables 1-2 used those, part of the gain over RGB-only baselines comes from free perfect perception rather than the textual representation itself.
  • A direct test would be to train and evaluate the same pipeline with a fixed off-the-shelf segmenter in the loop; the real-world section already uses such a segmenter, so the oracle-to-learned gap is measurable.
  • If the claim holds, it suggests VLN reasoning is largely text-level spatial reasoning; a useful probe would be perturbing depth strings or color names and watching which channels the policy relies on.
  • The recipe—convert sensor data into compact discrete text before a language model—may transfer to other embodied tasks like manipulation, where the main cost is modal alignment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SOL-Nav, a VLN framework that converts egocentric RGB-D observations into structured textual descriptions via a multi-resolution grid (current 6×6, short-term history 4×4, long-term history 2×2), extracting per-cell depth, dominant semantic class, and color name. These structured observations are concatenated with the navigation instruction and a system description into a pure-language prompt, which is fed to a 0.6B Qwen3-Embedding encoder with per-step classification heads that predict an action chunk of four discrete actions. The authors report state-of-the-art or competitive results on R2R-CE and RxR-CE val-unseen splits and describe real-world deployment on a Unitree Go2 robot. The central claim is that a small, language-only policy can match or surpass much larger multimodal VLN models without extra training data or a waypoint predictor, while being more parameter-efficient and generalizable.

Significance. If the empirical results were fully substantiated, SOL-Nav would be a notable contribution: it would demonstrate that structured semantic-language observations can replace learned visual encoders in VLN, enabling a 0.6B model to compete with 7B+ multimodal systems and potentially improving robustness to lighting/texture variation. The idea of converting visual input into an explicit semantic grid is interesting and could inspire further work on language-only embodied agents. However, the current manuscript does not yet establish this significance: the source of the semantic segmentation maps used in the benchmark experiments is undisclosed, several internal numerical and textual contradictions appear, and the real-world evaluation is purely qualitative. The claimed efficiency and generalization advantages cannot be assessed until these issues are resolved.

major comments (4)
  1. [Section 3.2, Tables 1-2] The paper does not specify how the semantic segmentation maps were obtained for the R2R-CE and RxR-CE experiments. Section 3.2 states that the map 'can be obtained directly from the datasets or using pre-trained semantic segmentation model', and the observation columns in Tables 1-2 list an undefined 'S.RGB' input. If 'directly from the datasets' means the Habitat/MP3D ground-truth semantic sensor, then SOL-Nav is granted an oracle perception channel that is not available to RGB-based baselines such as NaVid, NaVILA, or UniNaVid. This would make the comparison unfair and would mean the claimed generalization to unseen environments is not tested against real perception error. If a learned segmenter was used, its identity, checkpoint, and any fine-tuning must be disclosed. This ambiguity is load-bearing for the paper's central claim and must be resolved either by using a realistic segmente
  2. [Section 4.1, Tables 1-2] There are internal inconsistencies between the text and the tables. In the RxR-CE paragraph, the text reports NE=6.95 and names the metrics 'Objective Success', 'SP', and 'Success weighted by Path Length (SPL)', while Table 2 shows NE=6.87 and correctly lists OS, SR, and SPL. In the R2R-CE paragraph, the text says SOL-Nav 'performs slightly worse than InternVLA-N1 on OS, SR, and SPL', but Table 1 shows SOL-Nav's OS (72.9) is substantially higher than InternVLA-N1's OS (63.3); the very next sentence then claims SOL-Nav surpasses InternVLA-N1 on OS. These contradictions undermine confidence in the reported empirical results and must be corrected, with the tables and prose made consistent.
  3. [Section 4.1] The claim that 'our model is more than 50 times smaller than these counterparts' is not supported by any model-size figures in the paper. The baselines (NaVILA, UniNaVid, InternVLA-N1) are not given parameter counts, and typical 7B-scale models would be only about 10x larger than the 0.6B SOL-Nav. Please provide the actual model sizes from the cited papers or remove this claim.
  4. [Section 4.3] The real-world deployment section reports no quantitative results—no success rate, path length, or task completion metrics—only the statement that 'the results confirm SOL-Nav's real-time performance and robustness'. Moreover, the real-world variant uses a fine-tuned SegFormer, but this version is not evaluated on the R2R/RxR benchmark splits, so it is unclear whether the benchmark numbers and the deployed system use the same perception pipeline. Please add quantitative real-world evaluation or explicitly limit the claims to a feasibility demonstration.
minor comments (5)
  1. [Section 1] There is a duplicated word: 'Specifically, Specifically, we first use'.
  2. [Tables 1-2] The observation column heading 'S.RGB' is never defined. The text should explain whether this denotes the structured semantic-RGB-description input or something else.
  3. [Section 3.2] The notation for grid resolutions is introduced as N_curr=6, N_short=4, N_long=2, but the values are not explicitly marked as grid sizes in the text before the 'N×N grid' formula; consider stating clearly that these are grid dimensions for current, short-term, and long-term observations.
  4. [Section 4.1] The phrases 'second column' and 'third column' in the R2R comparison are ambiguous because Table 1 does not have explicit columns beyond Method/Observation/metrics. It would be clearer to refer to row groups (e.g., 'methods without waypoint predictor' and 'methods with additional training data').
  5. [General] No code or checkpoint is provided, which further limits reproducibility. Even a provisional link would be helpful.

Circularity Check

0 steps flagged

No significant circularity: SOL-Nav's claims are empirical benchmark comparisons and a supervised action-prediction pipeline, not derivations that reduce to their own inputs.

full rationale

I walked the claimed derivation chain: the structured observation in Eq. (1) is built from raw RGB-D inputs plus semantic labels, and the model in Eq. (3) is trained with cross-entropy to predict action blocks. The predicted actions are not defined in terms of the structured observations by construction, and no fitted parameter is later renamed as a prediction. The central comparisons in Tables 1-2 are against external baselines on standard benchmarks, so the main success claim is an empirical result rather than a self-referential derivation. There is no load-bearing self-citation chain or imported uniqueness theorem: the paper cites prior systems (SegFormer, Qwen3, NaVILA, etc.) as tools or baselines, not as justification that its own output is forced. The main concerns are correctness/reproducibility risks, not circularity: the paper states semantic segmentation maps 'can be obtained directly from the datasets' without disclosing whether benchmark results use simulator-supplied ground-truth semantics, and hyperparameters appear tuned on validation performance. These issues affect whether the comparison is apples-to-apples and whether results are reproducible, but they do not make the result equivalent to its inputs by definition. Therefore no specific circular step can be exhibited under the required standard.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central claims rest on the availability of semantic maps, the sufficiency of the grid representation, and the transferability of simulator benchmarks; none of these is proven in the paper. No new physical entities are introduced; SOL-Nav is a representational scheme, not an ontological postulate.

free parameters (6)
  • Current grid resolution N_curr = 6x6
    Chosen for the baseline; ablation tests only 4x4 vs 6x6, so 6 is selected on validation performance.
  • Short-term history grid resolution N_short = 4x4
    Hand-chosen; no ablation or search reported.
  • Long-term history grid resolution N_long = 2x2
    Hand-chosen; no ablation or search reported.
  • History lengths n_short, n_long = 2 and 16 frames
    Hand-chosen; no ablation or search reported.
  • Action chunk length N_a = 4
    Hand-chosen; defines the prediction target for the language model.
  • Balanced class weights w = computed from training class frequencies
    Fitted to the training set to counter class imbalance.
axioms (5)
  • domain assumption Ground-truth semantic segmentation maps are available for benchmark observations
    Section 3.2: 'semantic segmentation map ... can be obtained directly from the datasets'; this may give the model oracle semantic labels not available to baselines.
  • domain assumption A grid cell's dominant semantic label, average depth, and named color preserve enough spatial information for navigation
    Section 3.2 defines observations exclusively from these three per-cell strings; if they are insufficient, the policy cannot compensate.
  • domain assumption Qwen3-Embedding-0.6B can reason spatially from coordinate-tagged text
    Section 3.3 assumes a text-embedding model with classification heads provides the reasoning needed for 4-step action prediction.
  • domain assumption Standard benchmark metrics and splits (R2R-CE/RxR-CE val-unseen) measure generalization to unseen environments
    The generalization claim rests on the simulated benchmarks; real-world deployment is only qualitative.
  • domain assumption The Habitat simulator's action effects (15-degree turns, 25 cm steps) are an adequate proxy for the real robot
    The action space is defined in Section 3.1 and used throughout; real-world validation has no quantitative comparisons.

pith-pipeline@v1.3.0-alltime-deepseek · 13328 in / 19555 out tokens · 200916 ms · 2026-08-04T05:38:23.288458+00:00 · methodology

0 comments
read the original abstract

Vision-Language Navigation (VLN) requires an embodied agent to navigate complex environments by following natural language instructions, which typically demands tight fusion of visual and language modalities. Existing VLN methods often convert raw images into visual tokens or implicit features, requiring large-scale visual pre-training and suffering from poor generalization under environmental variations (e.g., lighting, texture). To address these issues, we propose SOL-Nav (Structured Observation Language for Navigation), a novel framework that translates egocentric visual observations into compact structured language descriptions for efficient and generalizable navigation. Specifically, we divide RGB-D images into a NxN grid, extract representative semantic, color, and depth information for each grid cell to form structured text, and concatenate this with the language instruction as pure language input to a pre-trained language model (PLM). Experimental results on standard VLN benchmarks (R2R, RxR) and real-world deployments demonstrate that SOL-Nav significantly reduces the model size and training data dependency, fully leverages the reasoning and representation capabilities of PLMs, and achieves strong generalization to unseen environments.

Figures

Figures reproduced from arXiv: 2603.27577 by Daojie Peng, Fulong Ma, Jun Ma.

Figure 1
Figure 1. Figure 1: Pipeline of SOL-Nav. RGB-D observations are converted into structured tex￾tual descriptions with 2×2/4×4/6×6 multi-resolution grids (long/short-term history, current observation) encoding depth, semantic, and color information. The structured observation sequence, navigation instruction, and system description form a pure lan￾guage prompt, which is input to a LLM to predict a consecutive action block for t… view at source ↗
Figure 2
Figure 2. Figure 2: Structured Observation Language Prompt for LLM. The prompt integrates system description (Dsystem), structured observation (Ostructure), and task instruction (Itask) to provide clear system definition, structured observations, and explicit predic￾tion requirements for the language model [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Real-world Deployments. We conduct real-world navigation experiments in three distinct scenarios with varying environmental characteristics (Tea Area, Hall Stairs, Meeting Room), to comprehensively evaluate the robustness and generalization of SOL-Nav. 4.3 Real-World Deployments To validate the practical applicability of SOL-Nav, we carried out real-world deployment experiments using the Unitree Go2 robot … view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Token-Wise Latent Streaming from Slow Reasoners to Fast Planners for Dynamic Vision Language Navigation

    cs.RO 2026-07 conditional novelty 6.0

    Streaming intermediate hidden states from a slow VLM to a fast flow-matching planner, token by token, improves dynamic social VLN success and reduces observation staleness.

  2. GeoSem-WAM: Geometry- and Semantic-Aware World Action Models

    cs.RO 2026-06 unverdicted novelty 5.0

    GeoSem-WAM adds geometric and semantic auxiliary prediction tasks to World Action Models during training to improve latent representations and action prediction accuracy while keeping inference efficient by avoiding e...

  3. AttenA+: Rectifying Action Inequality in Robotic Foundation Models

    cs.RO 2026-05 unverdicted novelty 5.0

    AttenA+ applies velocity-driven action attention to reweight training objectives toward kinematically critical low-velocity segments, yielding small benchmark gains on Libero and RoboTwin without added parameters.

  4. Towards Dual-Brain Minimal Sufficient Representation for Vision-Language Navigation

    cs.CV 2026-07 reject novelty 4.0

    A CP-decomposed, instruction-conditioned latent bottleneck (CompactNav) improves VLN-CE success rate by about 2% over prior state of the art on two benchmarks.

  5. AttenA+: Rectifying Action Inequality in Robotic Foundation Models

    cs.RO 2026-05 unverdicted novelty 4.0

    AttenA+ reweights action training objectives in VLA and WAM models via inverse velocity attention to prioritize kinematically critical segments, yielding small benchmark gains.

Reference graph

Works this paper leans on

61 extracted references · 22 linked inside Pith · cited by 4 Pith papers

  1. [1]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    An, D., Wang, H., Wang, W., Wang, Z., Huang, Y., He, K., Wang, L.: Etp- nav: Evolving topological planning for vision-language navigation in contin- uous environments. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    2024

  2. [2]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid, I., Gould, S., Van Den Hengel, A.: Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3674–3683 (2018)

    2018

  3. [3]

    arXiv preprint arXiv:2004.05150 (2020)

    Beltagy, I., Peters, M.E., Cohan, A.: Longformer: The long-document trans- former. arXiv preprint arXiv:2004.05150 (2020)

    Pith/arXiv arXiv 2004

  4. [4]

    arXiv preprint arXiv:2505.06111 (2025)

    Bu, Q., Yang, Y., Cai, J., Gao, S., Ren, G., Yao, M., Luo, P., Li, H.: Univla: Learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2505.06111 (2025)

    Pith/arXiv arXiv 2025

  5. [5]

    arXiv preprint arXiv:1709.06158 (2017)

    Chang, A., Dai, A., Funkhouser, T., Halber, M., Niessner, M., Savva, M., Song, S., Zeng, A., Zhang, Y.: Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158 (2017)

    Pith/arXiv arXiv 2017

  6. [6]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Chen, J., Lin, B., Xu, R., Chai, Z., Liang, X., Wong, K.Y.: Mapgpt: Map- guided prompting with adaptive path planning for vision-and-language nav- igation. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 9796–9810 (2024)

    2024

  7. [7]

    In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chen, K., Chen, J.K., Chuang, J., Vázquez, M., Savarese, S.: Topologi- cal planning with transformers for vision-and-language navigation. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11276–11286 (2021)

    2021

  8. [8]

    ArXivabs/2210.07506(2022)

    Chen, P., Ji, D., Lin, K.L.C., Zeng, R., Li, T.H., Tan, M., Gan, C.: Weakly- supervised multi-granularity map learning for vision-and-language naviga- tion. ArXivabs/2210.07506(2022)

    Pith/arXiv arXiv 2022

  9. [9]

    Advances in Neural Information Processing Systems35, 38149–38161 (2022)

    Chen, P., Ji, D., Lin, K., Zeng, R., Li, T., Tan, M., Gan, C.: Weakly- supervised multi-granularity map learning for vision-and-language naviga- tion. Advances in Neural Information Processing Systems35, 38149–38161 (2022)

    2022

  10. [10]

    arXiv preprint arXiv:2412.04453 (2024)

    Cheng, A.C., Ji, Y., Yang, Z., Gongye, Z., Zou, X., Kautz, J., Bıyık, E., Yin, H., Liu, S., Wang, X.: Navila: Legged robot vision-language-action model for navigation. arXiv preprint arXiv:2412.04453 (2024)

    Pith/arXiv arXiv 2024

  11. [11]

    IEEE Transactions on Emerging Topics in Computational Intelligence6(2), 230–244 (2022)

    Duan, J., Yu, S., Tan, H.L., Zhu, H., Tan, C.: A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence6(2), 230–244 (2022)

    2022

  12. [12]

    arXiv preprint arXiv:2506.09839 (2025) 16 F

    Gao, C., Jin, L., Peng, X., Zhang, J., Deng, Y., Li, A., Wang, H., Liu, S.: Octonav: Towards generalist embodied navigation. arXiv preprint arXiv:2506.09839 (2025) 16 F. Author et al

    Pith/arXiv arXiv 2025

  13. [13]

    In: Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition

    Georgakis, G., Schmeckpeper, K., Wanchoo, K., Dan, S., Miltsakaki, E., Roth, D., Daniilidis, K.: Cross-modal map learning for vision and language navigation. In: Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition. pp. 15460–15470 (2022)

    2022

  14. [14]

    2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp

    Georgakis, G., Schmeckpeper, K., Wanchoo, K., Dan, S., Miltsakaki, E., Roth, D., Daniilidis, K.: Cross-modal map learning for vision and language navigation. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 15439–15449 (2022)

    2022

  15. [15]

    arXiv preprint arXiv:2411.05755 (2024)

    Goetting, D., Singh, H.G., Loquercio, A.: End-to-end navigation with vision language models: Transforming spatial reasoning into question-answering. arXiv preprint arXiv:2411.05755 (2024)

    Pith/arXiv arXiv 2024

  16. [16]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Hong, Y., Wang, Z., Wu, Q., Gould, S.: Bridging the gap between learning in discrete and continuous environments for vision-and-language naviga- tion. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15439–15449 (2022)

    2022

  17. [17]

    Iclr 1(2), 3 (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. Iclr 1(2), 3 (2022)

    2022

  18. [18]

    In: The Twelfth International Conference on Learning Representations (2023)

    Islam, M.M., Gladstone, A., Islam, R., Iqbal, T.: Eqa-mx: Embodied ques- tion answering using multimodal expression. In: The Twelfth International Conference on Learning Representations (2023)

    2023

  19. [19]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Kamath, A., Anderson, P., Wang, S., Koh, J.Y., Ku, A., Waters, A., Yang, Y., Baldridge, J., Parekh, Z.: A new path: Scaling vision-and-language nav- igation with synthetic instructions and imitation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10813–10823 (2023)

    2023

  20. [20]

    In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision

    Krantz, J., Gokaslan, A., Batra, D., Lee, S., Maksymets, O.: Waypoint mod- els for instruction-guided navigation in continuous environments. In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15162–15171 (2021)

    2021

  21. [21]

    In: European conference on computer vision

    Krantz, J., Lee, S.: Sim-2-sim transfer for vision-and-language navigation in continuous environments. In: European conference on computer vision. pp. 588–603. Springer (2022)

    2022

  22. [22]

    In: European Conference on Computer Vision

    Krantz, J., Wijmans, E., Majumdar, A., Batra, D., Lee, S.: Beyond the nav-graph: Vision-and-language navigation in continuous environments. In: European Conference on Computer Vision. pp. 104–120. Springer (2020)

    2020

  23. [23]

    In: European Conference on Computer Vision (2020),https : / / api

    Krantz, J., Wijmans, E., Majumdar, A., Batra, D., Lee, S.: Beyond the nav-graph: Vision-and-language navigation in continuous environments. In: European Conference on Computer Vision (2020),https : / / api . semanticscholar.org/CorpusID:214802389

    2020

  24. [24]

    arXiv preprint arXiv:2010.07954 (2020)

    Ku, A., Anderson, P., Patel, R., Ie, E., Baldridge, J.: Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. arXiv preprint arXiv:2010.07954 (2020)

    Pith/arXiv arXiv 2010

  25. [25]

    Scientific Reports15(1), Abbreviated paper title 17 40854 (nov 2025).https://doi.org/10.1038/s41598- 025- 24576- 6, https://doi.org/10.1038/s41598-025-24576-6

    Li, X., Pu, X., Ling, W., et al.: Yolo-sam: an end-to-end framework for effi- cient real time object detection and segmentation. Scientific Reports15(1), Abbreviated paper title 17 40854 (nov 2025).https://doi.org/10.1038/s41598- 025- 24576- 6, https://doi.org/10.1038/s41598-025-24576-6

    doi:10.1038/s41598- 2025

  26. [26]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2025)

    Lin, B., Nie, Y., Wei, Z., Chen, J., Ma, S., Han, J., Xu, H., Chang, X., Liang, X.: Navcot: Boosting llm-based vision-and-language navigation via learning disentangled reasoning. IEEE Transactions on Pattern Analysis and Machine Intelligence (2025)

    2025

  27. [27]

    IEEE/ASME Transactions on Mechatronics (2025)

    Liu, Y., Chen, W., Bai, Y., Liang, X., Li, G., Gao, W., Lin, L.: Aligning cyber space with physical world: A comprehensive survey on embodied ai. IEEE/ASME Transactions on Mechatronics (2025)

    2025

  28. [28]

    arXiv preprint arXiv:2406.04882 (2024)

    Long, Y., Cai, W., Wang, H., Zhan, G., Dong, H.: Instructnav: Zero-shot system for generic instruction navigation in unexplored environment. arXiv preprint arXiv:2406.04882 (2024)

    Pith/arXiv arXiv 2024

  29. [29]

    arXiv preprint arXiv:2508.02046 (2025)

    Luo, Z., Yan, W., Gong, J., Wang, M., Zhang, Z., Wang, X., Xie, Y., Tan, X.: Navimaster: Learning a unified policy for gui and embodied navigation tasks. arXiv preprint arXiv:2508.02046 (2025)

    Pith/arXiv arXiv 2025

  30. [30]

    ACM Transactions on Human-Robot Interaction12(3), 1–39 (2023)

    Mavrogiannis, C., Baldini, F., Wang, A., Zhao, D., Trautman, P., Stein- feld, A., Oh, J.: Core challenges of social robot navigation: A survey. ACM Transactions on Human-Robot Interaction12(3), 1–39 (2023)

    2023

  31. [31]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Qi, Y., Wu, Q., Anderson, P., Wang, X., Wang, W.Y., Shen, C., Hengel, A.v.d.: Reverie: Remote embodied visual referring expression in real indoor environments. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9982–9991 (2020)

    2020

  32. [32]

    In: 2025 IEEE International Conference on Robotics and Automation (ICRA)

    Qiao, Y., Lyu, W., Wang, H., Wang, Z., Li, Z., Zhang, Y., Tan, M., Wu, Q.: Open-nav: Exploring zero-shot vision-and-language navigation in con- tinuous environment with open-source llms. In: 2025 IEEE International Conference on Robotics and Automation (ICRA). pp. 6710–6717. IEEE (2025)

    2025

  33. [33]

    arXiv preprint arXiv:2109.08238 (2021)

    Ramakrishnan, S.K., Gokaslan, A., Wijmans, E., Maksymets, O., Clegg, A., Turner, J., Undersander, E., Galuba, W., Westbury, A., Chang, A.X., et al.: Habitat-matterport3ddataset(hm3d):1000large-scale3denvironmentsfor embodied ai. arXiv preprint arXiv:2109.08238 (2021)

    Pith/arXiv arXiv 2021

  34. [34]

    In: Proceedings of the 2021 conference on empirical methods in natural language processing

    Raychaudhuri, S., Wani, S., Patel, S., Jain, U., Chang, A.: Language-aligned waypoint(law)supervisionforvision-and-languagenavigationincontinuous environments. In: Proceedings of the 2021 conference on empirical methods in natural language processing. pp. 4018–4028 (2021)

    2021

  35. [35]

    Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., Zeng, Z., Zhang, H., Li, F., Yang, J., Li, H., Jiang, Q., Zhang, L.: Grounded sam: Assembling open-world models for diverse visual tasks (2024)

    2024

  36. [36]

    In: Proceedings of the IEEE/CVF international con- ference on computer vision

    Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al.: Habitat: A platform for embodied ai research. In: Proceedings of the IEEE/CVF international con- ference on computer vision. pp. 9339–9347 (2019)

    2019

  37. [37]

    IEEE Transactions on Pattern Analysis and Machine Intelligence45(10), 11948–11960 (2023) 18 F

    Tan, S., Ge, M., Guo, D., Liu, H., Sun, F.: Knowledge-based embodied question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence45(10), 11948–11960 (2023) 18 F. Author et al

    2023

  38. [38]

    arXiv preprint arXiv:2505.23189 (2025)

    Wang, S., Zhang, J., Li, M., Liu, J., Li, A., Wu, K., Zhong, F., Yu, J., Zhang, Z., Wang, H.: Trackvla: Embodied visual tracking in the wild. arXiv preprint arXiv:2505.23189 (2025)

    Pith/arXiv arXiv 2025

  39. [39]

    arXiv preprint arXiv:2508.02549 (2025)

    Wang, S., Wang, Y., Fan, Z., Wang, Y., Chen, M., Wang, K., Su, Z., Li, W., Cai, X., Jin, Y., et al.: Monodream: Monocular vision-language navigation with panoramic dreaming. arXiv preprint arXiv:2508.02549 (2025)

    arXiv 2025

  40. [40]

    In: Proceedings of the IEEE/CVF In- ternational conference on computer vision

    Wang, Z., Li, X., Yang, J., Liu, Y., Jiang, S.: Gridmm: Grid memory map for vision-and-language navigation. In: Proceedings of the IEEE/CVF In- ternational conference on computer vision. pp. 15625–15636 (2023)

    2023

  41. [41]

    arXiv preprint arXiv:2406.09798 (2024)

    Wang, Z., Li, X., Yang, J., Liu, Y., Jiang, S.: Sim-to-real transfer via 3d feature fields for vision-and-language navigation. arXiv preprint arXiv:2406.09798 (2024)

    Pith/arXiv arXiv 2024

  42. [42]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Wang, Z., Li, J., Hong, Y., Wang, Y., Wu, Q., Bansal, M., Gould, S., Tan, H., Qiao, Y.: Scaling data generation in vision-and-language navigation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 12009–12020 (2023)

    2023

  43. [43]

    arXiv preprint arXiv:2512.08186 (2025)

    Wei, M., Wan, C., Peng, J., Yu, X., Yang, Y., Feng, D., Cai, W., Zhu, C., Wang, T., Pang, J., et al.: Ground slow, move fast: A dual-system founda- tion model for generalizable vision-and-language navigation. arXiv preprint arXiv:2512.08186 (2025)

    arXiv 2025

  44. [44]

    arXiv preprint arXiv:2507.05240 (2025)

    Wei, M., Wan, C., Yu, X., Wang, T., Yang, Y., Mao, X., Zhu, C., Cai, W., Wang, H., Chen, Y., et al.: Streamvln: Streaming vision-and-language navigation via slowfast context modeling. arXiv preprint arXiv:2507.05240 (2025)

    Pith/arXiv arXiv 2025

  45. [45]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wijmans, E., Datta, S., Maksymets, O., Das, A., Gkioxari, G., Lee, S., Essa, I., Parikh, D., Batra, D.: Embodied question answering in photorealistic en- vironments with point cloud perception. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6659–6668 (2019)

    2019

  46. [46]

    Neural Computing and Applications36(7), 3291– 3316 (2024)

    Wu, W., Chang, T., Li, X., Yin, Q., Hu, Y.: Vision-language navigation: a survey and taxonomy. Neural Computing and Applications36(7), 3291– 3316 (2024)

    2024

  47. [47]

    In: Neural Information Processing Systems (NeurIPS) (2021)

    Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Seg- former: Simple and efficient design for semantic segmentation with trans- formers. In: Neural Information Processing Systems (NeurIPS) (2021)

    2021

  48. [48]

    arXiv preprint arXiv:2510.18840 (2025)

    Xing, L., Wang, A.J., Yan, R., Qu, H., Li, Z., Tang, J.: See the text: From tokenization to visual reading. arXiv preprint arXiv:2510.18840 (2025)

    arXiv 2025

  49. [49]

    arXiv preprint arXiv:2505.09388 (2025)

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025)

    Pith/arXiv arXiv 2025

  50. [50]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yu, L., Chen, X., Gkioxari, G., Bansal, M., Berg, T.L., Batra, D.: Multi- target embodied question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6309–6318 (2019)

    2019

  51. [51]

    arXiv preprint arXiv:2406.10721 (2024) Abbreviated paper title 19

    Yuan, W., Duan, J., Blukis, V., Pumacay, W., Krishna, R., Murali, A., Mousavian, A., Fox, D.: Robopoint: A vision-language model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406.10721 (2024) Abbreviated paper title 19

    Pith/arXiv arXiv 2024

  52. [52]

    Advances in neural information processing systems33, 17283–17297 (2020)

    Zaheer, M., Guruganesh, G., Dubey, K.A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al.: Big bird: Transformers for longer sequences. Advances in neural information processing systems33, 17283–17297 (2020)

    2020

  53. [53]

    Zhang, J., Wang, K., Wang, S., Li, M., Liu, H., Wei, S., Wang, Z., Zhang, Z., Wang, H.: Uni-navid: A video-based vision-language-action model for unifyingembodiednavigationtasks.arXivpreprintarXiv:2412.06224(2024)

    Pith/arXiv arXiv 2024

  54. [54]

    arXiv preprint arXiv:2402.15852 (2024)

    Zhang, J., Wang, K., Xu, R., Zhou, G., Hong, Y., Fang, X., Wu, Q., Zhang, Z., Wang, H.: Navid: Video-based vlm plans the next step for vision-and- language navigation. arXiv preprint arXiv:2402.15852 (2024)

    Pith/arXiv arXiv 2024

  55. [55]

    arXiv preprint arXiv:2502.13451 (2025)

    Zhang, L., Hao, X., Xu, Q., Zhang, Q., Zhang, X., Wang, P., Zhang, J., Wang, Z., Zhang, S., Xu, R.M.: A novel memory representation via anno- tated semantic maps for vlm-based vision-and-language navigation. arXiv preprint arXiv:2502.13451 (2025)

    Pith/arXiv arXiv 2025

  56. [56]

    arXiv preprint arXiv:2508.04598 (2025)

    Zhang, L., Hao, X., Tang, Y., Fu, H., Zheng, X., Wang, P., Wang, Z., Ding, W., Zhang, S.: Nava3: Understanding any instruction, navigating anywhere, finding anything. arXiv preprint arXiv:2508.04598 (2025)

    Pith/arXiv arXiv 2025

  57. [57]

    arXiv preprint arXiv:2506.05176 (2025)

    Zhang, Y., Li, M., Long, D., Zhang, X., Lin, H., Yang, B., Xie, P., Yang, A., Liu, D., Lin, J., et al.: Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176 (2025)

    Pith/arXiv arXiv 2025

  58. [58]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zheng, D., Huang, S., Zhao, L., Zhong, Y., Wang, L.: Towards learn- ing a generalist model for embodied navigation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13624–13634 (2024)

    2024

  59. [59]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Zhou, G., Hong, Y., Wu, Q.: Navgpt: Explicit reasoning in vision-and- language navigation with large language models. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 7641–7649 (2024)

    2024

  60. [60]

    arXiv preprint arXiv:2108.04097 (2021)

    Zhu, F., Zhu, Y., Lee, V., Liang, X., Chang, X.: Deep learning for embodied vision navigation: A survey. arXiv preprint arXiv:2108.04097 (2021)

    Pith/arXiv arXiv 2021

  61. [61]

    In: 2017 IEEE international conference on robotics and au- tomation (ICRA)

    Zhu, Y., Mottaghi, R., Kolve, E., Lim, J.J., Gupta, A., Fei-Fei, L., Farhadi, A.: Target-driven visual navigation in indoor scenes using deep reinforce- ment learning. In: 2017 IEEE international conference on robotics and au- tomation (ICRA). pp. 3357–3364. IEEE (2017)

    2017